The sound of reading: Color-to-timbre substitution boosts reading performance via OVAL, a novel auditory orthography optimized for visual-to-auditory mapping

Reading is a unique human cognitive skill and its acquisition was proven to extensively affect both brain organization and neuroanatomy. Differently from western sighted individuals, literacy rates via tactile reading systems, such as Braille, are declining, thus imposing an alarming threat to literacy among non-visual readers. This decline is due to many reasons including the length of training needed to master Braille, which must also include extensive tactile sensitivity exercises, the lack of proper Braille instruction and the high costs of Braille devices. The far-reaching consequences of low literacy rates, raise the need to develop alternative, cheap and easy-to-master non-visual reading systems. To this aim, we developed OVAL, a new auditory orthography based on a visual-to-auditory sensory-substitution algorithm. Here we present its efficacy for successful words-reading, and investigation of the extent to which redundant features defining characters (i.e., adding specific colors to letters conveyed into audition via different musical instruments) facilitate or impede auditory reading outcomes. Thus, we tested two groups of blindfolded sighted participants who were either exposed to a monochromatic or to a color version of OVAL. First, we showed that even before training, all participants were able to discriminate between 11 OVAL characters significantly more than chance level. Following 6 hours of specific OVAL training, participants were able to identify all the learned characters, differentiate them from untrained letters, and read short words/pseudo-words of up to 5 characters. The Color group outperformed the Monochromatic group in all tasks, suggesting that redundant characters' features are beneficial for auditory reading. Overall, these results suggest that OVAL is a promising auditory-reading tool that can be used by blind individuals, by people with reading deficits as well as for the investigation of reading specific processing dissociated from the visual modality.

Does orthographic processing emerge rapidly after learning a new script?

Orthographic processing is characterized by location-invariant and location-specific processing (Grainger, 2018): (1) strings of letters are more vulnerable to transposition effects than the strings of symbols in same-different tasks (location-invariant processing); and (2) strings of letters, but not strings of symbols, show an initial position advantage in target-in-string identification tasks (location-specific processing). To examine the emergence of these two markers of orthographic processing, we conducted a same-different task and a target-in-string identification task with two unfamiliar scripts (pre-training experiments). Across six training sessions, participants learned to fluently read and write one of these scripts. The post-training experiments were parallel to the pre-training experiments. Results showed that the magnitude of the transposed-letter effect in the same-different task and the serial function in the target-in-string identification tasks were remarkably similar for the trained and untrained scripts. Thus, location-invariant and location-specific processing does not emerge rapidly after learning a new script; instead, they may require thorough experience with specific orthographic structures.

Dynamic relationships between phonological memory and reading: A five year longitudinal study from age 4 to 9

We reconcile competing theories of the role of phonological memory in reading development, by uncovering their dynamic relationship during the first 5 years of school. Phonological memory, reading and phoneme awareness were assessed in 780 phonics-educated children at age 4, 5, 6 and 9. Confirmatory factor analyses demonstrated that phonological memory loaded onto two factors: verbal short-term memory (verbal STM; phonological tasks that loaded primarily on serial order memory) and nonword repetition. Using longitudinal structural equation models, we found that verbal STM directly predicted early word-level reading from age 4 to 6, reflecting the importance of serial-order memory for letter-by-letter decoding. In contrast, reading had no reciprocal influence on the development of verbal STM. The relationship between nonword repetition and reading was bidirectional across the 5 years of study: nonword repetition and reading predicted each other both directly and indirectly (via phoneme awareness). Indirect effects from nonword repetition (and verbal STM) to reading support the view that phonological memory stimulates phonemically detailed representations through repeated encoding of complex verbal stimuli. Similarly, the indirect influence of reading on nonword repetition suggests that improved reading ability promotes the phoneme-level specificity of phonological representations. Finally, the direct influence from reading to nonword repetition suggests that better readers use orthographic cues to help them remember and repeat new words accurately. A video abstract of this article can be viewed at https://www.youtube.com/watch?v=70LZfTR0BjE.

Columban Simulation Project 2.0: Numerical Competence and Orthographic Processing in Pigeons and Primates

Thirty years ago Burrhus Frederic Skinner and Robert Epstein began what is known as the Columban Simulation Project. With pigeons as their subjects, they simulated a series of studies that purportedly demonstrated insight, self-recognition, and symbolic communication in chimpanzees. In each case, with the appropriate training, they demonstrated that pigeons performed in a comparable manner to chimpanzees. When discussing these studies in the context of his Null Hypothesis, Macphail paid little attention to how the pigeons and chimpanzees solved the tasks and simply assumed that successful performance on the tasks reflected a similar underlying mechanism. Here, following a similar process to the original Columban Simulation Project, we go beyond this success testing and employ the signature testing approach to assess whether pigeons and primates employ a similar mechanism on tasks that tap numerical competence and orthographic processing. Consistent with the Null Hypothesis, pigeons and primates successfully passed novel transfer tests and, critically, displayed comparable cognitive signatures. While these findings demonstrate the absence of a qualitative difference, the time taken to train pigeons on these tasks revealed a clear quantitative difference.

The characteristics of face configural effect in illiterates and literates

Literacy acquisition can modulate the way we process visual words and language. However, little is known about its function in reshaping how we process non-linguistic materials, like faces. In this study, we explored this question by comparing the facial recognition skills of illiterate and literate adults in China. Our results showed that illiterates were less sensitive to changes in spatial configuration among key features in upright faces when stimuli were presented simultaneously. The differences in sensitivity of spatial configuration between the literates and illiterates were also observed in house processing. These results thus provide evidence that literacy acquisition during childhood could reshape configural processing.

Positional encoding in cotton-top tamarins (Saguinus oedipus)

Strategies used in artificial grammar learning can shed light into the abilities of different species to extract regularities from the environment. In the A(X)ⁿB rule, A and B items are linked, but assigned to different positional categories and separated by distractor items. Open questions are how widespread is the ability to extract positional regularities from A(X)ⁿB patterns, which strategies are used to encode positional regularities and whether individuals exhibit preferences for absolute or relative position encoding. We used visual arrays to investigate whether cotton-top tamarins (Saguinusoedipus) can learn this rule and which strategies they use. After training on a subset of exemplars, two of the tested monkeys successfully generalized to novel combinations. These tamarins discriminated between categories of tokens with different properties (A, B, X) and detected a positional relationship between non-adjacent items even in the presence of novel distractors. The pattern of errors revealed that successful subjects used visual similarity with training stimuli to solve the task and that successful tamarins extracted the relative position of As and Bs rather than their absolute position, similarly to what has been observed in other species. Relative position encoding appears to be favoured in different tasks and taxa. Generalization, though, was incomplete, since we observed a failure with items that during training had always been presented in reinforced arrays, showing the limitations in grasping the underlying positional rule. These results suggest the use of local strategies in the extraction of positional rules in cotton-top tamarins.

What’s special about orthographic processing? Further evidence from transposition effects in same-different matching

We sought evidence for letter-specific processing in the same-different matching task by comparing performance to random consonant strings and either strings of symbols (Experiment 1) or strings of digits (Experiment 2). The strings could be aligned horizontally or vertically, and on “different” response trials the to-be-matched strings could differ by the transposition of two adjacent characters or by the substitution of two adjacent characters. Making a “different” response was harder when the difference involved a transposition compared with a substitution—the transposition effect. Crucially, the transposition effect was significantly greater for letters than for symbols or digits when stimuli were aligned horizontally, but did not differ significantly across stimulus type with vertically aligned strings. These results suggest that it is processing specific to horizontally aligned letter strings, a reading-specific mechanism, that causes the greater transposition effects for letter stimuli in the same-different matching task when stimuli are arranged horizontally.

Reading comprehension and immersion schooling: evidence from component skills

The present research aims to assess literacy acquisition in children becoming bilingual via second language immersion in school. We adopt a cognitive components approach, assessing text-level reading comprehension, a complex literacy skill, as well as underlying cognitive and linguistic components in 144 children aged 7 to 14 (72 immersion bilinguals, 72 controls). Using principal component analysis, a nuanced pattern of results was observed: although emergent bilinguals lag behind their monolingual counterparts on measures of linguistic processing, they showed enhanced performance on a memory and reasoning component. For reading comprehension, no between-group differences were evident, suggesting that selective benefits compensate costs at the level of underlying cognitive components. Overall, the results seem to indicate that literacy skills may be modulated by emerging bilingualism even when no between-group differences are evident at the level of complex skill, and the detection of such differences may depend on the focus and selectivity of the task battery used.

Orthographic processing: A ‘mid-level’ vision of reading: The 44th Sir Frederic Bartlett Lecture

I will describe how orthographic processing acts as a central interface between visual and linguistic processing during reading, and as such can be considered to be the ‘mid-level vision’ of reading research. In order to make this case, I first summarize the evidence in favour of letter-based word recognition before examining work investigating how orthographic similarities among words influence single word reading. I describe how evidence gradually accumulated against traditional measures of orthographic similarity and the associated theories of orthographic processing, forcing a reconsideration of how letter-position information is represented by skilled readers. Then, I present the theoretical framework that was developed to explain these findings, with a focus on the distinction between location-specific and location-invariant orthographic representations. Finally, I describe work extending this theoretical framework in two main directions: first, to the realm of reading development, with the aim to specify the key changes in the processing of letters and letter strings that accompany successful learning to read, and second, to the realm of sentence reading, in order to specify how orthographic information can be processed across several words in parallel, and how skilled readers keep track of which letters belong to which words.

Are baboons learning "orthographic" representations? Probably not

The ability of Baboons (papio papio) to distinguish between English words and nonwords has been modeled using a deep learning convolutional network model that simulates a ventral pathway in which lexical representations of different granularity develop. However, given that pigeons (columba livia), whose brain morphology is drastically different, can also be trained to distinguish between English words and nonwords, it appears that a less species-specific learning algorithm may be required to explain this behavior. Accordingly, we examined whether the learning model of Rescorla and Wagner, which has proved to be amazingly fruitful in understanding animal and human learning could account for these data. We show that a discrimination learning network using gradient orientation features as input units and word and nonword units as outputs succeeds in predicting baboon lexical decision behavior-including key lexical similarity effects and the ups and downs in accuracy as learning unfolds-with surprising precision. The models performance, in which words are not explicitly represented, is remarkable because it is usually assumed that lexicality decisions, including the decisions made by baboons and pigeons, are mediated by explicit lexical representations. By contrast, our results suggest that in learning to perform lexical decision tasks, baboons and pigeons do not construct a hierarchy of lexical units. Rather, they make optimal use of low-level information obtained through the massively parallel processing of gradient orientation features. Accordingly, we suggest that reading in humans first involves initially learning a high-level system building on letter representations acquired from explicit instruction in literacy, which is then integrated into a conventionalized oral communication system, and that like the latter, fluent reading involves the massively parallel processing of the low-level features encoding semantic contrasts.

Orthographic and phonological processing in developing readers revealed by ERPs

The development of neurocognitive mechanisms in single word reading was studied in children ages 8-10 years using ERPs combined with priming manipulations aimed at dissociating orthographic and phonological processes. Transposed-letter (TL) priming (barin-BRAIN vs. bosin-BRAIN) was used to assess orthographic processing, and pseudohomophone (PH) priming (brane-BRAIN vs. brant-BRAIN) was used to assess phonological processing. Children showed TL and PH priming effects on both the N250 and N400 ERP components, and the magnitude of TL priming correlated positively with reading ability, with better readers showing larger TL priming effects. Phonological priming, on the other hand, did not correlate with reading ability. The positive correlations between TL priming and reading ability in children points to a key role for flexible sublexical orthographic representations in reading development, in line with their hypothesized role in the efficient mapping of orthographic information onto semantic information in skilled readers.

Orthographic processing in pigeons (Columba livia)

Learning to read involves the acquisition of letter-sound relationships (i.e., decoding skills) and the ability to visually recognize words (i.e., orthographic knowledge). Although decoding skills are clearly human-unique, given they are seated in language, recent research and theory suggest that orthographic processing may derive from the exaptation or recycling of visual circuits that evolved to recognize everyday objects and shapes in our natural environment. An open question is whether orthographic processing is limited to visual circuits that are similar to our own or a product of plasticity common to many vertebrate visual systems. Here we show that pigeons, organisms that separated from humans more than 300 million y ago, process words orthographically. Specifically, we demonstrate that pigeons trained to discriminate words from nonwords picked up on the orthographic properties that define words and used this knowledge to identify words they had never seen before. In addition, the pigeons were sensitive to the bigram frequencies of words (i.e., the common co-occurrence of certain letter pairs), the edit distance between nonwords and words, and the internal structure of words. Our findings demonstrate that visual systems organizationally distinct from the primate visual system can also be exapted or recycled to process the visual word form.

A vision of graded hemispheric specialization

Understanding the process by which the cerebral hemispheres reach their mature functional organization remains challenging. We propose a theoretical account in which, in the domain of vision, faces and words come to be represented adjacent to retinotopic cortex by virtue of the need to discriminate among homogeneous exemplars. Orthographic representations are further constrained to be proximal to typically left-lateralized language-related information to minimize connectivity length between visual and language areas. As reading is acquired, orthography comes to rely more heavily (albeit not exclusively) on the left fusiform region to bridge vision and language. Consequently, due to competition from emerging word representations, face representations that were initially bilateral become lateralized to the right fusiform region (albeit, again, not exclusively). We review recent research that describes constraints that give rise to this graded hemispheric arrangement. We then summarize empirical evidence from a variety of studies (behavioral, evoked response potential, functional imaging) across different populations (children, adolescents, and adults; left handers and individuals with developmental dyslexia) that supports the claims that hemispheric lateralization is graded rather than binary and that this graded organization emerges dynamically over the course of development. Perturbations of this system either during development or in adulthood provide further insights into the principles governing hemispheric organization.

Timing the impact of literacy on visual processing

Learning to read requires the acquisition of an efficient visual procedure for quickly recognizing fine print. Thus, reading practice could induce a perceptual learning effect in early vision. Using functional magnetic resonance imaging (fMRI) in literate and illiterate adults, we previously demonstrated an impact of reading acquisition on both high- and low-level occipitotemporal visual areas, but could not resolve the time course of these effects. To clarify whether literacy affects early vs. late stages of visual processing, we measured event-related potentials to various categories of visual stimuli in healthy adults with variable levels of literacy, including completely illiterate subjects, early-schooled literate subjects, and subjects who learned to read in adulthood (ex-illiterates). The stimuli included written letter strings forming pseudowords, on which literacy is expected to have a major impact, as well as faces, houses, tools, checkerboards, and false fonts. To evaluate the precision with which these stimuli were encoded, we studied repetition effects by presenting the stimuli in pairs composed of repeated, mirrored, or unrelated pictures from the same category. The results indicate that reading ability is correlated with a broad enhancement of early visual processing, including increased repetition suppression, suggesting better exemplar discrimination, and increased mirror discrimination, as early as ∼ 100-150 ms in the left occipitotemporal region. These effects were found with letter strings and false fonts, but also were partially generalized to other visual categories. Thus, learning to read affects the magnitude, precision, and invariance of early visual processing.

Orthographic Coding: Brain Activation for Letters, Symbols, and Digits

The present experiment investigates the input coding mechanisms of 3 common printed characters: letters, numbers, and symbols. Despite research in this area, it is yet unclear whether the identity of these 3 elements is processed through the same or different brain pathways. In addition, some computational models propose that the position-in-string coding of these elements responds to general flexible mechanisms of the visual system that are not character-specific, whereas others suggest that the position coding of letters responds to specific processes that are different from those that guide the position-in-string assignment of other types of visual objects. Here, in an fMRI study, we manipulated character position and character identity through the transposition or substitution of 2 internal elements within strings of 4 elements. Participants were presented with 2 consecutive visual strings and asked to decide whether they were the same or different. The results showed: 1) that some brain areas responded more to letters than to numbers and vice versa, suggesting that processing may follow different brain pathways; 2) that the left parietal cortex is involved in letter identity, and critically in letter position coding, specifically contributing to the early stages of the reading process; and that 3) a stimulus-specific mechanism for letter position coding is operating during orthographic processing.