Position of phonetic components may influence how written words are processed in the brain: Evidence from Chinese phonetic compound pronunciation

Understanding Human Cognition Through Computational Modeling

One important goal of cognitive science is to understand the mind in terms of its representational and computational capacities, where computational modeling plays an essential role in providing theoretical explanations and predictions of human behavior and mental phenomena. In my research, I have been using computational modeling, together with behavioral experiments and cognitive neuroscience methods, to investigate the information processing mechanisms underlying learning and visual cognition in terms of perceptual representation and attention strategy. In perceptual representation, I have used neural network models to understand how the split architecture in the human visual system influences visual cognition, and to examine perceptual representation development as the results of expertise. In attention strategy, I have developed the Eye Movement analysis with Hidden Markov Models method for quantifying eye movement pattern and consistency using both spatial and temporal information, which has led to novel findings across disciplines not discoverable using traditional methods. By integrating it with deep neural networks (DNN), I have developed DNN+HMM to account for eye movement strategy learning in human visual cognition. The understanding of the human mind through computational modeling also facilitates research on artificial intelligence's (AI) comparability with human cognition, which can in turn help explainable AI systems infer humans' belief on AI's operations and provide human-centered explanations to enhance human-AI interaction and mutual understanding. Together, these demonstrate the essential role of computational modeling methods in providing theoretical accounts of the human mind as well as its interaction with its environment and AI systems.

Music reading experience modulates eye movement pattern in English reading but not in Chinese reading

Here we tested the hypothesis that in Chinese-English bilinguals, music reading experience may modulate eye movement planning in reading English but not Chinese sentences due to the similarity in perceptual demands on processing sequential symbol strings separated by spaces between music notation and English sentence reading. Chinese–English bilingual musicians and non-musicians read legal, semantically incorrect, and syntactically (and semantically) incorrect sentences in both English and Chinese. In English reading, musicians showed more dispersed eye movement patterns in reading syntactically incorrect sentences than legal sentences, whereas non-musicians did not. This effect was not observed in Chinese reading. Musicians also had shorter saccade lengths when viewing syntactically incorrect than correct musical notations and sentences in an unfamiliar alphabetic language (Tibetan), whereas non-musicians did not. Thus, musicians’ eye movement planning was disturbed by syntactic violations in both music and English reading but not in Chinese reading, and this effect was generalized to an unfamiliar alphabetic language. These results suggested that music reading experience may modulate perceptual processes in reading differentially in bilinguals’ two languages, depending on their processing similarities.

The contribution of phonological information to visual word recognition: Evidence from Chinese phonetic radicals

Lateralization is a critical characteristic of language production and also plays a role in visual word recognition. However, the neural mechanisms underlying the interactions between visual input and spoken word representations are still unclear. We investigated the contribution of sub-lexical phonological information in visual word processing by exploiting the fact that Chinese characters can contain phonetic radicals in either the left or right half of the character. FMRI data were collected while 39 Chinese participants read words in search of target color words. On the basis of whole-brain analysis and three laterality analyses of regions of interest, we argue that visual information from centrally presented Chinese characters is split in the fovea and projected to the contralateral visual cortex, from which phonological information can be extracted rapidly if the character contains a phonetic radical. Extra activation, suggestive of more effortful processing, is observed when the phonetic radical is situated in the left half of the character and therefore initially sent to the visual cortex in the right hemisphere that is less specialized for language processing. Our results are in line with the proposal that phonological information helps written word processing by means of top-down feedback.

Music reading expertise modulates hemispheric lateralization in English word processing but not in Chinese character processing

Music notation and English word reading both involve mapping horizontally arranged visual components to components in sound, in contrast to reading in logographic languages such as Chinese. Accordingly, music-reading expertise may influence English word processing more than Chinese character processing. Here we showed that musicians named English words significantly faster than non-musicians when words were presented in the left visual field/right hemisphere (RH) or the center position, suggesting an advantage of RH processing due to music reading experience. This effect was not observed in Chinese character naming. A follow-up ERP study showed that in a sequential matching task, musicians had reduced RH N170 responses to English non-words under the processing of musical segments as compared with non-musicians, suggesting a shared visual processing mechanism in the RH between music notation and English non-word reading. This shared mechanism may be related to the letter-by-letter, serial visual processing that characterizes RH English word recognition (e.g., Lavidor & Ellis, 2001), which may consequently facilitate English word processing in the RH in musicians. Thus, music reading experience may have differential influences on the processing of different languages, depending on their similarities in the cognitive processes involved.

Semantic Radicals Contribute More Than Phonetic Radicals to the Recognition of Chinese Phonograms: Behavioral and ERP Evidence in a Factorial Study

The Chinese phonograms consist of a semantic radical and a phonetic radical. The two types of radicals have different functional contributions to their host phonogram. The semantic radical typically signifies the meaning of the phonogram, while the phonetic radical usually contains a phonological clue to the phonogram's pronunciation. However, it is still unclear how they interplay with each other when we attempt to recognize a phonogram because previous studies rarely manipulated the functionality of the two types of radicals in a single design. Using a full factorial design, the present study aimed to probe this issue by directly manipulating the functional validity of the two types of radicals in a lexical decision task with both behavioral and event-related potential (ERP) measurements. The results showed that recognition of phonograms which were related to their semantic radicals in meaning took a shorter reaction time, showed a lower error rate, and elicited a smaller P200 and a larger N400 than did recognition of those which had no semantic relation with their semantic radicals. However, the validity of phonetic radicals did not show any main effect or interaction with that of semantic radicals on either behavioral or ERP measurements. These results indicated that semantic radicals played a dominant role in the recognition of phonograms. Transparent semantic radicals, which provide valid semantic cues to phonograms, can facilitate the recognition of phonograms.

Language differences in the brain network for reading in naturalistic story reading and lexical decision

Differences in how writing systems represent language raise important questions about whether there could be a universal functional architecture for reading across languages. In order to study potential language differences in the neural networks that support reading skill, we collected fMRI data from readers of alphabetic (English) and morpho-syllabic (Chinese) writing systems during two reading tasks. In one, participants read short stories under conditions that approximate natural reading, and in the other, participants decided whether individual stimuli were real words or not. Prior work comparing these two writing systems has overwhelmingly used meta-linguistic tasks, generally supporting the conclusion that the reading system is organized differently for skilled readers of Chinese and English. We observed that language differences in the reading network were greatly dependent on task. In lexical decision, a pattern consistent with prior research was observed in which the Middle Frontal Gyrus (MFG) and right Fusiform Gyrus (rFFG) were more active for Chinese than for English, whereas the posterior temporal sulcus was more active for English than for Chinese. We found a very different pattern of language effects in a naturalistic reading paradigm, during which significant differences were only observed in visual regions not typically considered specific to the reading network, and the middle temporal gyrus, which is thought to be important for direct mapping of orthography to semantics. Indeed, in areas that are often discussed as supporting distinct cognitive or linguistic functions between the two languages, we observed interaction. Specifically, language differences were most pronounced in MFG and rFFG during the lexical decision task, whereas no language differences were observed in these areas during silent reading of text for comprehension.

Visual Similarity of Words Alone Can Modulate Hemispheric Lateralization in Visual Word Recognition: Evidence From Modeling Chinese Character Recognition

In Chinese orthography, the most common character structure consists of a semantic radical on the left and a phonetic radical on the right (SP characters); the minority, opposite arrangement also exists (PS characters). Recent studies showed that SP character processing is more left hemisphere (LH) lateralized than PS character processing. Nevertheless, it remains unclear whether this is due to phonetic radical position or character type frequency. Through computational modeling with artificial lexicons, in which we implement a theory of hemispheric asymmetry in perception but do not assume phonological processing being LH lateralized, we show that the difference in character type frequency alone is sufficient to exhibit the effect that the dominant type has a stronger LH lateralization than the minority type. This effect is due to higher visual similarity among characters in the dominant type than the minority type, demonstrating the modulation of visual similarity of words on hemispheric lateralization.

Early MEG markers for reading Chinese phonograms: evidence from radical combinability and consistency effects

Studies using functional magnetic resonance imaging have indicated that activities in the left inferior frontal cortex and left temporoparietal regions are associated with orthographic neighborhood size. To elucidate the temporal dynamics of reading-related cortical activities, we manipulated two types of neighborhood properties for Chinese phonograms, phonetic combinability and consistency. By using source analysis techniques in combination with magnetoencephalography, the results demonstrated a combinability effect in the right fusiform gyrus at ∼ 170 ms, which may reflect perceptual expertise in processing Chinese orthography. During 200 ms to 250 ms, the left anterior insula showed larger activity in reading small combinability characters than in reading large combinability characters, and the left inferior parietal cortex showed greater activity in reading low consistency characters than in reading high consistency characters. These results indicate that the left anterior insula cortex and left inferior parietal cortex may play important roles in the early stages of reading Chinese phonograms.

The modulation of stimulus structure on visual field asymmetry effects: the case of Chinese character recognition

Recent research suggests that visual field (VF) asymmetry effects in visual recognition may be influenced by information distribution within the stimuli for the recognition task in addition to hemispheric processing differences: Stimuli with more information on the left have a right VF (RVF) advantage because the left part is closer to the centre, where the highest visual acuity is obtained. It remains unclear whether visual complexity distribution of the stimuli also has similar modulation effects. Here we used Chinese characters with contrasting structures-left-heavy, symmetric, and right-heavy, in terms of either visual complexity of components or information distribution defined by location of the phonetic component-and examined participants' naming performance. We found that left-heavy characters had the largest RVF advantage, followed by symmetric and right-heavy characters; this effect was only observed in characters that contrasted in information distribution, in which information for pronunciation was skewed to the phonetic component, but not in those that contrasted only in visual complexity distribution and had no phonetic component. This result provides strong evidence for the influence of information distribution within the stimuli on VF asymmetry effects; in contrast, visual complexity distribution within the stimuli does not have similar modulation effects.