1
|
Hsiao JHW. Understanding Human Cognition Through Computational Modeling. Top Cogn Sci 2024; 16:349-376. [PMID: 38781432 DOI: 10.1111/tops.12737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/02/2023] [Revised: 05/07/2024] [Accepted: 05/08/2024] [Indexed: 05/25/2024]
Abstract
One important goal of cognitive science is to understand the mind in terms of its representational and computational capacities, where computational modeling plays an essential role in providing theoretical explanations and predictions of human behavior and mental phenomena. In my research, I have been using computational modeling, together with behavioral experiments and cognitive neuroscience methods, to investigate the information processing mechanisms underlying learning and visual cognition in terms of perceptual representation and attention strategy. In perceptual representation, I have used neural network models to understand how the split architecture in the human visual system influences visual cognition, and to examine perceptual representation development as the results of expertise. In attention strategy, I have developed the Eye Movement analysis with Hidden Markov Models method for quantifying eye movement pattern and consistency using both spatial and temporal information, which has led to novel findings across disciplines not discoverable using traditional methods. By integrating it with deep neural networks (DNN), I have developed DNN+HMM to account for eye movement strategy learning in human visual cognition. The understanding of the human mind through computational modeling also facilitates research on artificial intelligence's (AI) comparability with human cognition, which can in turn help explainable AI systems infer humans' belief on AI's operations and provide human-centered explanations to enhance human-AI interaction and mutual understanding. Together, these demonstrate the essential role of computational modeling methods in providing theoretical accounts of the human mind as well as its interaction with its environment and AI systems.
Collapse
|
2
|
Liao W, Li STK, Hsiao JHW. Music reading experience modulates eye movement pattern in English reading but not in Chinese reading. Sci Rep 2022; 12:9144. [PMID: 35650229 PMCID: PMC9397380 DOI: 10.1038/s41598-022-12978-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2022] [Accepted: 05/19/2022] [Indexed: 11/30/2022] Open
Abstract
Here we tested the hypothesis that in Chinese-English bilinguals,
music reading experience may modulate eye movement planning in reading English but
not Chinese sentences due to the similarity in perceptual demands on processing
sequential symbol strings separated by spaces between music notation and English
sentence reading. Chinese–English bilingual musicians and non-musicians read legal,
semantically incorrect, and syntactically (and semantically) incorrect sentences in
both English and Chinese. In English reading, musicians showed more dispersed eye
movement patterns in reading syntactically incorrect sentences than legal sentences,
whereas non-musicians did not. This effect was not observed in Chinese reading.
Musicians also had shorter saccade lengths when viewing syntactically incorrect than
correct musical notations and sentences in an unfamiliar alphabetic language
(Tibetan), whereas non-musicians did not. Thus, musicians’ eye movement planning was
disturbed by syntactic violations in both music and English reading but not in
Chinese reading, and this effect was generalized to an unfamiliar alphabetic
language. These results suggested that music reading experience may modulate
perceptual processes in reading differentially in bilinguals’ two languages,
depending on their processing similarities.
Collapse
Affiliation(s)
- Weiyan Liao
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China
| | - Sara Tze Kwan Li
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China.,Department of Social Sciences, School of Arts and Social Sciences, Hong Kong Metropolitan University, Hong Kong SAR, China
| | - Janet Hui-Wen Hsiao
- Department of Psychology, University of Hong Kong, Hong Kong SAR, China. .,The State Key Laboratory of Brain and Cognitive Sciences, University of Hong Kong, Hong Kong SAR, China. .,The Institute of Data Science, University of Hong Kong, Hong Kong SAR, China.
| |
Collapse
|
3
|
Liu X, Vermeylen L, Wisniewski D, Brysbaert M. The contribution of phonological information to visual word recognition: Evidence from Chinese phonetic radicals. Cortex 2020; 133:48-64. [PMID: 33099075 DOI: 10.1016/j.cortex.2020.09.010] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2020] [Revised: 07/09/2020] [Accepted: 09/17/2020] [Indexed: 11/28/2022]
Abstract
Lateralization is a critical characteristic of language production and also plays a role in visual word recognition. However, the neural mechanisms underlying the interactions between visual input and spoken word representations are still unclear. We investigated the contribution of sub-lexical phonological information in visual word processing by exploiting the fact that Chinese characters can contain phonetic radicals in either the left or right half of the character. FMRI data were collected while 39 Chinese participants read words in search of target color words. On the basis of whole-brain analysis and three laterality analyses of regions of interest, we argue that visual information from centrally presented Chinese characters is split in the fovea and projected to the contralateral visual cortex, from which phonological information can be extracted rapidly if the character contains a phonetic radical. Extra activation, suggestive of more effortful processing, is observed when the phonetic radical is situated in the left half of the character and therefore initially sent to the visual cortex in the right hemisphere that is less specialized for language processing. Our results are in line with the proposal that phonological information helps written word processing by means of top-down feedback.
Collapse
Affiliation(s)
- Xiaodong Liu
- Department of Experimental Psychology, Ghent University, Belgium.
| | - Luc Vermeylen
- Department of Experimental Psychology, Ghent University, Belgium
| | - David Wisniewski
- Department of Experimental Psychology, Ghent University, Belgium
| | - Marc Brysbaert
- Department of Experimental Psychology, Ghent University, Belgium
| |
Collapse
|
4
|
Li STK, Hsiao JHW. Music reading expertise modulates hemispheric lateralization in English word processing but not in Chinese character processing. Cognition 2018; 176:159-173. [PMID: 29558721 DOI: 10.1016/j.cognition.2018.03.010] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2016] [Revised: 03/07/2018] [Accepted: 03/08/2018] [Indexed: 10/17/2022]
Abstract
Music notation and English word reading both involve mapping horizontally arranged visual components to components in sound, in contrast to reading in logographic languages such as Chinese. Accordingly, music-reading expertise may influence English word processing more than Chinese character processing. Here we showed that musicians named English words significantly faster than non-musicians when words were presented in the left visual field/right hemisphere (RH) or the center position, suggesting an advantage of RH processing due to music reading experience. This effect was not observed in Chinese character naming. A follow-up ERP study showed that in a sequential matching task, musicians had reduced RH N170 responses to English non-words under the processing of musical segments as compared with non-musicians, suggesting a shared visual processing mechanism in the RH between music notation and English non-word reading. This shared mechanism may be related to the letter-by-letter, serial visual processing that characterizes RH English word recognition (e.g., Lavidor & Ellis, 2001), which may consequently facilitate English word processing in the RH in musicians. Thus, music reading experience may have differential influences on the processing of different languages, depending on their similarities in the cognitive processes involved.
Collapse
Affiliation(s)
- Sara Tze Kwan Li
- Department of Psychology, University of Hong Kong, Hong Kong Special Administrative Region
| | - Janet Hui-Wen Hsiao
- Department of Psychology, University of Hong Kong, Hong Kong Special Administrative Region.
| |
Collapse
|
5
|
Wang X, Pei M, Wu Y, Su Y. Semantic Radicals Contribute More Than Phonetic Radicals to the Recognition of Chinese Phonograms: Behavioral and ERP Evidence in a Factorial Study. Front Psychol 2017; 8:2230. [PMID: 29312076 PMCID: PMC5742193 DOI: 10.3389/fpsyg.2017.02230] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2017] [Accepted: 12/08/2017] [Indexed: 11/13/2022] Open
Abstract
The Chinese phonograms consist of a semantic radical and a phonetic radical. The two types of radicals have different functional contributions to their host phonogram. The semantic radical typically signifies the meaning of the phonogram, while the phonetic radical usually contains a phonological clue to the phonogram's pronunciation. However, it is still unclear how they interplay with each other when we attempt to recognize a phonogram because previous studies rarely manipulated the functionality of the two types of radicals in a single design. Using a full factorial design, the present study aimed to probe this issue by directly manipulating the functional validity of the two types of radicals in a lexical decision task with both behavioral and event-related potential (ERP) measurements. The results showed that recognition of phonograms which were related to their semantic radicals in meaning took a shorter reaction time, showed a lower error rate, and elicited a smaller P200 and a larger N400 than did recognition of those which had no semantic relation with their semantic radicals. However, the validity of phonetic radicals did not show any main effect or interaction with that of semantic radicals on either behavioral or ERP measurements. These results indicated that semantic radicals played a dominant role in the recognition of phonograms. Transparent semantic radicals, which provide valid semantic cues to phonograms, can facilitate the recognition of phonograms.
Collapse
Affiliation(s)
- Xieshun Wang
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China
| | - Meng Pei
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China
| | - Yan Wu
- School of Psychology, Northeast Normal University, Changchun, China
| | - Yanjie Su
- School of Psychological and Cognitive Sciences, Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China
| |
Collapse
|
6
|
Wang X, Yang J, Yang J, Mencl WE, Shu H, Zevin JD. Language differences in the brain network for reading in naturalistic story reading and lexical decision. PLoS One 2015; 10:e0124388. [PMID: 26017384 PMCID: PMC4446262 DOI: 10.1371/journal.pone.0124388] [Citation(s) in RCA: 23] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2014] [Accepted: 03/13/2015] [Indexed: 11/18/2022] Open
Abstract
Differences in how writing systems represent language raise important questions about whether there could be a universal functional architecture for reading across languages. In order to study potential language differences in the neural networks that support reading skill, we collected fMRI data from readers of alphabetic (English) and morpho-syllabic (Chinese) writing systems during two reading tasks. In one, participants read short stories under conditions that approximate natural reading, and in the other, participants decided whether individual stimuli were real words or not. Prior work comparing these two writing systems has overwhelmingly used meta-linguistic tasks, generally supporting the conclusion that the reading system is organized differently for skilled readers of Chinese and English. We observed that language differences in the reading network were greatly dependent on task. In lexical decision, a pattern consistent with prior research was observed in which the Middle Frontal Gyrus (MFG) and right Fusiform Gyrus (rFFG) were more active for Chinese than for English, whereas the posterior temporal sulcus was more active for English than for Chinese. We found a very different pattern of language effects in a naturalistic reading paradigm, during which significant differences were only observed in visual regions not typically considered specific to the reading network, and the middle temporal gyrus, which is thought to be important for direct mapping of orthography to semantics. Indeed, in areas that are often discussed as supporting distinct cognitive or linguistic functions between the two languages, we observed interaction. Specifically, language differences were most pronounced in MFG and rFFG during the lexical decision task, whereas no language differences were observed in these areas during silent reading of text for comprehension.
Collapse
Affiliation(s)
- Xiaojuan Wang
- Key Lab for Behavior & Cognitive Neuroscience of Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi’an, China
| | - Jianfeng Yang
- Key Lab for Behavior & Cognitive Neuroscience of Shaanxi Province, School of Psychology, Shaanxi Normal University, Xi’an, China
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- * E-mail: (JDZ); (JY)
| | - Jie Yang
- Department of Neurology, University of California Irvine, Irvine, California, United States of America
| | - W. Einar Mencl
- Haskins Laboratories, Yale University, New Haven, CT, United States of America
| | - Hua Shu
- State Key laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Beijing, China
| | - Jason David Zevin
- Institute of Psychology, Chinese Academy of Sciences, Beijing, China
- Haskins Laboratories, Yale University, New Haven, CT, United States of America
- Departments of Psychology and Linguistics, University of Southern California, Los Angeles, CA, United States of America
- * E-mail: (JDZ); (JY)
| |
Collapse
|
7
|
Hsiao JH, Cheung K. Visual Similarity of Words Alone Can Modulate Hemispheric Lateralization in Visual Word Recognition: Evidence From Modeling Chinese Character Recognition. Cogn Sci 2015; 40:351-72. [PMID: 25818722 DOI: 10.1111/cogs.12233] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2013] [Revised: 11/05/2014] [Accepted: 12/19/2014] [Indexed: 11/27/2022]
Abstract
In Chinese orthography, the most common character structure consists of a semantic radical on the left and a phonetic radical on the right (SP characters); the minority, opposite arrangement also exists (PS characters). Recent studies showed that SP character processing is more left hemisphere (LH) lateralized than PS character processing. Nevertheless, it remains unclear whether this is due to phonetic radical position or character type frequency. Through computational modeling with artificial lexicons, in which we implement a theory of hemispheric asymmetry in perception but do not assume phonological processing being LH lateralized, we show that the difference in character type frequency alone is sufficient to exhibit the effect that the dominant type has a stronger LH lateralization than the minority type. This effect is due to higher visual similarity among characters in the dominant type than the minority type, demonstrating the modulation of visual similarity of words on hemispheric lateralization.
Collapse
Affiliation(s)
| | - Kit Cheung
- Department of Computing, Imperial College London
| |
Collapse
|
8
|
Hsu CH, Lee CY, Tzeng OJL. Early MEG markers for reading Chinese phonograms: evidence from radical combinability and consistency effects. BRAIN AND LANGUAGE 2014; 139:1-9. [PMID: 25463812 DOI: 10.1016/j.bandl.2014.09.008] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2013] [Revised: 08/15/2014] [Accepted: 09/22/2014] [Indexed: 06/04/2023]
Abstract
Studies using functional magnetic resonance imaging have indicated that activities in the left inferior frontal cortex and left temporoparietal regions are associated with orthographic neighborhood size. To elucidate the temporal dynamics of reading-related cortical activities, we manipulated two types of neighborhood properties for Chinese phonograms, phonetic combinability and consistency. By using source analysis techniques in combination with magnetoencephalography, the results demonstrated a combinability effect in the right fusiform gyrus at ∼ 170 ms, which may reflect perceptual expertise in processing Chinese orthography. During 200 ms to 250 ms, the left anterior insula showed larger activity in reading small combinability characters than in reading large combinability characters, and the left inferior parietal cortex showed greater activity in reading low consistency characters than in reading high consistency characters. These results indicate that the left anterior insula cortex and left inferior parietal cortex may play important roles in the early stages of reading Chinese phonograms.
Collapse
Affiliation(s)
- Chun-Hsien Hsu
- Institute of Linguistics, Academia Sinica, No. 128, Section 2, Academia Road, 115 Taipei, Taiwan.
| | - Chia-Ying Lee
- Institute of Linguistics, Academia Sinica, No. 128, Section 2, Academia Road, 115 Taipei, Taiwan; Institute of Neuroscience, National Yang-Ming University, No. 155, Section 2, Linong Street, 112 Taipei, Taiwan; Institute of Cognitive Neuroscience, National Central University, No. 300, Jhongda Rd., Jhongli City, Taoyuan County 32001, Taiwan
| | - Ovid J-L Tzeng
- Institute of Linguistics, Academia Sinica, No. 128, Section 2, Academia Road, 115 Taipei, Taiwan; Institute of Neuroscience, National Yang-Ming University, No. 155, Section 2, Linong Street, 112 Taipei, Taiwan; Department of Biological Science and Technology, National Chiao Tung University, 75 Bo-Ai Street, Hsin-Chu, Taiwan
| |
Collapse
|
9
|
Hsiao JH, Lam SM. The Modulation of Visual and Task Characteristics of a Writing System on Hemispheric Lateralization in Visual Word Recognition-A Computational Exploration. Cogn Sci 2013; 37:861-90. [DOI: 10.1111/cogs.12033] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2012] [Revised: 06/19/2012] [Accepted: 08/23/2012] [Indexed: 11/28/2022]
Affiliation(s)
| | - Sze Man Lam
- Department of Psychology; University of Hong Kong
| |
Collapse
|
10
|
Hsiao JH, Cheng L. The modulation of stimulus structure on visual field asymmetry effects: the case of Chinese character recognition. Q J Exp Psychol (Hove) 2013; 66:1739-55. [PMID: 23391072 DOI: 10.1080/17470218.2013.764902] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/27/2022]
Abstract
Recent research suggests that visual field (VF) asymmetry effects in visual recognition may be influenced by information distribution within the stimuli for the recognition task in addition to hemispheric processing differences: Stimuli with more information on the left have a right VF (RVF) advantage because the left part is closer to the centre, where the highest visual acuity is obtained. It remains unclear whether visual complexity distribution of the stimuli also has similar modulation effects. Here we used Chinese characters with contrasting structures-left-heavy, symmetric, and right-heavy, in terms of either visual complexity of components or information distribution defined by location of the phonetic component-and examined participants' naming performance. We found that left-heavy characters had the largest RVF advantage, followed by symmetric and right-heavy characters; this effect was only observed in characters that contrasted in information distribution, in which information for pronunciation was skewed to the phonetic component, but not in those that contrasted only in visual complexity distribution and had no phonetic component. This result provides strong evidence for the influence of information distribution within the stimuli on VF asymmetry effects; in contrast, visual complexity distribution within the stimuli does not have similar modulation effects.
Collapse
Affiliation(s)
- Janet H Hsiao
- a Department of Psychology , University of Hong Kong , Hong Kong
| | | |
Collapse
|