Quasiregularity and Its Discontents: The Legacy of the Past Tense Debate

Exploring Abstract Pattern Representation in The Brain and Non-symbolic Neural Networks

Human cognitive and linguistic generativity depends on the ability to identify abstract relationships between perceptually dissimilar items. Marcus et al. (1999) found that human infants can rapidly discover and generalize patterns of syllable repetition (reduplication) that depend on the abstract property of identity, but simple recurrent neural networks (SRNs) could not. They interpreted these results as evidence that purely associative neural network models provide an inadequate framework for characterizing the fundamental generativity of human cognition. Here, we present a series of deep long short-term memory (LSTM) models that identify abstract syllable repetition patterns and words based on training with cochleagrams that represent auditory stimuli. We demonstrate that models trained to identify individual syllable trigram words and models trained to identify reduplication patterns discover representations that support classification of abstract repetition patterns. Simulations examined the effects of training categories (words vs. patterns) and pretraining to identify syllables, on the development of hidden node representations that support repetition pattern discrimination. Representational similarity analyses (RSA) comparing patterns of regional brain activity based on MRI-constrained MEG/EEG data to patterns of hidden node activation elicited by the same stimuli showed a significant correlation between brain activity localized in primarily posterior temporal regions and representations discovered by the models. These results suggest that associative mechanisms operating over discoverable representations that capture abstract stimulus properties account for a critical example of human cognitive generativity.

Abstract representations in temporal cortex support generative linguistic processing

Generativity, the ability to create and evaluate novel constructions, is a fundamental property of human language and cognition. The productivity of generative processes is determined by the scope of the representations they engage. Here we examine the neural representation of reduplication, a productive phonological process that can create novel forms through patterned syllable copying (e.g. ba-mih → ba-ba-mih, ba-mih-mih, or ba-mih-ba). Using MRI-constrained source estimates of combined MEG/EEG data collected during an auditory artificial grammar task, we identified localized cortical activity associated with syllable reduplication pattern contrasts in novel trisyllabic nonwords. Neural decoding analyses identified a set of predominantly right hemisphere temporal lobe regions whose activity reliably discriminated reduplication patterns evoked by untrained, novel stimuli. Effective connectivity analyses suggested that sensitivity to abstracted reduplication patterns was propagated between these temporal regions. These results suggest that localized temporal lobe activity patterns function as abstract representations that support linguistic generativity.

Acquisition of orthographic forms via spoken complex word training

This study used a novel word-training paradigm to examine the integration of spoken word knowledge when learning to read morphologically complex novel words. Australian primary school children including Grades 3-5 were taught the oral form of a set of novel morphologically complex words (e.g., (/vɪbɪŋ/, /vɪbd/, /vɪbz/), with a second set serving as untrained items. Following oral training, participants saw the printed form of the novel word stems for the first time (e.g., vib), embedded in sentences, while their eye movements were monitored. Half of the stems were spelled predictably and half were spelled unpredictably. Reading times were shorter for orally trained stems with predictable than unpredictable spellings and this difference was greater for trained than untrained items. These findings suggest that children were able to form robust orthographic expectations of the embedded morphemic stems during spoken word learning, which may have occurred automatically without any explicit control of the applied mappings, despite still being in the early stages of reading development. Following the sentence reading task, children completed a reading-aloud task where they were exposed to the novel orthographic forms for a second time. The findings are discussed in the context of theories of reading acquisition.

From decomposition to distributed theories of morphological processing in reading

The morphological structure of complex words impacts how they are processed during visual word recognition. This impact varies over the course of reading acquisition and for different languages and writing systems. Many theories of morphological processing rely on a decomposition mechanism, in which words are decomposed into explicit representations of their constituent morphemes. In distributed accounts, in contrast, morphological sensitivity arises from the tuning of finer-grained representations to useful statistical regularities in the form-to-meaning mapping, without the need for explicit morpheme representations. In this theoretically guided review, we summarize research into the mechanisms of morphological processing, and discuss findings within the context of decomposition and distributed accounts. Although many findings fit within a decomposition model of morphological processing, we suggest that the full range of results is more naturally explained by a distributed approach, and discuss additional benefits of adopting this perspective.

How children learn to communicate discriminatively

How do children learn to communicate, and what do they learn? Traditionally, most theories have taken an associative, compositional approach to these questions, supposing children acquire an inventory of form-meaning associations, and procedures for composing / decomposing them; into / from messages in production and comprehension. This paper presents an alternative account of human communication and its acquisition based on the systematic, discriminative approach embodied in psychological and computational models of learning, and formally described by communication theory. It describes how discriminative learning theory offers an alternative perspective on the way that systems of semantic cues are conditioned onto communicative codes, while information theory provides a very different view of the nature of the codes themselves. It shows how the distributional properties of languages satisfy the communicative requirements described in information theory, enabling language learners to align their expectations despite the vastly different levels of experience among language users, and to master communication systems far more abstract than linguistic intuitions traditionally assume. Topics reviewed include morphological development, the acquisition of verb argument structures, and the functions of linguistic systems that have proven to be stumbling blocks for compositional theories: grammatical gender and personal names.

Language Usage and Second Language Morphosyntax: Effects of Availability, Reliability, and Formulaicity

A large body of psycholinguistic research demonstrates that both language processing and language acquisition are sensitive to the distributions of linguistic constructions in usage. Here we investigate how statistical distributions at different linguistic levels - morphological and lexical (Experiments 1 and 2), and phrasal (Experiment 2) - contribute to the ease with which morphosyntax is processed and produced by second language learners. We analyze Chinese ESL learners' knowledge of four English inflectional morphemes: -ed, -ing, and third-person -s on verbs, and plural -s on nouns. In Elicited Imitation Tasks, participants listened to length- and difficulty-matched sentences each containing one target morpheme and typed the whole sentence as accurately as they could after a short delay. Experiment 1 investigated lexical and morphemic levels, testing the hypotheses that a morpheme is expected to be more easily processed when it is (1) highly available (i.e., occurring in frequent word-forms), and (2) highly reliable (i.e., occurring in lemma words that are consistently conjugated in the form containing this morpheme). Thirty sentences were made for each morpheme, divided into three Availability-Reliability Distribution (ARD) groups on the basis of corpus analysis in the Corpus of Contemporary American English (COCA; Davies, 2008-): 10 target words high in availability, 10 high in reliability, and 10 low in both reliability and availability. Responses were scored on whether the target morpheme was accurately reproduced given the provision of the correct lemma. A generalized linear mixed-effects logit model (GLMM) revealed fixed effects of morpheme type, availability, and reliability on the accuracy of morpheme provision. There were no effects of lemma frequency. Experiment 2 successfully replicated these results and extended the investigation to explore phrasal formulaicity by manipulating the frequency of the four-word strings in which the morpheme was embedded. GLMMs replicated the effects of word-form availability and reliability and additionally revealed independent phrase-superiority effects where morphemes were better reproduced in contexts of higher string-frequency. Taken together, these findings demonstrate that morpheme acquisition reflects the distributional properties of learners' experience and the mappings therein between lexis, morphology, phraseology, and semantics. These conclusions support an emergentist view of the statistical symbolic learning of morphology where language acquisition involves the satisfaction of competing constraints across multiple grain-sizes of units.

Placing language in an integrated understanding system: Next steps toward human-level performance in neural language models

Language is crucial for human intelligence, but what exactly is its role? We take language to be a part of a system for understanding and communicating about situations. In humans, these abilities emerge gradually from experience and depend on domain-general principles of biological neural networks: connection-based learning, distributed representation, and context-sensitive, mutual constraint satisfaction-based processing. Current artificial language processing systems rely on the same domain general principles, embodied in artificial neural networks. Indeed, recent progress in this field depends on query-based attention, which extends the ability of these systems to exploit context and has contributed to remarkable breakthroughs. Nevertheless, most current models focus exclusively on language-internal tasks, limiting their ability to perform tasks that depend on understanding situations. These systems also lack memory for the contents of prior situations outside of a fixed contextual span. We describe the organization of the brain's distributed understanding system, which includes a fast learning system that addresses the memory problem. We sketch a framework for future models of understanding drawing equally on cognitive neuroscience and artificial intelligence and exploiting query-based attention. We highlight relevant current directions and consider further developments needed to fully capture human-level language understanding in a computational system.

Neural Population Dynamics and Cognitive Function

Representations in the brain are encoded as patterns of activity of large populations of neurons. The science of population encoded representations, also known as parallel distributed processing (PDP), achieves neurological verisimilitude and has been able to account for a large number of cognitive phenomena in normal people, including reaction times (and reading latencies), stimulus recognition, the effect of stimulus salience on attention, perceptual invariance, simultaneous egocentric and allocentric visual processing, top-down/bottom-up processing, language errors, the effect of statistical regularities of experience, frequency, and age of acquisition, instantiation of rules and symbols, content addressable memory and the capacity for pattern completion, preservation of function in the face of noisy or distorted input, inference, parallel constraint satisfaction, the binding problem and gamma coherence, principles of hippocampal function, the location of knowledge in the brain, limitations in the scope and depth of knowledge acquired through experience, and Piagetian stages of cognitive development. PDP studies have been able to provide a coherent account for impairment in a variety of language functions resulting from stroke or dementia in a large number of languages and the phenomenon of graceful degradation observed in such studies. They have also made important contributions to our understanding of attention (including hemispatial neglect), emotional function, executive function, motor planning, visual processing, decision making, and neuroeconomics. The relationship of neural network population dynamics to electroencephalographic rhythms is starting to emerge. Nevertheless, PDP approaches have scarcely penetrated major areas of study of cognition, including neuropsychology and cognitive neuropsychology, as well as much of cognitive psychology. This article attempts to provide an overview of PDP principles and applications that addresses a broader audience.

Sensitivity to Inflectional Morphemes in the Absence of Meaning: Evidence from a Novel Task

A number of studies in different languages have shown that speakers may be sensitive to the presence of inflectional morphology in the absence of verb meaning (Caramazza et al. in Cognition 28(3):297-332, 1988; Clahsen in Behav Brain Sci 22(06):991-1013, 1999; Post et al. in Cognition 109(1):1-17, 2008). In this study, sensitivity to inflectional morphemes was tested in a purposely developed task with English-like nonwords. Native speakers of English were presented with pairs of nonwords and were asked to judge whether the two nonwords in each pair were the same or different. Each pair was composed either of the same nonword repeated twice, or of two slightly different nonwords. The nonwords were created taking advantage of a specific morphophonological property of English, which is that regular inflectional morphemes agree in voicing with the ending of the stem. Using stems ending in /l/, thus, we created: (1) nonwords ending in potential inflectional morphemes, vɔld, (2) nonwords without inflectional morphemes, vɔlt, and (3) a phonological control condition, vɔlb. Our new task endorses some strengths presented in previous work. As in Post et al. (2008) the task accounts for the importance of phonological cues to morphological processing. In addition, as in Caramazza et al. (1988) and contrary to Post et al. (2008), the task never presents bare-stems, making it unlikely that the participants would be aware of the manipulation performed. Our results are in line with Caramazza et al. (1988), Clahsen (1999) and Post et al. (2008), and offer further evidence that morphologically inflected nonwords take longer to be discriminated compared to uninflected nonwords.

The role of complementary learning systems in learning and consolidation in a quasi-regular domain

We examine the role of off-line memory consolidation processes in the learning and retention of a new quasi-regular linguistic system similar to the English past tense. Quasi-regular systems are characterized by a dominance of systematic, regular forms (e.g., walk-walked, jump-jumped) alongside a smaller number of high frequency irregulars (e.g., sit-sat, go-went), and are found across many cognitive domains, from spelling-sound mappings to inflectional morphology to semantic cognition. Participants were trained on the novel morphological system using an artificial language paradigm, and then tested after different delays. Based on a complementary systems account of memory, we predicted that irregular forms would show stronger off-line changes due to consolidation processes. Across two experiments, participants were tested either immediately after learning, 12 h later with or without sleep, or 24 h later. Testing involved generalization of the morphological patterns to previously unseen words (both experiments) as well as recall of the trained words (Experiment 2). In generalization, participants showed 'default' regularization across a range of novel forms, as well as irregularization for previously unseen items that were similar to unique high-frequency irregular trained forms. Both patterns of performance remained stable across the delays. Generalizations involving competing tendencies to regularize and irregularize were balanced between the two immediately after learning. Crucially, at both 12-h delays the tendency to irregularize in these cases was strengthened, with further strengthening after 24 h. Consolidated knowledge of both regular and irregular trained items contributed significantly to generalization performance, with evidence of strengthening of irregular forms and weakening of regular forms. We interpret these findings in the context of a complementary systems model, and discuss how maintenance, strengthening, and forgetting of the new memories across sleep and wake can play a role in acquiring quasi-regular systems.

Patients with impaired verb-tense processing: do they know that yesterday is past?

This paper begins with a focus on the task of stem inflection, where participants are given a verb stem and asked to produce the verb's past-tense form, which can produce a neuropsychological double dissociation with respect to regular versus irregular verbs. Two differing theoretical interpretations are outlined: one is based on specifically morphological and separate brain mechanisms for processing regular versus irregular verbs; the other argues that the two sides of the dissociation can arise from one procedure, which is not specifically morphological, and which relies to differing extents on phonological versus semantic information for regular versus irregular verbs. We then present data from a different version of the task, in which patients were given past-tense forms and asked to produce the present-tense or stem forms (talked → talk and ate → eat). This change yielded a very different pattern of performance in four non-fluent aphasic patients as a function of the regular-irregular manipulation, an outcome which is argued to be more compatible with the single- than the dual-mechanism account. Finally, we present a small amount of data from a task in which the patient was asked to judge whether spoken regular and irregular verb stems and past-tense forms indicated actions occurring today or yesterday. This task produced an even more different and intriguing pattern of performance suggesting a deficit in morpho-syntactic knowledge: not how to produce past-tense forms but what such forms mean and how that understanding interacts with verb regularity. The paper concludes with a discussion of how the research field of acquired disorders of tense processing might advance as a result of new approaches, in particular those informed by studies of developmental disorders.

When words fail us: insights into language processing from developmental and acquired disorders

Acquired disorders of language represent loss of previously acquired skills, usually with relatively specific impairments. In children with developmental disorders of language, we may also see selective impairment in some skills; but in this case, the acquisition of language or literacy is affected from the outset. Because systems for processing spoken and written language change as they develop, we should beware of drawing too close a parallel between developmental and acquired disorders. Nevertheless, comparisons between the two may yield new insights. A key feature of connectionist models simulating acquired disorders is the interaction of components of language processing with each other and with other cognitive domains. This kind of model might help make sense of patterns of comorbidity in developmental disorders. Meanwhile, the study of developmental disorders emphasizes learning and change in underlying representations, allowing us to study how heterogeneity in cognitive profile may relate not just to neurobiology but also to experience. Children with persistent language difficulties pose challenges both to our efforts at intervention and to theories of learning of written and spoken language. Future attention to learning in individuals with developmental and acquired disorders could be of both theoretical and applied value.

Problems with tense marking in children with specific language impairment: not how but when

Many children with specific language impairment (SLI) have persisting problems in the correct use of verb tense, but there has been disagreement as to the underlying reason. When we take into account studies using receptive as well as expressive language tasks, the data suggest that the difficulty for children with SLI is in knowing when to inflect verbs for tense, rather than how to do so. This is perhaps not surprising when we consider that tense does not have a transparent semantic interpretation, but depends on complex relationships between inflections and hierarchically organized clauses. An explanation in terms of syntactic limitations contrasts with a popular morpho-phonological account, the Words and Rules model. This model, which attributes problems to difficulties with applying a rule to generate regular inflected forms, has been widely applied to adult-acquired disorders. There are striking similarities in the pattern of errors in adults with anterior aphasia and children with SLI, suggesting that impairments in appreciation of when to mark tense may apply to acquired as well as developmental disorders.

Beyond modeling abstractions: learning nouns over developmental time in atypical populations and individuals

Connectionist models that capture developmental change over time have much to offer in the field of language development research. Several models in the literature have made good contact with developmental data, effectively captured behavioral tasks, and accurately represented linguistic input available to young children. However, fewer models of language development have truly captured the process of developmental change over time. In this review paper, we discuss several prominent connectionist models of early word learning, focusing on semantic development, as well as our recent work modeling the emergence of word learning biases in different populations. We also discuss the potential of these kinds of models to capture children's language development at the individual level. We argue that a modeling approach that truly captures change over time has the potential to inform theory, guide research, and lead to innovations in early language intervention.

MEG masked priming evidence for form-based decomposition of irregular verbs

To what extent does morphological structure play a role in early processing of visually presented English past tense verbs? Previous masked priming studies have demonstrated effects of obligatory form-based decomposition for genuinely affixed words (teacher-TEACH) and pseudo-affixed words (corner-CORN), but not for orthographic controls (brothel-BROTH). Additionally, MEG single word reading studies have demonstrated that the transition probability from stem to affix (in genuinely affixed words) modulates an early evoked response known as the M170; parallel findings have been shown for the transition probability from stem to pseudo-affix (in pseudo-affixed words). Here, utilizing the M170 as a neural index of visual form-based morphological decomposition, we ask whether the M170 demonstrates masked morphological priming effects for irregular past tense verbs (following a previous study which obtained behavioral masked priming effects for irregulars). Dual mechanism theories of the English past tense predict a rule-based decomposition for regulars but not for irregulars, while certain single mechanism theories predict rule-based decomposition even for irregulars. MEG data was recorded for 16 subjects performing a visual masked priming lexical decision task. Using a functional region of interest (fROI) defined on the basis of repetition priming and regular morphological priming effects within the left fusiform and inferior temporal regions, we found that activity in this fROI was modulated by the masked priming manipulation for irregular verbs, during the time window of the M170. We also found effects of the scores generated by the learning model of Albright and Hayes (2003) on the degree of priming for irregular verbs. The results favor a single mechanism account of the English past tense, in which even irregulars are decomposed into stems and affixes prior to lexical access, as opposed to a dual mechanism model, in which irregulars are recognized as whole forms.