Breaking the mirror: Asymmetrical disconnection between the phonological input and output codes

A tale of two lexica: Investigating computational pressures on word representation with neural networks

Introduction

The notion of a single localized store of word representations has become increasingly less plausible as evidence has accumulated for the widely distributed neural representation of wordform grounded in motor, perceptual, and conceptual processes. Here, we attempt to combine machine learning methods and neurobiological frameworks to propose a computational model of brain systems potentially responsible for wordform representation. We tested the hypothesis that the functional specialization of word representation in the brain is driven partly by computational optimization. This hypothesis directly addresses the unique problem of mapping sound and articulation vs. mapping sound and meaning.

Results

We found that artificial neural networks trained on the mapping between sound and articulation performed poorly in recognizing the mapping between sound and meaning and vice versa. Moreover, a network trained on both tasks simultaneously could not discover the features required for efficient mapping between sound and higher-level cognitive states compared to the other two models. Furthermore, these networks developed internal representations reflecting specialized task-optimized functions without explicit training.

Discussion

Together, these findings demonstrate that different task-directed representations lead to more focused responses and better performance of a machine or algorithm and, hypothetically, the brain. Thus, we imply that the functional specialization of word representation mirrors a computational optimization strategy given the nature of the tasks that the human brain faces.

Using word-picture verification to inform language impairment locus in chronic post-stroke aphasia

Word-picture verification, a task that requires a yes/no response to whether a word and a picture match, has been used for both receptive and expressive language; however, there is limited systematic investigation on the linguistic subprocesses targeted by the task. Verification may help to identify linguistic strengths and weaknesses to ultimately provide more targeted, individualized lexical retrieval intervention. The current study assessed the association of semantic and phonological skills with verification performance to demonstrate early efficacy of the paradigm as an aphasia assessment. Sixteen adults with chronic post-stroke aphasia completed a battery of language assessments in addition to reading and auditory verification tasks. Verification scores were positively correlated with auditory and reading comprehension. Accuracy of semantic and phonological verification were positively correlated with accuracy on respective receptive language tasks. More semantic errors were made during verification than naming. The relationship of phonological errors between naming and verification varied by modality (reading or listening). Semantic and phonological performance significantly predicted verification response accuracy and latency. In sum, we propose that verification tasks are particularly useful because they inform semantics pre-lemma selection and phonological decoding, helping to localize individual linguistic strengths and weaknesses, especially in the presence of significant motor speech impairment that can obscure expressive language abilities.

A case-study of language-specific executive disorder

Executive control is recruited for language processing, particularly in complex linguistic tasks. Although the issue of the existence of an executive control specific to language is still an open issue, there is much evidence that executively-demanding language tasks rely on domain-general rather than language-specific executive resources. Here, we addressed this issue by assessing verbal and non-verbal executive capacities in LG, an aphasic patient after a stroke. First, we showed that LG's performance was spared in all non-verbal tasks regardless of the executive demands. Second, by contrasting conditions of high and low executive demand in verbal tasks, we showed that LG was only impaired in verbal task with high executive demand. The performance dissociation between low and high executive demand conditions in the verbal domain, not observed in the non-verbal domain, shows that verbal executive control partly dissociates from non-verbal executive control. This language-specific executive disorder suggests that some executive processes might be language-specific.

A Comprehension- or a Production-Based Monitor? Response to Roelofs (2020)

Roelofs (2020) has put forth a rebuttal of the criticisms raised against comprehension-based monitoring and has also raised a number of objections against production-based monitors. In this response, I clarify that the model defended by Roelofs is not a comprehension-based monitor, but belongs to a class of monitoring models which I refer to as production-perception models. I review comprehension-based and production-perception models, highlight the strength of each, and point out the differences between them. I then discuss the limitations of both for monitoring production at higher levels, which has been the motivation for production-based monitors. Next, I address the specific criticisms raised by Roelofs (2020) in light of the current evidence. I end by presenting several lines of arguments that preclude a single monitoring mechanism as meeting all the demands of monitoring in a task as complex as communication. A more fruitful avenue is perhaps to focus on what theories are compatible with the nature of representations at specific levels of the production system and with specific aims of monitoring in language production.

How to get by with half a loop - An investigation of visual and auditory codes in a case of impaired phonological short-term memory (pSTM)

In the field of cognitive neuropsychology of phonological short-term memory (pSTM), a key debate surrounds the issue of how impairment on tasks deemed to tap this system imply a dissociable phonological input and output buffer system, with the implication that impairments can be fractionated across disruption to separate functional components (Nickels, Howard & Best, 1997). This study presents CT, a conduction aphasic who showed no impairment on basic auditory discrimination tasks, but had very poor nonword repetition. Clear-cut examples of such cases are very rare (see Jacquemot, Dupoux & Bachoud-Levi, 2007), and we interpret the case with reference to a pSTM model that includes input and output buffers. The dissociation between performance on auditory phonological tasks and visual phonological tasks we interpret as consistent with disruption to the link from input buffer to output buffer without concurrent damage to connections from output to input. Previous research has also shown that patients with impairments of pSTM can make visual confusions with orthographically presented items in tasks seeking to tap this mechanism (Warrington & Shallice, 1972), which might stem from having an incomplete pSTM loop. In light of this we examined whether CT's ability on tests of ISR was affected by visual orthographic similarity among list items, and this is indeed what we observed. On balance then, CT's overall profile is considered best interpreted with respect to a dual buffer pSTM model (e.g., Vallar & Papagno, 2002).

Bridging computational approaches to speech production: The semantic-lexical-auditory-motor model (SLAM)

Speech production is studied from both psycholinguistic and motor-control perspectives, with little interaction between the approaches. We assessed the explanatory value of integrating psycholinguistic and motor-control concepts for theories of speech production. By augmenting a popular psycholinguistic model of lexical retrieval with a motor-control-inspired architecture, we created a new computational model to explain speech errors in the context of aphasia. Comparing the model fits to picture-naming data from 255 aphasic patients, we found that our new model improves fits for a theoretically predictable subtype of aphasia: conduction. We discovered that the improved fits for this group were a result of strong auditory-lexical feedback activation, combined with weaker auditory-motor feedforward activation, leading to increased competition from phonologically related neighbors during lexical selection. We discuss the implications of our findings with respect to other extant models of lexical retrieval.

Neural basis of speech perception

The functional neuroanatomy of speech processing has been difficult to characterize. One major impediment to progress has been the failure to consider task effects when mapping speech-related processing systems. We summarize a dual-stream model of speech processing that addresses this situation. In this model, a ventral stream processes speech signals for comprehension, and a dorsal stream maps acoustic speech signals to parietal and frontal-lobe articulatory networks. The model assumes that the ventral stream is largely bilaterally organized, although there are important computational differences between the left- and right-hemisphere systems, whereas the dorsal stream is strongly left-hemisphere-dominant.

The architecture of speech production and the role of the phoneme in speech processing

Speech production has been studied within a number of traditions including linguistics, psycholinguistics, motor control, neuropsychology, and neuroscience. These traditions have had limited interaction, ostensibly because they target different levels of speech production or different dimensions such as representation, processing, or implementation. However, closer examination of reveals a substantial convergence of ideas across the traditions and recent proposals have suggested that an integrated approach may help move the field forward. The present article reviews one such attempt at integration, the state feedback control model and its descendent, the hierarchical state feedback control model. Also considered is how phoneme-level representations might fit in the context of the model.

Parsing the phonological loop: activation timing in the dorsal speech stream determines accuracy in speech reproduction

Despite significant research and important clinical correlates, direct neural evidence for a phonological loop linking speech perception, short-term memory and production remains elusive. To investigate these processes, we acquired whole-head magnetoencephalographic (MEG) recordings from human subjects performing a variable-length syllable sequence reproduction task. The MEG sensor data were source localized using a time-frequency optimized spatially adaptive filter, and we examined the time courses of cortical oscillatory power and the correlations of oscillatory power with behavior between onset of the audio stimulus and the overt speech response. We found dissociations between time courses of behaviorally relevant activations in a network of regions falling primarily within the dorsal speech stream. In particular, verbal working memory load modulated high gamma power in both Sylvian-parietal-temporal and Broca's areas. The time courses of the correlations between high gamma power and subject performance clearly alternated between these two regions throughout the task. Our results provide the first evidence of a reverberating input-output buffer system in the dorsal stream underlying speech sensorimotor integration, consistent with recent phonological loop, competitive queuing, and speech-motor control models. These findings also shed new light on potential sources of speech dysfunction in aphasia and neuropsychiatric disorders, identifying anatomically and behaviorally dissociable activation time windows critical for successful speech reproduction.

The cortical organization of speech processing: feedback control and predictive coding the context of a dual-stream model

Speech recognition is an active process that involves some form of predictive coding. This statement is relatively uncontroversial. What is less clear is the source of the prediction. The dual-stream model of speech processing suggests that there are two possible sources of predictive coding in speech perception: the motor speech system and the lexical-conceptual system. Here I provide an overview of the dual-stream model of speech processing and then discuss evidence concerning the source of predictive coding during speech recognition. I conclude that, in contrast to recent theoretical trends, the dorsal sensory-motor stream is not a source of forward prediction that can facilitate speech recognition. Rather, it is forward prediction coming out of the ventral stream that serves this function.

Computational neuroanatomy of speech production

Speech production has been studied predominantly from within two traditions, psycholinguistics and motor control. These traditions have rarely interacted, and the resulting chasm between these approaches seems to reflect a level of analysis difference: whereas motor control is concerned with lower-level articulatory control, psycholinguistics focuses on higher-level linguistic processing. However, closer examination of both approaches reveals a substantial convergence of ideas. The goal of this article is to integrate psycholinguistic and motor control approaches to speech production. The result of this synthesis is a neuroanatomically grounded, hierarchical state feedback control model of speech production.

Conduction aphasia, sensory-motor integration, and phonological short-term memory - an aggregate analysis of lesion and fMRI data

Conduction aphasia is a language disorder characterized by frequent speech errors, impaired verbatim repetition, a deficit in phonological short-term memory, and naming difficulties in the presence of otherwise fluent and grammatical speech output. While traditional models of conduction aphasia have typically implicated white matter pathways, recent advances in lesions reconstruction methodology applied to groups of patients have implicated left temporoparietal zones. Parallel work using functional magnetic resonance imaging (fMRI) has pinpointed a region in the posterior most portion of the left planum temporale, area Spt, which is critical for phonological working memory. Here we show that the region of maximal lesion overlap in a sample of 14 patients with conduction aphasia perfectly circumscribes area Spt, as defined in an aggregate fMRI analysis of 105 subjects performing a phonological working memory task. We provide a review of the evidence supporting the idea that Spt is an interface site for the integration of sensory and vocal tract-related motor representations of complex sound sequences, such as speech and music and show how the symptoms of conduction aphasia can be explained by damage to this system.

Shared language: overlap and segregation of the neuronal infrastructure for speaking and listening revealed by functional MRI

Whether the brain's speech-production system is also involved in speech comprehension is a topic of much debate. Research has focused on whether motor areas are involved in listening, but overlap between speaking and listening might occur not only at primary sensory and motor levels, but also at linguistic levels (where semantic, lexical, and syntactic processes occur). Using functional MRI adaptation during speech comprehension and production, we found that the brain areas involved in semantic, lexical, and syntactic processing are mostly the same for speaking and for listening. Effects of primary processing load (indicative of sensory and motor processes) overlapped in auditory cortex and left inferior frontal cortex, but not in motor cortex, where processing load affected activity only in speaking. These results indicate that the linguistic parts of the language system are used for both speaking and listening, but that the motor system does not seem to provide a crucial contribution to listening.

Sensorimotor integration in speech processing: computational basis and neural organization

Sensorimotor integration is an active domain of speech research and is characterized by two main ideas, that the auditory system is critically involved in speech production and that the motor system is critically involved in speech perception. Despite the complementarity of these ideas, there is little crosstalk between these literatures. We propose an integrative model of the speech-related "dorsal stream" in which sensorimotor interaction primarily supports speech production, in the form of a state feedback control architecture. A critical component of this control system is forward sensory prediction, which affords a natural mechanism for limited motor influence on perception, as recent perceptual research has suggested. Evidence shows that this influence is modulatory but not necessary for speech perception. The neuroanatomy of the proposed circuit is discussed as well as some probable clinical correlates including conduction aphasia, stuttering, and aspects of schizophrenia.

Temporal processing capabilities in repetition conduction aphasia

This study investigates the temporal resolution capacities of the central-auditory system in a subject (NP) suffering from repetition conduction aphasia. More specifically, the patient was asked to detect brief gaps between two stretches of broadband noise (gap detection task) and to evaluate the duration of two biphasic (WN-3) continuous noise elements, starting with white noise (WN) followed by 3kHz bandpass-filtered noise (duration discrimination task). During the gap detection task, the two portions of each stimulus were either identical ("intra-channel condition") or differed ("inter-channel condition") in the spectral characteristics of the leading and trailing acoustic segments. NP did not exhibit any deficits in the intra-channel condition of the gap detection task, indicating intact auditory temporal resolution across intervals of 1-3ms. By contrast, the inter-channel condition yielded increased threshold values. Based upon the "multiple-looks" model of central-auditory processing, this profile points at a defective integration window operating across a few tens of milliseconds - a temporal range associated with critical features of the acoustic speech signal such as voice onset time and formant transitions. Additionally, NP was found impaired during a duration discrimination task addressing longer integration windows (ca. 150ms). Concerning speech, this latter time domain approximately corresponds to the duration of stationary segmental units such as fricatives and long vowels. On the basis of our results we suggest, that the patient's auditory timing deficits in non-speech tasks may account, at least partially, for his impairments in speech processing.

Naming and repetition in aphasia: Steps, routes, and frequency effects

This paper investigates the cognitive processes underlying picture naming and auditory word repetition. In the 2-step model of lexical access, both the semantic and phonological steps are involved in naming, but the former has no role in repetition. Assuming recognition of the to-be-repeated word, repetition could consist of retrieving the word's output phonemes from the lexicon (the lexical-route model), retrieving the output phonology directly from input phonology (the nonlexical-route model) or employing both routes together (the summation dual-route model). We tested these accounts by comparing the size of the word frequency effect (an index of lexical retrieval) in naming and repetition data from 59 aphasic patients with simulations of naming and repetition models. The magnitude of the frequency effect (and the influence of other lexical variables) was found to be comparable in naming and repetition, and equally large for both the lexical and summation dual-route models. However, only the dual-route model was fully consistent with data from patients, suggesting that nonlexical input is added on top of a fully-utilized lexical route.

What is the relationship between phonological short-term memory and speech processing?

Traditionally, models of speech comprehension and production do not depend on concepts and processes from the phonological short-term memory (pSTM) literature. Likewise, in working memory research, pSTM is considered to be a language-independent system that facilitates language acquisition rather than speech processing per se. We discuss couplings between pSTM, speech perception and speech production, and we propose that pSTM arises from the cycling of information between two phonological buffers, one involved in speech perception and one in speech production. We discuss the specific role of these processes in speech processing, and argue that models of speech perception and production, and our understanding of their neural bases, will benefit from incorporating them.