Modeling Interactions between Speech Production and Perception: Speech Error Detection at Semantic and Phonological Levels and the Inner Speech Loop

How trial-to-trial learning shapes mappings in the mental lexicon: Modelling lexical decision with linear discriminative learning

Trial-to-trial effects have been found in a number of studies, indicating that processing a stimulus influences responses in subsequent trials. A special case are priming effects which have been modelled successfully with error-driven learning (Marsolek, 2008), implying that participants are continuously learning during experiments. This study investigates whether trial-to-trial learning can be detected in an unprimed lexical decision experiment. We used the Discriminative Lexicon Model (DLM; Baayen et al., 2019), a model of the mental lexicon with meaning representations from distributional semantics, which models error-driven incremental learning with the Widrow-Hoff rule. We used data from the British Lexicon Project (BLP; Keuleers et al., 2012) and simulated the lexical decision experiment with the DLM on a trial-by-trial basis for each subject individually. Then, reaction times were predicted with Generalized Additive Models (GAMs), using measures derived from the DLM simulations as predictors. We extracted measures from two simulations per subject (one with learning updates between trials and one without), and used them as input to two GAMs. Learning-based models showed better model fit than the non-learning ones for the majority of subjects. Our measures also provide insights into lexical processing and individual differences. This demonstrates the potential of the DLM to model behavioural data and leads to the conclusion that trial-to-trial learning can indeed be detected in unprimed lexical decision. Our results support the possibility that our lexical knowledge is subject to continuous changes.

Automatic Assessment of Aphasic Speech Sensed by Audio Sensors for Classification into Aphasia Severity Levels to Recommend Speech Therapies

Aphasia is a type of speech disorder that can cause speech defects in a person. Identifying the severity level of the aphasia patient is critical for the rehabilitation process. In this research, we identify ten aphasia severity levels motivated by specific speech therapies based on the presence or absence of identified characteristics in aphasic speech in order to give more specific treatment to the patient. In the aphasia severity level classification process, we experiment on different speech feature extraction techniques, lengths of input audio samples, and machine learning classifiers toward classification performance. Aphasic speech is required to be sensed by an audio sensor and then recorded and divided into audio frames and passed through an audio feature extractor before feeding into the machine learning classifier. According to the results, the mel frequency cepstral coefficient (MFCC) is the most suitable audio feature extraction method for the aphasic speech level classification process, as it outperformed the classification performance of all mel-spectrogram, chroma, and zero crossing rates by a large margin. Furthermore, the classification performance is higher when 20 s audio samples are used compared with 10 s chunks, even though the performance gap is narrow. Finally, the deep neural network approach resulted in the best classification performance, which was slightly better than both K-nearest neighbor (KNN) and random forest classifiers, and it was significantly better than decision tree algorithms. Therefore, the study shows that aphasia level classification can be completed with accuracy, precision, recall, and F1-score values of 0.99 using MFCC for 20 s audio samples using the deep neural network approach in order to recommend corresponding speech therapy for the identified level. A web application was developed for English-speaking aphasia patients to self-diagnose the severity level and engage in speech therapies.

On the Emergence of Phonological Knowledge and on Motor Planning and Motor Programming in a Developmental Model of Speech Production

A broad sketch for a model of speech production is outlined which describes developmental aspects of its cognitive-linguistic and sensorimotor components. A description of the emergence of phonological knowledge is a central point in our model sketch. It will be shown that the phonological form level emerges during speech acquisition and becomes an important representation at the interface between cognitive-linguistic and sensorimotor processes. Motor planning as well as motor programming are defined as separate processes in our model sketch and it will be shown that both processes revert to the phonological information. Two computational simulation experiments based on quantitative implementations (simulation models) are undertaken to show proof of principle of key ideas of the model sketch: (i) the emergence of phonological information over developmental stages, (ii) the adaptation process for generating new motor programs, and (iii) the importance of various forms of phonological representation in that process. Based on the ideas developed within our sketch of a production model and its quantitative spell-out within the simulation models, motor planning can be defined here as the process of identifying a succession of executable chunks from a currently activated phoneme sequence and of coding them as raw gesture scores. Motor programming can be defined as the process of building up the complete set of motor commands by specifying all gestures in detail (fully specified gesture score including temporal relations). This full specification of gesture scores is achieved in our model by adapting motor information from phonologically similar syllables (adapting approach) or by assembling motor programs from sub-syllabic units (assembling approach).

Hierarchical Sequencing and Feedforward and Feedback Control Mechanisms in Speech Production: A Preliminary Approach for Modeling Normal and Disordered Speech

Our understanding of the neurofunctional mechanisms of speech production and their pathologies is still incomplete. In this paper, a comprehensive model of speech production based on the Neural Engineering Framework (NEF) is presented. This model is able to activate sensorimotor plans based on cognitive-functional processes (i.e., generation of the intention of an utterance, selection of words and syntactic frames, generation of the phonological form and motor plan; feedforward mechanism). Since the generation of different states of the utterance are tied to different levels in the speech production hierarchy, it is shown that different forms of speech errors as well as speech disorders can arise at different levels in the production hierarchy or are linked to different levels and different modules in the speech production model. In addition, the influence of the inner feedback mechanisms on normal as well as on disordered speech is examined in terms of the model. The model uses a small number of core concepts provided by the NEF, and we show that these are sufficient to create this neurobiologically detailed model of the complex process of speech production in a manner that is, we believe, clear, efficient, and understandable.

Self-Monitoring in Speaking: In Defense of a Comprehension-Based Account

Speakers occasionally make speech errors, which may be detected and corrected. According to the comprehension-based account proposed by Levelt, Roelofs, and Meyer (1999) and Roelofs (2004), speakers detect errors by using their speech comprehension system for the monitoring of overt as well as inner speech. According to the production-based account of Nozari, Dell, and Schwartz (2011), speakers may use their comprehension system for external monitoring but error detection in internal monitoring is based on the amount of conflict within the speech production system, assessed by the anterior cingulate cortex (ACC). Here, I address three main arguments of Nozari et al. and Nozari and Novick (2017) against a comprehension-based account of internal monitoring, which concern cross-talk interference between inner and overt speech, a double dissociation between comprehension and self-monitoring ability in patients with aphasia, and a domain-general error-related negativity in the ACC that is allegedly independent of conscious awareness. I argue that none of the arguments are conclusive, and conclude that comprehension-based monitoring remains a viable account of self-monitoring in speaking.

A Comprehension- or a Production-Based Monitor? Response to Roelofs (2020)

Roelofs (2020) has put forth a rebuttal of the criticisms raised against comprehension-based monitoring and has also raised a number of objections against production-based monitors. In this response, I clarify that the model defended by Roelofs is not a comprehension-based monitor, but belongs to a class of monitoring models which I refer to as production-perception models. I review comprehension-based and production-perception models, highlight the strength of each, and point out the differences between them. I then discuss the limitations of both for monitoring production at higher levels, which has been the motivation for production-based monitors. Next, I address the specific criticisms raised by Roelofs (2020) in light of the current evidence. I end by presenting several lines of arguments that preclude a single monitoring mechanism as meeting all the demands of monitoring in a task as complex as communication. A more fruitful avenue is perhaps to focus on what theories are compatible with the nature of representations at specific levels of the production system and with specific aims of monitoring in language production.

Modeling the Mental Lexicon as Part of Long-Term and Working Memory and Simulating Lexical Access in a Naming Task Including Semantic and Phonological Cues

BACKGROUND

To produce and understand words, humans access the mental lexicon. From a functional perspective, the long-term memory component of the mental lexicon is comprised of three levels: the concept level, the lemma level, and the phonological level. At each level, different kinds of word information are stored. Semantic as well as phonological cues can help to facilitate word access during a naming task, especially when neural dysfunctions are present. The processing corresponding to word access occurs in specific parts of working memory. Neural models for simulating speech processing help to uncover the complex relationships that exist between neural dysfunctions and corresponding behavioral patterns.

METHODS

The Neural Engineering Framework (NEF) and the Semantic Pointer Architecture (SPA) are used to develop a quantitative neural model of the mental lexicon and its access during speech processing. By simulating a picture-naming task (WWT 6-10), the influence of cues is investigated by introducing neural dysfunctions within the neural model at different levels of the mental lexicon.

RESULTS

First, the neural model is able to simulate the test behavior for normal children that exhibit no lexical dysfunction. Second, the model shows worse results in test performance as larger degrees of dysfunction are introduced. Third, if the severity of dysfunction is not too high, phonological and semantic cues are observed to lead to an increase in the number of correctly named words. Phonological cues are observed to be more effective than semantic cues.

CONCLUSION

Our simulation results are in line with human experimental data. Specifically, phonological cues seem not only to activate phonologically similar items within the phonological level. Moreover, phonological cues support higher-level processing during access of the mental lexicon. Thus, the neural model introduced in this paper offers a promising approach to modeling the mental lexicon, and to incorporating the mental lexicon into a complex model of language processing.

Receptive and Expressive Vocabulary Skills and Their Correlates in Mandarin-Speaking Infants with Unrepaired Cleft Lip and/or Palate

Background: Vocabulary skills in infants with cleft lip and/or palate (CL/P) are related to various factors. They remain underexplored among Mandarin-speaking infants with CL/P. This study identified receptive and expressive vocabulary skills among Mandarin-speaking infants with unrepaired CL/P prior to cleft palate surgery and their associated factors. Methods: This is a cross-sectional study involving patients at the Cleft Lip and Palate Center of the Stomatological Hospital of Xi’an Jiaotong University between July 2017 and December 2018. The Putonghua Communicative Development Inventories-Short Form (PCDI-SF) was used to assess early vocabulary skills. Results: A total of 134 children aged 9–16 months prior to cleft palate surgery were included in the study. The prevalences of delays in receptive and expressive vocabulary skills were 72.39% (95% CI: 64.00–79.76%) and 85.07% (95% CI: 77.89–90.64%), respectively. Multiple logistic regression identified that children aged 11–13 months (OR = 6.46, 95% CI: 1.76–23.76) and 14–16 months (OR = 24.32, 95% CI: 3.86–153.05), and those with hard/soft cleft palate and soft cleft palate (HSCP/SCP) (OR = 5.63, 95% CI: 1.02–31.01) were more likely to be delayed in receptive vocabulary skills. Conclusions: Delays in vocabulary skills were common among Mandarin-speaking CL/P infants, and age was positively associated with impaired and lagging vocabulary skills. The findings suggest the necessity and importance of early and effective identification of CL/P, and early intervention programs and effective treatment are recommended for Chinese CL/P infants.

The role of conflict, feedback, and action comprehension in monitoring of action errors: Evidence for internal and external routes

The mechanisms and brain regions underlying error monitoring in complex action are poorly understood, yet errors and impaired error correction in these tasks are hallmarks of apraxia, a common disorder associated with left hemisphere stroke. Accounts of monitoring of language posit an internal route by which production planning or competition between candidate representations provide predictive signals that monitoring is required to prevent error, and an external route in which output is monitored using the comprehension system. Abnormal reliance on the external route has been associated with damage to brain regions critical for sensory-motor transformation and a pattern of gradual error 'clean-up' called conduite d'approche (CD). Action pantomime data from 67 participants with left hemisphere stroke were consistent with versions of internal route theories positing that competition signals monitoring requirements. Support Vector Regression Lesion Symptom Mapping (SVR-LSM) showed that lesions in the inferior parietal, posterior temporal, and arcuate fasciculus/superior longitudinal fasciculus predicted action conduite d'approche, overlapping the regions previously observed in the language domain. A second experiment with 12 patients who produced substantial action CD assessed whether factors impacting the internal route (action production ability, competition) versus external route (vision of produced actions, action comprehension) influenced correction attempts. In these 'high CD' patients, vision of produced actions and integrity of gesture comprehension interacted to determine successful error correction, supporting external route theories. Viewed together, these and other data suggest that skilled actions are monitored both by an internal route in which conflict aids in detection and correction of errors during production planning, and an external route that detects mismatches between produced actions and stored knowledge of action appearance. The parallels between language and action monitoring mechanisms and neuroanatomical networks pave the way for further exploration of common and distinct processes across these domains.

Metacognitive Monitoring of Text Comprehension: An Investigation on Postdictive Judgments in Typically Developing Children and Children With Reading Comprehension Difficulties

The ability to assess and monitor one's own understanding of a written text is fundamental for learning and academic achievement. In the current paper, postdictive monitoring of text comprehension (i.e., the ability to judge the accuracy of responses previously given to a reading comprehension test) was investigated in both typically developing (TD) children and children with reading comprehension difficulties. Children from primary school (3rd to 5th grade) and secondary school (6th to 8th grade) participated in the study (N = 245). They were administered standardized tasks for reading comprehension, in which they had to read two texts and answer 12 multiple-choice questions after each text; subsequently, they had to provide postdictive judgments evaluating their performance: for each answer they had to select whether they judged it as correct, incorrect or whether they were uncertain. Two scores were calculated: Bias score, indicating the difference between metacognitive judgments of accuracy and actual performance; and Accurate estimation, indicating the sum of correct answers judged as "correct" and incorrect answers judged as "incorrect." Results showed that primary school children were more overconfident than secondary school children and made fewer Accurate estimations especially for "correct" responses. Furthermore, the consideration of a group of children with reading comprehension difficulties showed that these failures are linked to worse metacognitive monitoring ability of comprehension performance in comparison not only to age-matched controls but also to the TD group of third-graders. Implications for learning and achievement are discussed.