Self-, other-, and joint monitoring using forward models

The electrophysiological correlates of word pre-activation during associative word learning

Human beings continuously make use of learned associations to generate predictions about future occurrences in the environment. Such memory-related predictive processes provide a scaffold for learning in that mental representations of foreseeable events can be adjusted or strengthened based on a specific outcome. Learning the meaning of novel words through picture-word associations constitutes a prime example of associative learning because pictures preceding words can trigger word prediction through the pre-activation of the related mnemonic representations. In the present electroencephalography (EEG) study, we used event-related potentials (ERPs) to compare neural indices of word pre-activation between a word learning condition with maximal prediction likelihood and a non-learning control condition with low prediction. Results revealed that prediction-related N400 amplitudes in response to pictures decreased over time at central electrodes as a function of word learning, whereas late positive component (LPC) amplitudes increased. Notably, N400 but not LPC changes were also predictive of word learning performance, suggesting that the N400 component constitutes a sensitive marker of word pre-activation during associative word learning.

Interference in the shared-Stroop task: a comparison of self- and other-monitoring

Co-actors represent and integrate each other's actions, even when they need not monitor one another. However, monitoring is important for successful interactions, particularly those involving language, and monitoring others' utterances probably relies on similar mechanisms as monitoring one's own. We investigated the effect of monitoring on the integration of self- and other-generated utterances in the shared-Stroop task. In a solo version of the Stroop task (with a single participant responding to all stimuli; Experiment 1), participants named the ink colour of mismatching colour words (incongruent stimuli) more slowly than matching colour words (congruent). In the shared-Stroop task, one participant named the ink colour of words in one colour (e.g. red), while ignoring stimuli in the other colour (e.g. green); the other participant either named the other ink colour or did not respond. Crucially, participants either provided feedback about the correctness of their partner's response (Experiment 3) or did not (Experiment 2). Interference was greater when both participants responded than when they did not, but only when their partners provided feedback. We argue that feedback increased interference because monitoring one's partner enhanced representations of the partner's target utterance, which in turn interfered with self-monitoring of the participant's own utterance.

Towards a New Model of Verbal Monitoring

As all human activities, verbal communication is fraught with errors. It is estimated that humans produce around 16,000 words per day, but the word that is selected for production is not always correct and neither is the articulation always flawless. However, to facilitate communication, it is important to limit the number of errors. This is accomplished via the verbal monitoring mechanism. A body of research over the last century has uncovered a number of properties of the mechanisms at work during verbal monitoring. Over a dozen routes for verbal monitoring have been postulated. However, to date a complete account of verbal monitoring does not exist. In the current paper we first outline the properties of verbal monitoring that have been empirically demonstrated. This is followed by a discussion of current verbal monitoring models: the perceptual loop theory, conflict monitoring, the hierarchical state feedback control model, and the forward model theory. Each of these models is evaluated given empirical findings and theoretical considerations. We then outline lacunae of current theories, which we address with a proposal for a new model of verbal monitoring for production and perception, based on conflict monitoring models. Additionally, this novel model suggests a mechanism of how a detected error leads to a correction. The error resolution mechanism proposed in our new model is then tested in a computational model. Finally, we outline the advances and predictions of the model.

The role of working memory for syntactic formulation in language production

Four picture-description experiments investigated if syntactic formulation in language production can proceed with only minimal working memory involvement. Experiments 1-3 compared the initiation latencies, utterance durations, and errors for syntactically simpler picture descriptions (adjective-noun phrases, e.g., the red book) to those of more complex descriptions (relative clauses, e.g., the book that is red). In Experiment 4, the syntactically more complex descriptions were also lexically more complex (e.g., the book and the car vs. the book). Simpler and more complex descriptions were produced under verbal memory load consisting of 2 or 4 unrelated nouns, or under no load. Across experiments, load actually made production more efficient (as manifested in shorter latencies, shorter durations or both), and sped up the durations of relative clauses more than those of adjective-noun phrases. The only evidence for disproportional disruption of more complex descriptions by load was a greater increase of production errors for these descriptions than for simpler descriptions under load in Experiments 2 and 4. We thus conclude that syntactic formulation in production (for certain constructions or in certain situations) can proceed with minimal working memory involvement. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

I see what you meant to say: Anticipating speech errors during online sentence processing

Everyday speech is rife with errors and disfluencies, yet processing what we hear usually feels effortless. How does the language comprehension system accomplish such an impressive feat? The current experiment tests the hypothesis that listeners draw on relevant contextual and linguistic cues to anticipate speech errors and mentally correct them, even before receiving an explicit correction from the speaker. In the current visual-world eye-tracking experiment, we monitored participants' eye movements to objects in a display while they listened to utterances containing reparandum-repair speech errors (e.g., . . . his cat, uh I mean his dog . . .). The contextual plausibility of the misspoken word and the certainty with which the speaker uttered this word were systematically manipulated. Results showed that listeners immediately exploited these cues to generate top-down expectations regarding the speaker's communicative intention. Crucially, listeners used these expectations to constrain the bottom-up speech input and mentally correct perceived speech errors, even before the speaker initiated the correction. The results provide powerful evidence regarding the joint process of correcting speech errors that involves both the speaker and the listener. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Structural priming is a useful but imperfect technique for studying all linguistic representations, including those of pragmatics

Structural priming is a useful tool for investigating linguistics representations. We argue that structural priming can be extended to the investigation of pragmatic representations such as Gricean enrichments. That is not to say priming is without its limitations, however. Interpreting a failure to observe priming may not be as simple as Branigan & Pickering (B&P) imply.

Verbal monitoring in Parkinson's disease: A comparison between internal and external monitoring

Patients with Parkinson's disease (PD) display a variety of impairments in motor and non-motor language processes; speech is decreased on motor aspects such as amplitude, prosody and speed and on linguistic aspects including grammar and fluency. Here we investigated whether verbal monitoring is impaired and what the relative contributions of the internal and external monitoring route are on verbal monitoring in patients with PD relative to controls. Furthermore, the data were used to investigate whether internal monitoring performance could be predicted by internal speech perception tasks, as perception based monitoring theories assume. Performance of 18 patients with Parkinson's disease was measured on two cognitive performance tasks and a battery of 11 linguistic tasks, including tasks that measured performance on internal and external monitoring. Results were compared with those of 16 age-matched healthy controls. PD patients and controls generally performed similarly on the linguistic and monitoring measures. However, we observed qualitative differences in the effects of noise masking on monitoring and disfluencies and in the extent to which the linguistic tasks predicted monitoring behavior. We suggest that the patients differ from healthy subjects in their recruitment of monitoring channels.

Form Overrides Meaning When Bilinguals Monitor for Errors

Bilinguals rarely produce unintended language switches, which may in part be because switches are detected and corrected by an internal monitor. But are language switches easier or harder to detect than within-language semantic errors? To approximate internal monitoring, bilinguals listened (Experiment 1) or read aloud (Experiment 2) stories, and detected language switches (translation equivalents or semantically unrelated to expected words) and within-language errors (semantically related or unrelated to expected words). Bilinguals detected semantically related within-language errors most slowly and least accurately, language switches more quickly and accurately than within-language errors, and (in Experiment 2), translation equivalents as quickly and accurately as unrelated language switches. These results suggest that internal monitoring of form (which can detect mismatches in language membership) completes earlier than, and is independent of, monitoring of meaning. However, analysis of reading times prior to error detection revealed meaning violations to be more disruptive for processing than language violations.

Right Lateral Cerebellum Represents Linguistic Predictability

Mounting evidence indicates that posterolateral portions of the cerebellum (right Crus I/II) contribute to language processing, but the nature of this role remains unclear. Based on a well-supported theory of cerebellar motor function, which ascribes to the cerebellum a role in short-term prediction through internal modeling, we hypothesize that right cerebellar Crus I/II supports prediction of upcoming sentence content. We tested this hypothesis using event-related fMRI in male and female human subjects by manipulating the predictability of written sentences. Our design controlled for motor planning and execution, as well as for linguistic features and working memory load; it also allowed separation of the prediction interval from the presentation of the final sentence item. In addition, three further fMRI tasks captured semantic, phonological, and orthographic processing to shed light on the nature of the information processed. As hypothesized, activity in right posterolateral cerebellum correlated with the predictability of the upcoming target word. This cerebellar region also responded to prediction error during the outcome of the trial. Further, this region was engaged in phonological, but not semantic or orthographic, processing. This is the first imaging study to demonstrate a right cerebellar contribution in language comprehension independently from motor, cognitive, and linguistic confounds. These results complement our work using other methodologies showing cerebellar engagement in linguistic prediction and suggest that internal modeling of phonological representations aids language production and comprehension.

SIGNIFICANCE STATEMENT The cerebellum is traditionally seen as a motor structure that allows for smooth movement by predicting upcoming signals. However, the cerebellum is also consistently implicated in nonmotor functions such as language and working memory. Using fMRI, we identify a cerebellar area that is active when words are predicted and when these predictions are violated. This area is active in a separate task that requires phonological processing, but not in tasks that require semantic or visuospatial processing. Our results support the idea of prediction as a unifying cerebellar function in motor and nonmotor domains. We provide new insights by linking the cerebellar role in prediction to its role in verbal working memory, suggesting that these predictions involve phonological processing.

The timing of spontaneous detection and repair of naming errors in aphasia

This study examined the timing of spontaneous self-monitoring in the naming responses of people with aphasia. Twelve people with aphasia completed a 615-item naming test twice, in separate sessions. Naming attempts were scored for accuracy and error type, and verbalizations indicating detection were coded as negation (e.g., "no, not that") or repair attempts (i.e., a changed naming attempt). Focusing on phonological and semantic errors, we measured the timing of the errors and of the utterances that provided evidence of detection. The effects of error type and detection response type on error-to-detection latencies were analyzed using mixed-effects regression modeling. We first asked whether phonological errors and semantic errors differed in the timing of the detection process or repair planning. Results suggested that the two error types primarily differed with respect to repair planning. Specifically, repair attempts for phonological errors were initiated more quickly than repair attempts for semantic errors. We next asked whether this difference between the error types could be attributed to the tendency for phonological errors to have a high degree of phonological similarity with the subsequent repair attempts, thereby speeding the programming of the repairs. Results showed that greater phonological similarity between the error and the repair was associated with faster repair times for both error types, providing evidence of error-to-repair priming in spontaneous self-monitoring. When controlling for phonological overlap, significant effects of error type and repair accuracy on repair times were also found. These effects indicated that correct repairs of phonological errors were initiated particularly quickly, whereas repairs of semantic errors were initiated relatively slowly, regardless of their accuracy. We discuss the implications of these findings for theoretical accounts of self-monitoring and the role of speech error repair in learning.

The real-time prediction and inhibition of linguistic outcomes: Effects of language and literacy skill

Recent studies have found considerable individual variation in language comprehenders' predictive behaviors, as revealed by their anticipatory eye movements during language comprehension. The current study investigated the relationship between these predictive behaviors and the language and literacy skills of a diverse, community-based sample of young adults. We found that rapid automatized naming (RAN) was a key determinant of comprehenders' prediction ability (e.g., as reflected in predictive eye movements to a white cake on hearing "The boy will eat the white…"). Simultaneously, comprehension-based measures predicted participants' ability to inhibit eye movements to objects that shared features with predictable referents but were implausible completions (e.g., as reflected in eye movements to a white but inedible white car). These findings suggest that the excitatory and inhibitory mechanisms that support prediction during language processing are closely linked with specific cognitive abilities that support literacy. We show that a self-organizing cognitive architecture captures this pattern of results.

Cerebellar BOLD signal during the acquisition of a new lexicon predicts its early consolidation

Cerebellar contributions to language are presently poorly understood, but it has been argued that the cerebellar role in motor learning can be extended to learning in cognitive and linguistic domains. Here, we used fMRI to investigate whether the cerebellum is recruited in mapping novel words onto existing semantic concepts. On separate days, participants performed a Basque vocabulary learning task and a control English synonym task in the MRI scanner. Learning-related BOLD activity was found in left inferior frontal gyrus, bilateral insula, pre-SMA, left superior parietal cortex, right caudate, the right cerebellar vermis and right cerebellar Crus II. The extent to which the cerebellar regions, but not the cerebral areas, were recruited during learning correlated positively with participants' off-line improvement in performance after the learning task. These data provide evidence for a cerebellar role in lexical learning, and suggest that the right cerebellum may contribute toward consolidation of lexico-semantic associations in the language network.

Conflict monitoring in speech processing: An fMRI study of error detection in speech production and perception

To minimize the number of errors in speech, and thereby facilitate communication, speech is monitored before articulation. It is, however, unclear at which level during speech production monitoring takes place, and what mechanisms are used to detect and correct errors. The present study investigated whether internal verbal monitoring takes place through the speech perception system, as proposed by perception-based theories of speech monitoring, or whether mechanisms independent of perception are applied, as proposed by production-based theories of speech monitoring. With the use of fMRI during a tongue twister task we observed that error detection in internal speech during noise-masked overt speech production and error detection in speech perception both recruit the same neural network, which includes pre-supplementary motor area (pre-SMA), dorsal anterior cingulate cortex (dACC), anterior insula (AI), and inferior frontal gyrus (IFG). Although production and perception recruit similar areas, as proposed by perception-based accounts, we did not find activation in superior temporal areas (which are typically associated with speech perception) during internal speech monitoring in speech production as hypothesized by these accounts. On the contrary, results are highly compatible with a domain general approach to speech monitoring, by which internal speech monitoring takes place through detection of conflict between response options, which is subsequently resolved by a domain general executive center (e.g., the ACC).

Prediction and Production of Simple Mathematical Equations: Evidence from Visual World Eye-Tracking

The relationship between the production and the comprehension systems has recently become a topic of interest for many psycholinguists. It has been argued that these systems are tightly linked and in particular that listeners use the production system to predict upcoming content. In this study, we tested how similar production and prediction processes are in a novel version of the visual world paradigm. Dutch speaking participants (native speakers in Experiment 1; German-Dutch bilinguals in Experiment 2) listened to mathematical equations while looking at a clock face featuring the numbers 1 to 12. On alternating trials, they either heard a complete equation ("three plus eight is eleven") or they heard the first part ("three plus eight is") and had to produce the result ("eleven") themselves. Participants were encouraged to look at the relevant numbers throughout the trial. Their eye movements were recorded and analyzed. We found that the participants' eye movements in the two tasks were overall very similar. They fixated the first and second number of the equations shortly after they were mentioned, and fixated the result number well before they named it on production trials and well before the recorded speaker named it on comprehension trials. However, all fixation latencies were shorter on production than on comprehension trials. These findings suggest that the processes involved in planning to say a word and anticipating hearing a word are quite similar, but that people are more aroused or engaged when they intend to respond than when they merely listen to another person.

The use of content and timing to predict turn transitions

For addressees to respond in a timely fashion, they cannot simply process the speaker's utterance as it occurs and wait till it finishes. Instead, they predict both when the speaker will conclude and what linguistic forms will be used. While doing this, they must also prepare their own response. To explain this, we draw on the account proposed by Pickering and Garrod (2013a), in which addressees covertly imitate the speaker's utterance and use this to determine the intention that underlies their upcoming utterance. They use this intention to predict when and how the utterance will end, and also to drive their own production mechanisms for preparing their response. Following Arnal and Giraud (2012), we distinguish between mechanisms that predict timing and content. In particular, we propose that the timing mechanism relies on entrainment of low-frequency oscillations between speech envelope and brain. This constrains the context that feeds into the determination of the speaker's intention and hence the timing and form of the upcoming utterance. This approach typically leads to well-timed contributions, but also provides a mechanism for resolving conflicts, for example when there is unintended speaker overlap.

Sensorimotor synchronization with tempo-changing auditory sequences: Modeling temporal adaptation and anticipation

The current study investigated the human ability to synchronize movements with event sequences containing continuous tempo changes. This capacity is evident, for example, in ensemble musicians who maintain precise interpersonal coordination while modulating the performance tempo for expressive purposes. Here we tested an ADaptation and Anticipation Model (ADAM) that was developed to account for such behavior by combining error correction processes (adaptation) with a predictive temporal extrapolation process (anticipation). While previous computational models of synchronization incorporate error correction, they do not account for prediction during tempo-changing behavior. The fit between behavioral data and computer simulations based on four versions of ADAM was assessed. These versions included a model with adaptation only, one in which adaptation and anticipation act in combination (error correction is applied on the basis of predicted tempo changes), and two models in which adaptation and anticipation were linked in a joint module that corrects for predicted discrepancies between the outcomes of adaptive and anticipatory processes. The behavioral experiment required participants to tap their finger in time with three auditory pacing sequences containing tempo changes that differed in the rate of change and the number of turning points. Behavioral results indicated that sensorimotor synchronization accuracy and precision, while generally high, decreased with increases in the rate of tempo change and number of turning points. Simulations and model-based parameter estimates showed that adaptation mechanisms alone could not fully explain the observed precision of sensorimotor synchronization. Including anticipation in the model increased the precision of simulated sensorimotor synchronization and improved the fit of model to behavioral data, especially when adaptation and anticipation mechanisms were linked via a joint module based on the notion of joint internal models. Overall results suggest that adaptation and anticipation mechanisms both play an important role during sensorimotor synchronization with tempo-changing sequences. This article is part of a Special Issue entitled SI: Prediction and Attention.

Voice, (inter-)subjectivity, and real time recurrent interaction

Received approaches to a unified phenomenon called “language” are firmly committed to a Cartesian view of distinct unobservable minds. Questioning this commitment leads us to recognize that the boundaries conventionally separating the linguistic from the non-linguistic can appear arbitrary, omitting much that is regularly present during vocal communication. The thesis is put forward that uttering, or voicing, is a much older phenomenon than the formal structures studied by the linguist, and that the voice has found elaborations and codifications in other domains too, such as in systems of ritual and rite. Voice, it is suggested, necessarily gives rise to a temporally bound subjectivity, whether it is in inner speech (Descartes' “cogito”), in conversation, or in the synchronized utterances of collective speech found in prayer, protest, and sports arenas world wide. The notion of a fleeting subjective pole tied to dynamically entwined participants who exert reciprocal influence upon each other in real time provides an insightful way to understand notions of common ground, or socially shared cognition. It suggests that the remarkable capacity to construct a shared world that is so characteristic of Homo sapiens may be grounded in this ability to become dynamically entangled as seen, e.g., in the centrality of joint attention in human interaction. Empirical evidence of dynamic entanglement in joint speaking is found in behavioral and neuroimaging studies. A convergent theoretical vocabulary is now available in the concept of participatory sense-making, leading to the development of a rich scientific agenda liberated from a stifling metaphysics that obscures, rather than illuminates, the means by which we come to inhabit a shared world.

Getting ahead: forward models and their place in cognitive architecture

The use of forward models (mechanisms that predict the future state of a system) is well established in cognitive and computational neuroscience. We compare and contrast two recent, but interestingly divergent, accounts of the place of forward models in the human cognitive architecture. On the Auxiliary Forward Model (AFM) account, forward models are special-purpose prediction mechanisms implemented by additional circuitry distinct from core mechanisms of perception and action. On the Integral Forward Model (IFM) account, forward models lie at the heart of all forms of perception and action. We compare these neighbouring but importantly different visions and consider their implications for the cognitive sciences. We end by asking what kinds of empirical research might offer evidence favouring one or the other of these approaches.