Combined eye tracking and fMRI reveals neural basis of linguistic predictions during sentence comprehension

Specific lexico-semantic predictions are associated with unique spatial and temporal patterns of neural activity

We used Magnetoencephalography (MEG) in combination with Representational Similarity Analysis to probe neural activity associated with distinct, item-specific lexico-semantic predictions during language comprehension. MEG activity was measured as participants read highly constraining sentences in which the final words could be predicted. Before the onset of the predicted words, both the spatial and temporal patterns of brain activity were more similar when the same words were predicted than when different words were predicted. The temporal patterns localized to the left inferior and medial temporal lobe. These findings provide evidence that unique spatial and temporal patterns of neural activity are associated with item-specific lexico-semantic predictions. We suggest that the unique spatial patterns reflected the prediction of spatially distributed semantic features associated with the predicted word, and that the left inferior/medial temporal lobe played a role in temporally 'binding' these features, giving rise to unique lexico-semantic predictions.

Universal neural basis of structure building evidenced by network modulations emerging from Broca's area: The case of Chinese

The basic steps in building up language involve binding words of different categories into a hierarchical structure. To what extent these steps are universal or differ across languages is an open issue. Here we examine the neural dynamics of phrase structure building in Chinese-a language that in contrast to other languages heavily depends on contextual semantic information. We used functional magnetic resonance imaging and dynamic causal modeling to identify the relevant brain regions and their dynamic relations. Language stimuli consisted of syntax-driving determiners, semantics-embedded classifiers, and nonverbal symbols making up for two-component sequences manipulated by the factors structure (phrase/list) and number of words (2-word/1-word). Processing phrases compared with word lists elicited greater activation in the anterior part of Broca's area, Brodmann area (BA) 45, and the left posterior superior/middle temporal gyri (pSTG/pMTG), while processing two words against one word led to stronger involvement of the left BA 45, BA 44, and insula. Differential network modulations emerging from subparts of Broca's area revealed that phrasal construction in particular highly modulated the direct connection from BA 44 to left pMTG, suggesting BA 44's primary role in phrase structure building. Conversely, the involvement of BA 45 rather appears sensitive to the reliance on lexico-semantic information in Chinese. Against the background of previous findings from other languages, the present results indicate that phrase structure building has a universal neural basis within the left fronto-temporal network. Most importantly, they provide the first evidence demonstrating that the structure-building network may be modulated by language-specific characteristics.

Emergence of the neural network underlying phonological processing from the prereading to the emergent reading stage: A longitudinal study

Numerous studies have shown that phonological skills are critical for successful reading acquisition. However, how the brain network supporting phonological processing evolves and how it supports the initial course of learning to read is largely unknown. Here, for the first time, we characterized the emergence of the phonological network in 28 children over three stages (prereading, beginning reading, and emergent reading) longitudinally. Across these three time points, decreases in neural activation in the left inferior parietal cortex (LIPC) were observed during an audiovisual phonological processing task, suggesting a specialization process in response to reading instruction/experience. Furthermore, using the LIPC as the seed, a functional network consisting of the left inferior frontal, left posterior occipitotemporal, and right angular gyri was identified. The connection strength in this network co-developed with the growth of phonological skills. Moreover, children with above-average gains in phonological processing showed a significant developmental increase in connection strength in this network longitudinally, while children with below-average gains in phonological processing exhibited the opposite trajectory. Finally, the connection strength between the LIPC and the left posterior occipitotemporal cortex at the prereading level significantly predicted reading performance at the emergent reading stage. Our findings highlight the importance of the early emerging phonological network for reading development, providing direct evidence for the Interactive Specialization Theory and neurodevelopmental models of reading.

Understanding environmental sounds in sentence context

There is debate about how individuals use context to successfully predict and recognize words. One view argues that context supports neural predictions that make use of the speech motor system, whereas other views argue for a sensory or conceptual level of prediction. While environmental sounds can convey clear referential meaning, they are not linguistic signals, and are thus neither produced with the vocal tract nor typically encountered in sentence context. We compared the effect of spoken sentence context on recognition and comprehension of spoken words versus nonspeech, environmental sounds. In Experiment 1, sentence context decreased the amount of signal needed for recognition of spoken words and environmental sounds in similar fashion. In Experiment 2, listeners judged sentence meaning in both high and low contextually constraining sentence frames, when the final word was present or replaced with a matching environmental sound. Results showed that sentence constraint affected decision time similarly for speech and nonspeech, such that high constraint sentences (i.e., frame plus completion) were processed faster than low constraint sentences for speech and nonspeech. Linguistic context facilitates the recognition and understanding of nonspeech sounds in much the same way as for spoken words. This argues against a simple form of a speech-motor explanation of predictive coding in spoken language understanding, and suggests support for conceptual-level predictions.

Prediction is Production: The missing link between language production and comprehension

Language comprehension often involves the generation of predictions. It has been hypothesized that such prediction-for-comprehension entails actual language production. Recent studies provided evidence that the production system is recruited during language comprehension, but the link between production and prediction during comprehension remains hypothetical. Here, we tested this hypothesis by comparing prediction during sentence comprehension (primary task) in participants having the production system either available or not (non-verbal versus verbal secondary task). In the primary task, sentences containing an expected or unexpected target noun-phrase were presented during electroencephalography recording. Prediction, measured as the magnitude of the N400 effect elicited by the article (expected versus unexpected), was hindered only when the production system was taxed during sentence context reading. The present study provides the first direct evidence that the availability of the speech production system is necessary for generating lexical prediction during sentence comprehension. Furthermore, these important results provide an explanation for the recruitment of language production during comprehension.

Mouse-tracking evidence for parallel anticipatory option evaluation

In fast-paced, dynamic tasks, the ability to anticipate the future outcome of a sequence of events is crucial to quickly selecting an appropriate course of action among multiple alternative options. There are two classes of theories that describe how anticipation occurs. Serial theories assume options are generated and evaluated one at a time, in order of quality, whereas parallel theories assume simultaneous generation and evaluation. The present research examined the option evaluation process during a task designed to be analogous to prior anticipation tasks, but within the domain of narrative text comprehension. Prior research has relied on indirect, off-line measurement of the option evaluation process during anticipation tasks. Because the movement of the hand can provide a window into underlying cognitive processes, online metrics such as continuous mouse tracking provide more fine-grained measurements of cognitive processing as it occurs in real time. In this study, participants listened to three-sentence stories and predicted the protagonists' final action by moving a mouse toward one of three possible options. Each story was presented with either one (control condition) or two (distractor condition) plausible ending options. Results seem most consistent with a parallel option evaluation process because initial mouse trajectories deviated further from the best option in the distractor condition compared to the control condition. It is difficult to completely rule out all possible serial processing accounts, although the results do place constraints on the time frame in which a serial processing explanation must operate.

Language Prediction Is Reflected by Coupling between Frontal Gamma and Posterior Alpha Oscillations

Readers and listeners actively predict upcoming words during language processing. These predictions might serve to support the unification of incoming words into sentence context and thus rely on interactions between areas in the language network. In the current magnetoencephalography study, participants read sentences that varied in contextual constraints so that the predictability of the sentence-final words was either high or low. Before the sentence-final words, we observed stronger alpha power suppression for the highly compared with low constraining sentences in the left inferior frontal cortex, left posterior temporal region, and visual word form area. Importantly, the temporal and visual word form area alpha power correlated negatively with left frontal gamma power for the highly constraining sentences. We suggest that the correlation between alpha power decrease in temporal language areas and left prefrontal gamma power reflects the initiation of an anticipatory unification process in the language network.

Prediction Signatures in the Brain: Semantic Pre-Activation during Language Comprehension

There is broad agreement that context-based predictions facilitate lexical-semantic processing. A robust index of semantic prediction during language comprehension is an evoked response, known as the N400, whose amplitude is modulated as a function of semantic context. However, the underlying neural mechanisms that utilize relations of the prior context and the embedded word within it are largely unknown. We measured magnetoencephalography (MEG) data while participants were listening to simple German sentences in which the verbs were either highly predictive for the occurrence of a particular noun (i.e., provided context) or not. The identical set of nouns was presented in both conditions. Hence, differences for the evoked responses of the nouns can only be due to differences in the earlier context. We observed a reduction of the N400 response for highly predicted nouns. Interestingly, the opposite pattern was observed for the preceding verbs: highly predictive (that is more informative) verbs yielded stronger neural magnitude compared to less predictive verbs. A negative correlation between the N400 effect of the verb and that of the noun was found in a distributed brain network, indicating an integral relation between the predictive power of the verb and the processing of the subsequent noun. This network consisted of left hemispheric superior and middle temporal areas and a subcortical area; the parahippocampus. Enhanced activity for highly predictive relative to less predictive verbs, likely reflects establishing semantic features associated with the expected nouns, that is a pre-activation of the expected nouns.

Expanding the Language Network: Direct Contributions from the Hippocampus

New research suggests that the same hippocampal computations used in support of memory are also used for language processing, providing direct neurophysiological evidence of a shared neural mechanism for memory and language. This work expands classic memory and language models and represents a new opportunity for studying the memory-language interface.

Words in Context: The Effects of Length, Frequency, and Predictability on Brain Responses During Natural Reading

Word length, frequency, and predictability count among the most influential variables during reading. Their effects are well-documented in eye movement studies, but pertinent evidence from neuroimaging primarily stem from single-word presentations. We investigated the effects of these variables during reading of whole sentences with simultaneous eye-tracking and functional magnetic resonance imaging (fixation-related fMRI). Increasing word length was associated with increasing activation in occipital areas linked to visual analysis. Additionally, length elicited a U-shaped modulation (i.e., least activation for medium-length words) within a brain stem region presumably linked to eye movement control. These effects, however, were diminished when accounting for multiple fixation cases. Increasing frequency was associated with decreasing activation within left inferior frontal, superior parietal, and occipito-temporal regions. The function of the latter region—hosting the putative visual word form area—was originally considered as limited to sublexical processing. An exploratory analysis revealed that increasing predictability was associated with decreasing activation within middle temporal and inferior frontal regions previously implicated in memory access and unification. The findings are discussed with regard to their correspondence with findings from single-word presentations and with regard to neurocognitive models of visual word recognition, semantic processing, and eye movement control during reading.

Anticipating syntax during reading: Evidence from the boundary change paradigm

Previous evidence suggests that grammatical constraints have a rapid influence during language comprehension, particularly at the level of word categories (noun, verb, preposition). These findings are in conflict with a recent study from Angele, Laishley, Rayner, and Liversedge (2014), in which sentential fit had no early influence on word skipping rates during reading. In the present study, we used a gaze-contingent boundary change paradigm to manipulate the syntactic congruity of an upcoming noun or verb outside of participants' awareness. Across 3 experiments (total N = 148), we observed higher skipping rates for syntactically valid previews (The admiral would not confess . . .), when compared with violation previews (The admiral would not surgeon . . .). Readers were less likely to skip an ungrammatical continuation, even when that word was repeated within the same sentence (The admiral would not admiral . . .), suggesting that word-class constraints can take precedence over lexical repetition effects. To our knowledge, these results provide the first evidence for an influence of syntactic context during parafoveal word recognition. On the basis of the early time-course of this effect, we argue that readers can use grammatical constraints to generate syntactic expectations for upcoming words. (PsycINFO Database Record

Visualization of Data Regarding Infections Using Eye Tracking Techniques

OBJECTIVE

To evaluate ease of use and usefulness for nurses of visualizations of infectious disease transmission in a hospital.

DESIGN

An observational study was used to evaluate perceptions of several visualizations of data extracted from electronic health records designed using a participatory approach. Twelve nurses in the master's program in an urban research-intensive nursing school participated in May 2015.

METHODS

A convergent parallel mixed method was used to evaluate nurses' perceptions on ease of use and usefulness of five visualization conveying trends in hospital infection transmission applying think-aloud, interview, and eye-tracking techniques.

FINDINGS

Subjective data from the interview and think-aloud techniques indicated that participants preferred the traditional line graphs in simple data representation due to their familiarity, clarity, and easiness to read. An objective quantitative measure of eye movement analysis (444,421 gaze events) identified a high degree of participants' attention span in infographics in all three scenarios. All participants responded with the correct answer within 1 min in comprehensive tests.

CONCLUSIONS

A user-centric approach was effective in developing and evaluating visualizations for hospital infection transmission. For the visualizations designed by the users, the participants were easily able to comprehend the infection visualizations on both line graphs and infographics for simple visualization. The findings from the objective comprehension test and eye movement and subjective attitudes support the feasibility of integrating user-centric visualization designs into electronic health records, which may inspire clinicians to be mindful of hospital infection transmission. Future studies are needed to investigate visualizations and motivation, and the effectiveness of visualization on infection rate.

CLINICAL RELEVANCE

This study designed visualization images using clinical data from electronic health records applying a user-centric approach. The design insights can be applied for visualizing patient data in electronic health records.