Analysing data from the psycholinguistic visual-world paradigm: Comparison of different analysis methods

The acceptability and validity of AI-generated psycholinguistic stimuli

Sentence stimuli pervade psycholinguistics research. Yet, limited attention has been paid to the automatic construction of sentence stimuli. Given their linguistic capabilities, this study investigated the efficacy of ChatGPT in generating sentence stimuli and AI tools in producing auditory sentence stimuli. In three psycholinguistic experiments, this study examined the acceptability and validity of AI-formulated auditory sentences and written sentences in one of the two languages: English and Arabic. In Experiment 1 and 3, participants gave English AI-generated stimuli similar to or higher acceptability ratings than human-composed stimuli. In Experiment 2, Arabic AI-generated stimuli received lower acceptability ratings than their human-composed counterparts. The validity of AI-developed stimuli relied on the study design, with only Experiment 2 demonstrating the target psycholinguistic effect. These results highlight the promising role of AI as a stimuli developer, which could facilitate psycholinguistic research and increase its diversity. Implications for psycholinguistic research were discussed.

A linking hypothesis for eyetracking and mousetracking in the visual world paradigm

For a linking hypothesis in the visual world paradigm to clearly accommodate existing findings and make unambiguous predictions, it needs to be computationally implemented in a fashion that transparently draws the causal connection between the activations of internal representations and the measured output of saccades and reaching movements. Quantitatively implemented linking hypotheses provide an opportunity to not only demonstrate an existence proof of that causal connection but also to test the fidelity of the measuring methods themselves. When a system of interest is measured one way (e.g., ballistic dichotomous outputs) or another way (e.g., smooth graded outputs), the apparent results can differ substantially. What is needed is one linking hypothesis that can produce both types of outputs. The localist attractor network simulation of spoken word recognition demonstrated here recreates eye and mouse movements that capture key findings in the visual world paradigm, and especially relies on one particularly powerful theoretical construct: feedback from the action-perception cycle. Visual feedback from the eye position enhancing the cognitive prominence of the fixated object allows the simulation to fit a wider range of findings, and points to predictions for new experiments. When that feedback is absent, the linking hypothesis simulation no longer fits human data as well. Future experiments, and improvements of this network simulation, are discussed.

Multimodal oculomotor assessment reveals prodromal markers of Parkinson's disease in non-manifesting LRRK2 G2019S mutation carriers

Oculomotor behaviour changes in patients with Parkinson's disease (PD) are a promising source of prodromal disease markers. Capitalizing on this phenomenon to facilitate early diagnosis requires oculomotor assessment in prodromal cohorts. We examined oculomotor behaviour in non-manifesting LRRK2 G2019S mutation carriers (LRRK2-NM), who have heightened PD risk.Seventeen LRRK2-NM participants, 47 patients with idiopathic PD, and 63 healthy age-matched control participants completed an interleaved pro- and antisaccade task while undergoing video-based eye-tracking. We analyzed between-group differences in saccade, pupil, blink, and fixation acquisition behaviour. Patients with PD showed previously demonstrated abnormalities (saccade hypometria, antisaccade errors). Relative to controls, LRRK2-NM participants and patients with PD both displayed increased short-latency prosaccades and reduced pupil velocity, plus altered fixation acquisition-less preemptive returning of gaze to the future fixation point location. Interestingly, the effect on blink probability was opposite-higher than controls in LRRK2-NM participants but lower in patients with PD. Future longitudinal studies must confirm the viability of these features as prodromal PD markers.

The role of acoustic cues in the checked-unchecked tone merging of the Qixian Jin dialect

The weakening/loss of the stop coda in checked tone syllables (also known as "Ru syllable opening") may lead to a subsequent merger of tonal contrasts in Chinese. This study examined the role of acoustic cues in checked-unchecked tone merging in the Qixian Jin dialect by comparing three age groups. Results showed that duration served as a robust cue for the tonal contrast regardless of age, whereas glottalization did not. The F0 contour signaled the tone merging process with variations across age groups. The findings have implications for modeling complex F0 (falling-rising) contours to enable further cross-dialect comparisons from a phonetic perspective.

Conflict Adaptation in Aphasia: Upregulating Cognitive Control for Improved Sentence Comprehension

PURPOSE

This study investigated conflict adaptation in aphasia, specifically whether upregulating cognitive control improves sentence comprehension.

METHOD

Four individuals with mild aphasia completed four eye tracking sessions with interleaved auditory Stroop and sentence-to-picture matching trials (critical and filler sentences). Auditory Stroop congruency (congruent/incongruent across a male/female voice saying "boy"/"girl") was crossed with sentence congruency (syntactically correct sentences that are semantically plausible/implausible), resulting in four experimental conditions (congruent auditory Stroop followed by incongruent sentence [CI], incongruent auditory Stroop followed by incongruent sentence [II], congruent auditory Stroop followed by congruent sentence [CC], and incongruent auditory Stroop followed by congruent sentence [IC]). Critical sentences were always preceded by auditory Stroop trials. At the end of each session, a five-item questionnaire was administered to assess overall well-being and fatigue. We conducted individual-level mixed-effects regressions on reaction times and growth curve analyses on the proportion of eye fixations to target pictures during incongruent sentences.

RESULTS

One participant showed conflict adaptation indicated by faster reaction times on active sentences and more rapid growth in fixations to target pictures on passive sentences in the II condition compared to the CI condition. Incongruent auditory Stroop also modulated active-sentence processing in an additional participant, as indicated by eye movements.

CONCLUSIONS

This is the first study to observe conflict adaptation in sentence comprehension in people with aphasia. The extent of adaptation varied across individuals. Eye tracking revealed subtler effects than overt behavioral measures. The results extend the study of conflict adaptation beyond neurotypical adults and suggest that upregulating cognitive control may be a potential treatment avenue for some individuals with aphasia.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.27056149.

Grammatical gender in spoken word recognition in school-age Spanish monolingual and Spanish-English bilingual children

This study examined grammatical gender processing in school-aged children with varying levels of cumulative English exposure. Children participated in a visual world paradigm with a four-picture display where they heard a gendered article followed by a target noun and were in the context where all images were the same gender (same gender), where all of the distractor images were the opposite gender than the target noun (different gender), and where all of the distractor images were the opposite gender, but there was a mismatch in the gendered article and target noun pair. We investigated 51 children (aged 5;0-10;0) who were exposed to Spanish since infancy but varied in their amount of cumulative English exposure. In addition to the visual word paradigm, all children completed an article-noun naming task, a grammaticality judgment task, and standardized vocabulary tests. Parents reported on their child's cumulative English language exposure and current English language use. To investigate the time course of lexical facilitation effects, looks to the target were analyzed with a cluster-based permutation test. The results revealed that all children used gender in a facilitatory way (during the noun region), and comprehension was significantly inhibited when the article-noun pairing was ungrammatical rather than grammatical. Compared to children with less cumulative English exposure, children with more cumulative English exposure looked at the target noun significantly less often overall, and compared to younger children, older children looked at the target noun significantly more often overall. Additionally, children with lower cumulative English exposure looked at target nouns more in the different-gender condition than the same-gender condition for masculine items more than feminine items.

Comparing Functional Trend and Learning among Groups in Intensive Binary Longitudinal Eye-Tracking Data using By-Variable Smooth Functions of GAMM

This paper presents a model specification for group comparisons regarding a functional trend over time within a trial and learning across a series of trials in intensive binary longitudinal eye-tracking data. The functional trend and learning effects are modeled using by-variable smooth functions. This model specification is formulated as a generalized additive mixed model, which allowed for the use of the freely available mgcv package (Wood in Package 'mgcv.' https://cran.r-project.org/web/packages/mgcv/mgcv.pdf , 2023) in R. The model specification was applied to intensive binary longitudinal eye-tracking data, where the questions of interest concern differences between individuals with and without brain injury in their real-time language comprehension and how this affects their learning over time. The results of the simulation study show that the model parameters are recovered well and the by-variable smooth functions are adequately predicted in the same condition as those found in the application.

Sex differences in social brain neural responses in autism: temporal profiles of configural face-processing within data-driven time windows

Face-processing timing differences may underlie visual social attention differences between autistic and non-autistic people, and males and females. This study investigates the timing of the effects of neurotype and sex on face-processing, and their dependence on age. We analysed EEG data during upright and inverted photographs of faces from 492 participants from the Longitudinal European Autism Project (141 neurotypical males, 76 neurotypical females, 202 autistic males, 73 autistic females; age 6-30 years). We detected timings of sex/diagnosis effects on event-related potential amplitudes at the posterior-temporal channel P8 with Bootstrapped Cluster-based Permutation Analysis and conducted Growth Curve Analysis (GCA) to investigate the timecourse and dependence on age of neural signals. The periods of influence of neurotype and sex overlapped but differed in onset (respectively, 260 and 310 ms post-stimulus), with sex effects lasting longer. GCA revealed a smaller and later amplitude peak in autistic female children compared to non-autistic female children; this difference decreased in adolescence and was not significant in adulthood. No age-dependent neurotype difference was significant in males. These findings indicate that sex and neurotype influence longer latency face processing and implicates cognitive rather than perceptual processing. Sex may have more overarching effects than neurotype on configural face processing.

What eye and hand movements tell us about expectations towards argument order: An eye- and mouse-tracking study in German

Previous research on real-time sentence processing in German has shown that listeners use the morphological marking of accusative case on a sentence-initial noun phrase to not only interpret the current argument as the object and patient, but also to predict a plausible agent. So far, less is known about the use of case marking to predict the semantic role of upcoming arguments after the subject/agent has been encountered. In the present study, we examined the use of case marking for argument interpretation in transitive as well as ditransitive structures. We aimed to control for multiple factors that could have influenced processing in previous studies, including the animacy of arguments, world knowledge, and the perceptibility of the case cue. Our results from eye- and mouse-tracking indicate that the exploitation of the first case cue that enables the interpretation of the unfolding sentence is influenced by (i) the strength of argument order expectation and (ii) the perceptual salience of the case cue. PsycINFO code: 2720 Linguistics & Language & Speech.

Fine-grained differences in gender-cue strength affect predictive processing in children: Cross-linguistic evidence from Russian and Bulgarian

We tested predictive gender agreement processing in adjective-noun phrases by 45 4- to 6-year-old Russian- and Bulgarian-speaking children using the visual world eye-tracking paradigm. Russian and Bulgarian are closely related languages that have three genders but differ in the nature and number of gender cues on adjectives. Analysis of the proportion and time course of looks to the target noun showed that only Bulgarian children used gender cues to predict the upcoming noun. We argue that the cross-linguistic difference in the gender cue strength is revealed through the operation of economy, transparency, and interdependence in a gender complexity matrix. The documented advantage for Bulgarian children in gender agreement processing and acquisition underscores the need for a comparative language acquisition approach to typologically close languages.

Online eye tracking and real-time sentence processing: On opportunities and efficacy for capturing psycholinguistic effects of different magnitudes and diversity

Online research methods have the potential to facilitate equitable accessibility to otherwise-expensive research resources, as well as to more diverse populations and language combinations than currently populate our studies. In psycholinguistics specifically, webcam-based eye tracking is emerging as a powerful online tool capable of capturing sentence processing effects in real time. The present paper asks whether webcam-based eye tracking provides the necessary granularity to replicate effects-crucially both large and small-that tracker-based eye tracking has shown. Using the Gorilla Experiment Builder platform, this study set out to replicate two psycholinguistic effects: a robust one, the verb semantic constraint effect, first reported in Altmann and Kamide,  Cognition 73(3), 247-264 (1999), and a smaller one, the lexical interference effect, first examined by Kukona et al. Journal of Experimental Psychology: Learning, Memory, and Cognition, 40(2), 326 (2014). Webcam-based eye tracking was able to replicate both effects, thus showing that its functionality is not limited to large effects. Moreover, the paper also reports two approaches to computing statistical power and discusses the differences in their outputs. Beyond discussing several important methodological, theoretical, and practical implications, we offer some further technical details and advice on how to implement webcam-based eye-tracking studies. We believe that the advent of webcam-based eye tracking, at least in respect of the visual world paradigm, will kickstart a new wave of more diverse studies with more diverse populations.

Modulating Complex Sentence Processing in Aphasia Through Attention and Semantic Networks

PURPOSE

Lexical processing impairments such as delayed and reduced activation of lexical-semantic information have been linked to syntactic processing disruptions and sentence comprehension deficits in individuals with aphasia (IWAs). Lexical-level deficits can also preclude successful lexical encoding during sentence processing and amplify the processing costs of similarity-based interference during syntactic retrieval. We investigate whether two manipulations to engage attention and pre-activate semantic features of a target (to-be-retrieved) noun will (a) boost lexical activation during initial lexical encoding and (b) facilitate syntactic dependency linking through improved resolution of interference in IWAs and neurologically unimpaired age-matched controls (AMCs).

METHOD

Eye-tracking-while-listening with a visual world paradigm was used to investigate whether semantic and attentional manipulations modulated initial lexical processing and downstream syntactic retrieval of the direct-object noun in object-relative sentences.

RESULTS

In the attention and semantic manipulations, the AMC group showed no changes in initial lexical access levels; however, gaze patterns revealed clear facilitations in dependency linking and interference resolution. In the IWA group, the attentional cue increased and maintained activation of N1 with modest facilitations in dependency linking. In the semantic condition, IWA results showed a greater degree of facilitation during dependency linking.

CONCLUSIONS

The results suggest that attention and semantic activation are parameters that may be manipulated to strengthen encoding of lexical representations to facilitate retrieval (i.e., dependency linking) and mitigate similarity-based interference. In IWAs, these manipulations may help to reduce lexical processing deficits that can preclude successful encoding.

Segmentation of Spoken Overlapping Ambiguity Strings in Chinese: An Eye-Tracking Study

PURPOSE

Using the printed-word paradigm with eye tracking, this study conducted three experiments to examine (a) how multiple words in spoken overlapping ambiguity strings (OASs) are activated, (b) how word frequency influences the word segmentation of spoken OASs, and (c) whether the multiple words in spoken OASs are activated competitively or independently.

METHOD

In this study, participants listened to a four-character spoken OAS (ABCD) and were presented with a visual display composed of a semantic associate of the "middle word" (BC; Experiments 1 and 2) or the "left word" (AB; Experiment 3) and two distractors. In Experiment 1, the word frequency of the middle words was manipulated to be higher than that of the neighbor words. In Experiment 2, the word frequency of the middle words was manipulated to be either higher or lower than that of the neighbor words. In Experiment 3, participants listened to either spoken OASs (ABCD) or spoken unambiguous strings (ABEF).

RESULTS

In Experiment 1, we observed a significant semantic competition effect; that is, more fixations fell on the semantic competitors than on distractors, suggesting that the semantic information of the middle words in the spoken OASs was activated. In Experiment 2, the semantic competition effect was only observed in the high-frequency condition and was absent in the low-frequency condition. In Experiment 3, the results showed significant semantic competition effects for the left words under both conditions, and the observed effect was similar between the ambiguity condition and the control condition.

CONCLUSIONS

These findings show that multiple words in spoken OASs are all activated and the activation level is modulated by word frequency. In addition, multiple words in the spoken OASs may be processed independently during spoken comprehension.