Using mouse cursor tracking to investigate online cognition: Preserving methodological ingenuity while moving toward reproducible science

Movement trajectories as a window into the dynamics of emerging neural representations

The rapid transformation of sensory inputs into meaningful neural representations is critical to adaptive human behaviour. While non-invasive neuroimaging methods are the de-facto method for investigating neural representations, they remain expensive, not widely available, time-consuming, and restrictive. Here we show that movement trajectories can be used to measure emerging neural representations with fine temporal resolution. By combining online computer mouse-tracking and publicly available neuroimaging data via representational similarity analysis (RSA), we show that movement trajectories track the unfolding of stimulus- and category-wise neural representations along key dimensions of the human visual system. We demonstrate that time-resolved representational structures derived from movement trajectories overlap with those derived from M/EEG (albeit delayed) and those derived from fMRI in functionally-relevant brain areas. Our findings highlight the richness of movement trajectories and the power of the RSA framework to reveal and compare their information content, opening new avenues to better understand human perception.

Temporal dynamics of costly avoidance in naturalistic fears: Evidence for sequential-sampling of fear and reward information

Excessive avoidance is characteristic for anxiety disorders, even when approach would lead to positive outcomes. The process of how such approach-avoidance conflicts are resolved is not sufficiently understood. We examined the temporal dynamics of approach-avoidance in intense fear of spiders. Highly fearful and non-fearful participants chose repeatedly between a fixed no spider/low reward and a spider/high reward option with varying fear (probability of spider presentation) and reward information (reward magnitude). By sequentially presenting fear and reward information, we distinguished whether decisions are dynamically driven by both information (sequential-sampling) or whether the impact of fear information is inhibited (cognitive control). Mouse movements were recorded to assess temporal decision dynamics (i.e., how strongly which information impacts decision preference at which timepoint). Highly fearful participants showed stronger avoidance despite lower gains (i.e., costly avoidance). Time-continuous multiple regression of their mouse movements yielded a stronger impact of fear compared to reward information. Importantly, presenting either information first (fear or reward) enhanced its impact during the early decision process. These findings support sequential sampling of fear and reward information, but not inhibitory control. Hence, pathological avoidance may be characterized by biased evidence accumulation rather than altered cognitive control.

TAT-HUM: Trajectory analysis toolkit for human movements in Python

Human movement trajectories can reveal useful insights regarding the underlying mechanisms of human behaviors. Extracting information from movement trajectories, however, can be challenging because of their complex and dynamic nature. The current paper presents a Python toolkit developed to help users analyze and extract meaningful information from the trajectories of discrete rapid aiming movements executed by humans. This toolkit uses various open-source Python libraries, such as NumPy and SciPy, and offers a collection of common functionalities to analyze movement trajectory data. To ensure flexibility and ease of use, the toolkit offers two approaches: an automated approach that processes raw data and generates relevant measures automatically, and a manual approach that allows users to selectively use different functions based on their specific needs. A behavioral experiment based on the spatial cueing paradigm was conducted to illustrate how one can use this toolkit in practice. Readers are encouraged to access the publicly available data and relevant analysis scripts as an opportunity to learn about kinematic analysis for human movements.

Age-related changes in motor planning for prior intentions: a mouse tracking reach-to-click task

When we complete sequential movements with different intentions, we plan our movements and adjust ahead. Such a phenomenon is called anticipatory planning for prior intentions and is known to decline with age. In daily life activities, we often need to consider and plan for multiple demands in one movement sequence. However, previous studies only considered one dimension of prior intentions, either different types of onward actions or different precisions of fit or placement. Therefore, in this study, we investigated anticipatory planning for both extrinsic (movement direction) and intrinsic (fit precision) target-related properties in a computer-based movement task and analyzed the computer cursor movement kinematics of both young and older adults. We found that older people consider and adjust for different properties step-by-step, with movement direction being considered as a prior intention during reach movement and fit precision as a motor constraint during drop movement. The age-related changes in the completion of onward actions are constrained by one's general cognitive ability, sensorimotor performance and effective motor planning for prior intentions. Age-related decline in motor planning can manifest as counterproductive movement profiles, resulting in suboptimal performance of intended actions.

How flexible is cognitive control? (Mouse) tracking conflict adaptation across context similarities

Exerting cognitive control to remain on-task and reach our goals is a crucial skill, as is the ability to flexibly adapt our responding in rapidly changing environments. The dynamics of cognitive control are typically studied by examining how participants process stimuli that contain competing relevant and irrelevant information in so-called conflict tasks. Adjustments in performance following the experience of conflict, also termed conflict adaptation, suggests a certain degree of flexibility in the deployment of cognitive control. The present study investigated to what extent conflict adaptation effects transfer across trials of the same and different tasks in three online mouse-tracking experiments. Adaptations of the Simon and Stroop tasks were combined to create different levels of context similarity between the paired tasks. Based on a previous review (Braem et al., Frontiers in Psychology 5:1-13, 2014), across-task conflict adaptation was expected only in the most and least similar contexts. In contrast to our hypothesis, conflict adaptation effects were observed in at least one measure in all three experiments. To our surprise, task order also seemed to impact the size of across-task conflict adaptation effects. The heterogeneity in the current results highlight the importance of using sensitive measurement tools to evaluate conflict adaptation and suggest that the occurrence of across-task conflict adaptation may be conditional on more than just shared relevant and irrelevant dimensions.

FAB: A "Dummy's" program for self-paced forward and backward reading

The self-paced reading paradigm has been popular and widely used in psycholinguistic research for several decades. The tool described in this paper, FAB (Forward and Backward reading), is a tool created to hopefully and maximally reduce the coding demands and simplify the operation costs for experimental researchers and clinical researchers who are doing experimental work, in their designing, coding, implementing, and analyzing self-paced reading tasks. Its basis in web languages (HTML, JavaScript) also promotes experimental implementation and material sharing in our era of open science. In addition, FAB has a unique forward-and-backward mode that can track regressive-like behaviors that are usually only recordable using eye-tracking or mouse-tracking equipment. In this paper, the specific application and usage of FAB is demonstrated in one laboratory and two online validation experiments. We hope this free and open-sourced tool can benefit research in a diverse range of contexts where self-paced reading is desirable.

The temporal development of memory processes in source monitoring: An investigation with mouse tracking

The present study investigated whether we first remember an item (e.g., a word itself) and then its source (e.g., position on the screen) or whether the retrieval of item and source information can (partially) overlap. Participants were tested on the source either in immediate sequence to item recognition (as standard in source-monitoring research) or following as a separate block after full completion of the item recognition test to separate these processes in time, providing a baseline. Using the mouse-tracking procedure during the item and source tests, we analyzed how item and source decisions unfolded qualitatively over time. Despite no significant difference in the aggregated trajectory curvatures, more thorough analyses based on the individual trajectories revealed differences across the test formats. In the standard format, trajectories were less curved in the source than in the item test. In contrast, in the blocked format, this difference was in the other direction with source showing more curved trajectories than item. Alternative interpretations of mouse-trajectory curvatures on the source-monitoring paradigm and what their difference may imply for item and source processing are discussed.

Exploring the impact of stimulus-stimulus and stimulus-response conflicts on computer mouse trajectories: continuous flow of information from stimulus encoding to response preparation to motor action

In recent years, mouse tracking (designing experiments in which participants provide responses via dynamic computer mouse movements) has enjoyed increasing experience in experimental psychology. Mouse-tracking studies typically involve some form of stimulus-response (S-R) conflict, and S-R effects emerge in movement trajectories (as well as in latencies). By contrast, it is currently unclear how stimulus-stimulus (S-S) compatibility affects movements. Here, we used a spatial arrow task which allowed us to generate S-R and S-S effects within the same experiment. Experiment 1 clarified in a key press experiment that this manipulation generates clear S-S and S-R effects in latencies. More critically, Experiment 2 demonstrated that both types of conflict impact mouse trajectories with incompatibility emerging as increased 'curvature' of responses when compared to congruent responses. We argue that these results are best explained via the assumption of 'continuous flow' of information, from stimulus encoding to response preparation and finally into motor action. By contrast, the S-S effect on trajectories contradicts the notion that processing is 'thresholded' between stimulus encoding and response preparation.

Mouse-cursor trajectories reveal reduced contextual influence on decision conflict during delay discounting in anorexia nervosa

OBJECTIVE

The capacity of individuals with anorexia nervosa (AN) to forgo immediate food rewards in their long-term pursuit of thinness is thought to reflect elevated self-control and/or abnormal reward sensitivity. Prior research attempted to capture an increased tendency to delay gratification in AN using delay-discounting tasks that assess how rapidly the subjective value of rewards decreases as a function of time until receipt. However, significant effects were mostly subtle or absent. Here, we tested whether the process leading to such decisions might be altered in AN.

METHOD

We recorded mouse-cursor movement trajectories leading to the final choice in a computerized delay-discounting task (238 trials) in 55 acutely underweight females with AN and pairwise age-matched female healthy controls (HC). We tested for group differences in deviations from a direct choice path, a measure of conflict strength in decision making, and whether group moderated the effect of several predictors of conflict strength (e.g., choice difficulty, consistency). We also explored reaction times and changes in trajectory directions (X-flips).

RESULTS

No group differences in delay-discounting parameters or movement trajectories were detected. However, the effect of the aforementioned predictors on deviations (and to a lesser extent reaction times) was reduced in AN.

DISCUSSION

These findings suggest that while delay discounting and conflict strength in decision making are generally unaltered in AN, conflict strength was more stable across different decisions in the disorder. This might enable individuals with AN to pursue (maladaptive) long-term body-weight goals, because particularly conflicting choices may not be experienced as such.

PUBLIC SIGNIFICANCE

The deviations from a direct path of mouse-cursor movements during a computerized delay-discounting task varied less in people with anorexia nervosa. Assuming such deviations measure decision conflict, we speculate that this increased stability might help people with anorexia nervosa achieve their long-term weight goals, as for them the struggle with the decision to eat high-calorie meals when hungry will be milder, so they would be more likely to skip them.

Temporal aspects of two types of backward crosstalk in dual-tasks: An analysis of continuous mouse-tracking data

A common explanation for processing limitations in dual-tasking is the existence of a bottleneck during response selection, meaning that the selection of responses can only occur serially for different tasks. However, a large body of data shows that features of a (secondary) Task 2 can already influence the processing of a (primary) Task 1. Such effects are referred to as backward crosstalk effects (BCEs). In the present study, two types of such BCEs were investigated: the compatibility-based BCE, which depends on the dimensional (often spatial) overlap between task features, and a BCE based on a go/no-go task in Task 2 (no-go BCE). Joining a line of research that suggests different mechanisms for these two types of BCEs, we investigated them using a mouse-tracking setup. Time continuous analyses revealed that the compatibility-based BCE triggered a spatial activation of the Task 2 response early during Task 1 processing, whereas the no-go BCE triggered an inhibitory effect in the case of a no-go Task 2, which spills over to Task 1 execution. This occurred, however, earlier in the time course than expected. The results are discussed with regard to recent models of dual-task processing.

Continuous cursor-captured conceptual competition: Investigating the spatiotemporal dynamics of spoken word comprehension

Semantically related concepts are coactivated during spoken word comprehension. Two internet-mediated cursor-tracking experiments examined the spatiotemporal dynamics of this coactivation. Participants viewed visual arrays containing images of a target (e.g., accordion) and a semantically related (e.g., banjo) or unrelated (e.g., plum) distractor whilst hearing the target word (e.g., "accordion"). Participants were tasked with moving their cursor from the bottom of the visual array to the target in one of the upper corners. In contrast to Experiment 1, the onset of stimulus presentation was triggered by cursor movement in Experiment 2. Across both experiments, temporal (e.g., RT) and spatial (e.g., AUC) measures revealed significantly greater attraction to images of semantically related compared with unrelated distractors. These results reveal that online cursor-tracking methods are sensitive to semantic competition and suitable for studying the activation of semantic knowledge during language comprehension.

Replacing vertical actions by mouse movements: a web-suited paradigm for investigating vertical spatial associations

The number of web-based studies in experimental psychology has been growing tremendously throughout the last few years. However, a straightforward web-based implementation does not exist for all types of experimental paradigms. In the current paper, we focus on how vertical response movements-which play a crucial role in spatial cognition and language research-can be translated into a web-based setup. Specifically, we introduce a web-suited counterpart of the vertical Stroop task (e.g., Fox & Shor, in Bull Psychon Soc 7:187-189, 1976; Lachmair et al., in Psychon Bull Rev 18:1180-1188, 2011; Thornton et al., in J Exp Psychol Hum Percept Perform 39:964-973, 2013). We employed nouns referring to entities typically located in lower or upper vertical space (e.g., "worm" and "bird", respectively) in Experiments 1 and 2, and emotional valence words associated with a crouched or an upward bodily posture (e.g., "sadness" and "excitement", respectively) in Experiment 3. Depending on the font color, our participants used their mouse to drag the words to the lower or upper screen location. Across all experiments, we consistently observed congruency effects analogous to those obtained with the lab paradigm using actual vertical arm movements. Consequently, we conclude that our web-suited paradigm establishes a reliable approach to examining vertical spatial associations.

Online mouse cursor trajectories distinguish phonological activation by linguistic and nonlinguistic sounds

Four online mouse cursor tracking experiments (total N = 208) examined the activation of phonological representations by linguistic and nonlinguistic auditory stimuli. Participants hearing spoken words (e.g., "bell") produced less direct mouse cursor trajectories toward corresponding pictures or text when visual arrays also included phonologically related competitors (e.g., belt) as compared with unrelated distractors (e.g., hose), but no such phonological competition was observed during environmental sounds (e.g., the ring of a bell). While important similarities have been observed between spoken words and environmental sounds, these experiments provide novel mouse cursor evidence that environmental sounds directly activate conceptual knowledge without needing to engage linguistic knowledge, contrasting with spoken words. Implications for theories of conceptual knowledge are discussed.

Effects of conflict in cognitive control: Evidence from mouse tracking

It has long been debated whether the "congruency sequence effect (CSE)" in conflict tasks such as Flanker could reflect adaptive control. The current study used "mouse tracking" to tackle the issue in a combination of three conflict tasks (i.e., Flanker, Simon, and Spatial Stroop tasks). Congruency effects from previous and current trials emerged in latencies as well as curvature of movement trajectories in all three tasks. Critically, movement initiation times were affected only by congruency on previous but not on current trials. A further analysis showed that even when initiation time on the previous trials was taken into account, a subtle but highly significant effect of conflict arising from trial N-1 on initiation times remained. Although not necessarily implying "conflict adaptation," i.e., a dynamic up- and downregulation of cognitive control in response to a recent conflict, our finding indicates a specific sensitivity to the presence or absence of recent "conflict" in the cognitive environment.

The 5I's of Virtual Technologies in Laboratory Teaching for Faculties of Higher Education in Kerala

In this paper, the effectiveness of training faculty in laboratory teaching (the teaching of science in a laboratory setting using experiments and similar exercises) through the use of Information and Communications Technology (ICT)-virtual technologies for faculties in institutions of higher education in the Indian state of Kerala-was evaluated and measured. The efficacy of employing ICT to train teachers in higher education is important, and we have identified 5I factors (innovative, interactive, involvement, informative, and influential) to help ascertain the effectiveness of such technology training during pandemic teaching. The laboratory learning using VL can describe the student's engagement in the online learning process. This work more specifically identifies how ICT helps in laboratory teaching and identifies the critical pedagogical aspects of the ICT. If the technology has these 5I factors, then it will be an effective teaching method for laboratory learning. Here, we used the ICT-virtual labs in science as the technology to evaluate these five factors. The research first began by conducting an ethnicity profile of science teachers in the middle and high/secondary stages of school consisting of classes VII, IX, and X (i.e., students of ages 11 to 15). To evaluate the use of VL in the 5I framework, the faculties in science were divided into experimental and control groups (n = 101). The experimental group practiced in a virtual lab in the first stage, but the control group did not. Test I was then performed on both groups. In the second stage, both groups practiced with real lab equipment, and test II was conducted on both groups. The tests and other data from the two groups were statistically analyzed using independent t tests. There were notable differences between the experimental and control groups: in terms of time for understanding the concepts behind the experiment, time for doing the experiment, and accuracy in results, with the experimental group performing significantly better. On the other hand, there was no significant difference between the two groups in task completion accuracy. Overall, there was a beneficial transfer of training from the virtual lab exercise to the real lab, with the experimental group's average score being higher.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1007/s10956-022-09995-8.

Costly avoidance of Pavlovian fear stimuli and the temporal dynamics of its decision process

Conflicts between avoiding feared stimuli versus approaching them for competing rewards are essential for functional behavior and anxious psychopathology. Yet, little is known about the underlying decision process. We examined approach-avoidance decisions and their temporal dynamics when avoiding Pavlovian fear stimuli conflicted with gaining rewards. First, a formerly neutral stimulus (CS+) was repeatedly paired with an aversive stimulus (US) to establish Pavlovian fear. Another stimulus (CS−) was never paired with the US. A control group received neutral tones instead of aversive USs. Next, in each of 324 trials, participants chose between a CS−/low reward and a CS+/high reward option. For the latter, probability of CS+ presentation (Pavlovian fear information) and reward magnitude (reward information) varied. Computer mouse movements were tracked to capture the decision dynamics. Although no more USs occurred, pronounced and persistent costly avoidance of the Pavlovian fear CS+ was found. Time-continuous multiple regression of movement trajectories revealed a stronger and faster impact of Pavlovian fear compared to reward information during decision-making. The impact of fear information, but not reward information, modestly decreased across trials. These findings suggest a persistently stronger weighting of fear compared to reward information during approach-avoidance decisions, which may facilitate the development of pathological avoidance.

A method for estimating the time of initiating correct categorization in mouse-tracking

Mouse-tracking facilitates exploration of the mental processes underlying decision-making. As the cognitive system works to settle on a decision, response competition manifests in the motor movements of the hand, bringing the mouse relatively closer to one alternative versus the other. Many metrics provide insight into decision-making processes by indexing the shape or complexity of the mouse trajectory. Lacking, however, is a metric that estimates the point in time when a participant begins to correctly categorize a stimulus. We rectify this absence by introducing a metric we refer to as time of initiating correct categorization (TICC), which is the point in time when people began moving relatively closer to the selected target relative to the distractor. We briefly review existing approaches to measuring time in mouse-tracking before describing the TICC and demonstrating its utility in three data sets.