Fixation duration surpasses pupil size as a measure of memory load in free viewing

Saccade size predicts onset time of object processing during visual search of an open world virtual environment

OBJECTIVE

To date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component.

APPROACH

The current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems.

MAIN RESULTS

The results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods.

SIGNIFICANCE

The results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.

Using eye tracking to assess learning of a multifunction prosthetic hand: an exploratory study from a rehabilitation perspective

BACKGROUND

Eye tracking technology not only reveals the acquisition of visual information at fixation but also has the potential to unveil underlying cognitive processes involved in learning to use a multifunction prosthetic hand. It also reveals gaze behaviours observed during standardized tasks and self-chosen tasks. The aim of the study was to explore the use of eye tracking to track learning progress of multifunction hands at two different time points in prosthetic rehabilitation.

METHODS

Three amputees received control training of a multifunction hand with new control strategy. Detailed description of control training was collected first. They wore Tobii Pro2 eye-tracking glasses and performed a set of standardized tasks (required to switch to different grips for each task) after one day of training and at one-year-follow-up (missing data for Subject 3 at the follow up due to socket problem). They also performed a self-chosen task (free to use any grip for any object) and were instructed to perform the task in a way how they would normally do at home. The gaze-overlaid videos were analysed using the Tobii Pro Lab and the following metrics were extracted: fixation duration, saccade amplitude, eye-hand latency, fixation count and time to first fixation.

RESULTS

During control training, the subjects learned 3 to 4 grips. Some grips were easier, and others were more difficult because they forgot or were confused with the switching strategies. At the one-year-follow-up, a decrease in performance time, fixation duration, eye-hand latency, and fixation count was observed in Subject 1 and 2, indicating an improvement in the ability to control the multifunction hand and a reduction of cognitive load. An increase in saccade amplitude was observed in both subjects, suggesting a decrease in difficulty to control the prosthetic hand. During the standardized tasks, the first fixation of all three subjects were on the multifunction hand in all objects. During the self-chosen tasks, the first fixations were mostly on the objects first.

CONCLUSION

The qualitative data from control training and the quantitative eye tracking data from clinical standardized tasks provided a rich exploration of cognitive processing in learning to control a multifunction hand. Many prosthesis users prefer multifunction hands and with this study we have demonstrated that a targeted prosthetic training protocol with reliable assessment methods will help to lay the foundation for measuring functional benefits of multifunction hands.

Just-in-Time Encoding Into Visual Working Memory Is Contingent Upon Constant Availability of External Information

Humans maintain an intricate balance between storing information in visual working memory (VWM) and just-in-time sampling of the external world, rooted in a trade-off between the cost of maintaining items in VWM versus retrieving information as it is needed. Previous studies have consistently shown that one prerequisite of just-in-time sampling is a high degree of availability of external information, and that introducing a delay before being able to access information led participants to rely less on the external world and more on VWM. However, these studies manipulated availability in such a manner that the cost of sampling was stable and predictable. It is yet unclear whether participants become less reliant on external information when it is more difficult to factor in the cost of sampling that information. In two experiments, participants copied an example layout from the left to the right side of the screen. In Experiment 1, intermittent occlusion of the example layout led participants to attempt to encode more items per inspection than when the layout was constantly available, but this did not consistently result in more correct placements. However, these findings could potentially be explained by inherent differences in how long the example layout could be viewed. Therefore in Experiment 2, the example layout only became available after a gaze-contingent delay, which could be constant or variable. Here, the introduction of any delay led to increased VWM load compared to no delay, although the degree of variability in the delay did not alter behaviour. These results reaffirm that the nature of when we engage VWM is dynamical, and suggest that any disruption to the continuous availability of external information is the main driver of increased VWM usage relative to whether availability is predictable or not.

An updating-based working memory load alters the dynamics of eye movements but not their spatial extent during free viewing of natural scenes

The relationship between spatial deployments of attention and working memory load is an important topic of study, with clear implications for real-world tasks such as driving. Previous research has generally shown that attentional breadth broadens under higher load, while exploratory eye-movement behaviour also appears to change with increasing load. However, relatively little research has compared the effects of working memory load on different kinds of spatial deployment, especially in conditions that require updating of the contents of working memory rather than simple retrieval. The present study undertook such a comparison by measuring participants' attentional breadth (via an undirected Navon task) and their exploratory eye-movement behaviour (a free-viewing recall task) under low and high updating working memory loads. While spatial aspects of task performance (attentional breadth, and peripheral extent of image exploration in the free-viewing task) were unaffected by the load manipulation, the exploratory dynamics of the free-viewing task (including fixation durations and scan-path lengths) changed under increasing load. These findings suggest that temporal dynamics, rather than the spatial extent of exploration, are the primary mechanism affected by working memory load during the spatial deployment of attention. Further, individual differences in exploratory behaviour were observed on the free-viewing task: all metrics were highly correlated across working memory load blocks. These findings suggest a need for further investigation of individual differences in eye-movement behaviour; potential factors associated with these individual differences, including working memory capacity and persistence versus flexibility orientations, are discussed.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Amyloid-β Deposition Predicts Grocery Shopping Performance in Older Adults Without Cognitive Impairment

Background

A screening tool sensitive to Alzheimer's disease (AD) risk factors, such as amyloid-β (Aβ) deposition, and subtle cognitive changes, best elicited by complex everyday tasks, is needed.

Objective

To determine if grocery shopping performance could differentiate older adults at elevated risk of developing AD (OAer), older adults at low risk of developing AD (OAlr), and young adults (YA), and if amount of Aβ deposition could predict grocery shopping performance in older adults (OA).

Methods

Twenty-one OAer (78±5 years), 33 OAlr (78±5 years), and 28 YA (31±3 years) performed four grocery shopping trials, with the best and worst performances analyzed. Measures included trial time, number of correct items, number of grocery note fixations, and number of fixations and percentage of time fixating on the correct shelving unit, correct brand, and correct shelf. Linear mixed effects models compared measures by performance rank (best, worst) and group (OAer, OAlr, YA), and estimated the effect of Aβ deposition on measures in OA.

Results

Relative to their best performance, OAer and OAlr exhibited more correct shelving unit fixations and correct brand fixations during their worst performance, while YA did not. Within OA's worst performance, greater Aβ deposition was associated with a smaller percentage of time fixating on the correct shelving unit, correct shelf, and correct brand. Within OA, greater Aβ deposition was associated with more grocery note fixations.

Conclusions

OA with elevated Aβ deposition may exhibit subtle working memory impairments and less efficient visual search strategies while performing a cognitively demanding everyday task.

A time slice analysis of dentistry students' visual search strategies and pupil dilation during diagnosing radiographs

Diagnosing orthopantomograms (OPTs: panoramic radiographs) is an essential skill dentistry students acquire during university training. While prior research described experts' visual search behavior in radiology as global-to-focal for chest radiographs and mammography, generalizability to a hybrid search task in OPTs (i.e., searching for multiple, diverse anomalies) remains unclear. Addressing this gap, this study investigated visual search of N = 107 dentistry students while they were diagnosing anomalies in OPTs. Following a global-to-focal expert model, we hypothesized that students would use many, short fixations representing global search in earlier stages, and few, long fixations representing focal search in later stages. Furthermore, pupil dilation and mean fixation duration served as cognitive load measures. We hypothesized that later stages would be characterized by elaboration and a reflective search strategy, leading to higher cognitive load being associated with higher diagnostic performance in late compared to earlier stages. In line with the first hypothesis, students' visual search comprised of a three-stage process that grew increasingly focal in terms of the number of fixations and anomalies fixated. Contrary to the second hypothesis, mean fixation duration during anomaly fixations was positively associated with diagnostic performance across all stages. As OPTs greatly varied in how difficult it was to identify the anomalies contained therein, OPTs with above-average difficulty were sampled for exploratory analysis. Pupil dilation predicted diagnostic performance for difficult OPTs, possibly capturing elaborative cognitive processes and cognitive load compared to mean fixation duration. A visual analysis of fine-grained time slices indicated large cognitive load differences towards the end of trials, showcasing a richness-resolution-trade-off in data sampling crucial for future studies using time-slicing of eye tracking data.

Differential aspects of attention predict the depth of visual working memory encoding: Evidence from pupillometry

What determines how much one encodes into visual working memory? Traditionally, encoding depth is considered to be indexed by spatiotemporal properties of gaze, such as gaze position and dwell time. Although these properties inform about where and how long one looks, they do not necessarily inform about the current arousal state or how strongly attention is deployed to facilitate encoding. Here, we found that two types of pupillary dynamics predict how much information is encoded during a copy task. The task involved encoding a spatial pattern of multiple items for later reproduction. Results showed that smaller baseline pupil sizes preceding and stronger pupil orienting responses during encoding predicted that more information was encoded into visual working memory. Additionally, we show that pupil size reflects not only how much but also how precisely material is encoded. We argue that a smaller pupil size preceding encoding is related to increased exploitation, whereas larger pupil constrictions signal stronger attentional (re)orienting to the to-be-encoded pattern. Our findings support the notion that the depth of visual working memory encoding is the integrative outcome of differential aspects of attention: how alert one is, how much attention one deploys, and how long it is deployed. Together, these factors determine how much information is encoded into visual working memory.

Eye Gaze Patterns during Reasoning Provide Insights Regarding Individual Differences in Underlying Cognitive Abilities

Sequences of eye movements during performance of a reasoning task has provided insights into the strategies individuals use to solve that specific task; however, prior studies have not examined whether eye gaze metrics reflect cognitive abilities in a way that transcends a specific task. Thus, our study aimed to explore the relationship between eye movement sequences and other behavioral measures. Here, we present two studies that related different eye gaze metrics in a matrix reasoning task with performance on a different test of fluid reasoning and tests of planning, working memory, and cognitive flexibility. Additionally, we related gaze metrics to self-reported executive functioning in daily life, as measured by BRIEF-A. To perform this, we classified the participants' eye gaze in each item of the matrices test using an algorithm and then used LASSO regression models with the cognitive abilities as the dependent variable to select eye-tracking metrics to predict it. Overall, specific and distinct eye gaze metrics predicted 57% variance in the fluid reasoning scores; 17% variance in the planning scores; and 18% variance in the working memory scores. Taken together, these results support the hypothesis that the selected eye-tracking metrics reflect cognitive abilities that transcend specific tasks.

EEG and pupillometric signatures of working memory overload

Understanding the physiological correlates of cognitive overload has implications for gauging the limits of human cognition, developing novel methods to define cognitive overload, and mitigating the negative outcomes associated with overload. Most previous psychophysiological studies manipulated verbal working memory load in a narrow range (an average load of 5 items). It is unclear, however, how the nervous system responds to a working memory load exceeding typical capacity limits. The objective of the current study was to characterize the central and autonomic nervous system changes associated with memory overload, by means of combined recording of electroencephalogram (EEG) and pupillometry. Eighty-six participants were presented with a digit span task involving the serial auditory presentation of items. Each trial consisted of sequences of either 5, 9, or 13 digits, each separated by 2 s. Both theta activity and pupil size, after the initial rise, expressed a pattern of a short plateau and a decrease with reaching the state of memory overload, indicating that pupil size and theta possibly have similar neural mechanisms. Based on the described above triphasic pattern of pupil size temporal dynamics, we concluded that cognitive overload causes physiological systems to reset, and release effort. Although memory capacity limits were exceeded and effort was released (as indicated by pupil dilation), alpha continued to decrease with increasing memory load. These results suggest that associating alpha with the focus of attention and distractor suppression is not warranted.

Enhanced cognitive interference during visuomotor tasks may cause eye-hand dyscoordination

In complex visuomotor tasks, such as cooking, people make many saccades to continuously search for items before and during reaching movements. These tasks require cognitive resources, such as short-term memory and task-switching. Cognitive load may impact limb motor performance by increasing demands on mental processes, but mechanisms remain unclear. The Trail-Making Tests, in which participants sequentially search for and make reaching movements to 25 targets, consist of a simple numeric variant (Trails-A) and a cognitively challenging variant that requires alphanumeric switching (Trails-B). We have previously shown that stroke survivors and age-matched controls make many more saccades in Trails-B, and those increases in saccades are associated with decreases in speed and smoothness of reaching movements. However, it remains unclear how patients with neurological injuries, e.g., stroke, manage progressive increases in cognitive load during visuomotor tasks, such as the Trail-Making Tests. As Trails-B trial progresses, switching between numbers and letters leads to progressive increases in cognitive load. Here, we show that stroke survivors with damage to frontoparietal areas and age-matched controls made more saccades and had longer fixations as they progressed through the 25 alphanumeric targets in Trails-B. Furthermore, when stroke survivors made saccades during reaching movements in Trails-B, their movement speed slowed down significantly. Thus, damage to frontoparietal areas serving cognitive motor functions may cause interference between oculomotor, visual, and limb motor functions, which could lead to significant disruptions in activities of daily living. These findings augment our understanding of the mechanisms that underpin cognitive-motor interference during complex visuomotor tasks.

Memory-driven eye movements prospectively predict dementia in people at risk of Alzheimer's disease

Introduction

Oculomotor behaviors linked to cognitive performance revealed neurocognitive features of Alzheimer's disease (AD) that can enhance the accuracy of its assessment and diagnosis.

Methods

A sample of 107 participants (i.e., 65 mild cognitive impairment [MCI] and 42 controls) were recruited and followed up for 40 months. At baseline, they underwent assessment with the ViewMind digital biomarker, which draws cognitive-related patterns of eye movement while people perform the visual short-term memory binding task.

Results

Baseline data predicted that 36 patients with MCI would progress to the AD clinical syndrome (ADS Progressing). The remaining 29 MCI patients were predicted to remain as MCI or progress to other forms of dementia. After 40 months of follow-up, 94% of ADS Progressing patients had received a diagnosis of dementia, whereas none of the non-ADS Progressing had.

Discussion

The analysis of eye movement behavior combined with cognitive markers for AD can effectively predict progression to ADS among patients with MCI.

Average fixation duration in infancy: Stability and predictive utility

The current study examined the stability, consistency, and predictive utility of average fixation durations in infancy. In Study 1, infants' (N = 80) average fixation duration when viewing social stimuli was found to show strong relative stability from 3.5 to 9 months of age. In Study 2, strong within-infant consistency was found in 3.5-month-old infants' (N = 73) average fixation durations to social and nonsocial stimuli. In Study 3, 3.5- to 9-month-old infants' (N = 89) average fixation duration was found to systematically vary with parent-reported symptoms of attention deficit hyperactivity disorder (ADHD) at 4-6 years of age. These results suggest that average fixation duration serves as a stable and systematic measure of individual differences in cognitive development beginning early in life.

Do your eye movements reveal your performance on an IQ test? A study linking eye movements and socio-demographic information to fluid intelligence

Understanding the main factors contributing to individual differences in fluid intelligence is one of the main challenges of psychology. A vast body of research has evolved from the theoretical framework put forward by Cattell, who developed the Culture-Fair IQ Test (CFT 20-R) to assess fluid intelligence. In this work, we extend and complement the current state of research by analysing the differential and combined relationship between eye-movement patterns and socio-demographic information and the ability of a participant to correctly solve a CFT item. Our work shows that a participant’s eye movements while solving a CFT item contain discriminative information and can be used to predict whether the participant will succeed in solving the test item. Moreover, the information related to eye movements complements the information provided by socio-demographic data when it comes to success prediction. In combination, both types of information yield a significantly higher predictive performance than each information type individually. To better understand the contributions of features related to eye movements and socio-demographic information to predict a participant’s success in solving a CFT item, we employ state-of-the-art explainability techniques and show that, along with socio-demographic variables, eye-movement data. Especially the number of saccades and the mean pupil diameter, significantly increase the discriminating power. The eye-movement features are likely indicative of processing efficiency and invested mental effort. Beyond the specific contribution to research on how eye movements can serve as a means to uncover mechanisms underlying cognitive processes, the findings presented in this work pave the way for further in-depth investigations of factors predicting individual differences in fluid intelligence.

Pupillary correlates of individual differences in long-term memory

The present study is the first to examine individual differences in long-term memory, arousal dysregulation, and intensity of attention within the same experiment. Participants (N = 106) completed 28 lists of an immediate free-recall task while their pupil diameter was recorded via an eye-tracker during the encoding period. Two main pupillary measures were extracted: intraindividual variability in pre-list pupil diameter and evoked pupillary responses during item encoding. Variability in pre-list pupil diameter served as a measure of arousal dysregulation, and evoked pupillary responses served as a measure of intensity of attention. Based on prior work, we hypothesized that there would be a positive association between intensity of attention and recall ability, and that there would be a negative association between arousal dysregulation and recall ability. Collectively these two measures accounted for 19% of interindividual variance in recall, with 5% attributable uniquely to intensity of attention and 12% attributable uniquely to arousal regulation. The findings demonstrate that there are sources of individual differences in long-term memory that can be revealed via pupillometry, notably the amount of effort deployed during item encoding and the degree to which people exhibit dysregulated arousal. Both findings are consistent with recent theorizing regarding the role of the locus coeruleus (LC)-norepinephrine (NE) system's role in goal-directed cognition. Specifically, the LC governs both moment-to-moment arousal and NE release to cortical regions subserving cognitive processing. Among people for whom this system operates most optimally, long-term memory retention is superior.

Effects of pointing movements on visuospatial working memory in a joint-action condition: Evidence from eye movements

Previous studies showed that (a) performing pointing movements towards to-be-remembered locations enhanced their later recognition, and (b) in a joint-action condition, experimenter-performed pointing movements benefited memory to the same extent as self-performed movements. The present study replicated these findings and additionally recorded participants’ fixations towards studied arrays. Each trial involved the presentation of two consecutive spatial arrays, where each item occupied a different spatial location. The item locations of one array were encoded by mere visual observation (the no-move array), whereas the locations of the other array were encoded by observation plus pointing movements (the move array). Critically, in Experiment 1, participants took turns with the experimenter in pointing towards the move arrays (joint-action condition), while in Experiment 2 pointing was performed only by the experimenter (passive condition). The results showed that the locations of move arrays were recognized better than the locations of no-move arrays in Experiment 1, but not in Experiment 2. The pattern of eye-fixations was in line with behavioral findings, indicating that in Experiment 1, fixations to the locations of move arrays were higher in number and longer in duration than fixations to the locations of no-move arrays, irrespective of the agent who performed the movements. In contrast, no differences emerged in Experiment 2. We propose that, in the joint-action condition, self- and other-performed pointing movements are coded at the same representational level and their functional equivalency is reflected in a similar pattern of eye-fixations.

Gaze Behavior During Navigation and Visual Search of an Open-World Virtual Environment

Eye tracking has been an essential tool within the vision science community for many years. However, the majority of studies involving eye-tracking technology employ a relatively passive approach through the use of static imagery, prescribed motion, or video stimuli. This is in contrast to our everyday interaction with the natural world where we navigate our environment while actively seeking and using task-relevant visual information. For this reason, an increasing number of vision researchers are employing virtual environment platforms, which offer interactive, realistic visual environments while maintaining a substantial level of experimental control. Here, we recorded eye movement behavior while subjects freely navigated through a rich, open-world virtual environment. Within this environment, subjects completed a visual search task where they were asked to find and count occurrence of specific targets among numerous distractor items. We assigned each participant into one of four target conditions: Humvees, motorcycles, aircraft, or furniture. Our results show a statistically significant relationship between gaze behavior and target objects across Target Conditions with increased visual attention toward assigned targets. Specifically, we see an increase in the number of fixations and an increase in dwell time on target relative to distractor objects. In addition, we included a divided attention task to investigate how search changed with the addition of a secondary task. With increased cognitive load, subjects slowed their speed, decreased gaze on objects, and increased the number of objects scanned in the environment. Overall, our results confirm previous findings and support that complex virtual environments can be used for active visual search experimentation, maintaining a high level of precision in the quantification of gaze information and visual attention. This study contributes to our understanding of how individuals search for information in a naturalistic (open-world) virtual environment. Likewise, our paradigm provides an intriguing look into the heterogeneity of individual behaviors when completing an un-timed visual search task while actively navigating.

Increased pupil dilation during tip-of-the-tongue states

Tip-of-the-tongue states (TOTs) are feelings of impending word retrieval success during a current failure to retrieve a target word. Though much is known and understood about TOT states from decades of research, research on potential psychophysiological correlates of the TOT state is still in its infancy, and existing studies point toward the involvement of neural processes that are associated with enhanced attention, motivation, and information-seeking. In the present study, we demonstrate that, during instances of target retrieval failure, TOT states are associated with greater pupillary dilation (i.e., autonomic arousal) in 91% of our sample. This is the first study to demonstrate a pupillometric correlate of the TOT experience, and this finding provides an important step toward understanding emotional attributes associated with TOT states. Mean pupil dilation also increased such that instances of target identification failure that were unaccompanied by TOT states < instances in which TOTs occurred < instances of target identification success. It is possible that TOTs reflect an intermediary state between complete target retrieval failure and full target retrieval.

Eye-movements reveal semantic interference effects during the encoding of naturalistic scenes in long-term memory

Similarity-based semantic interference (SI) hinders memory recognition. Within long-term visual memory paradigms, the more scenes (or objects) from the same semantic category are viewed, the harder it is to recognize each individual instance. A growing body of evidence shows that overt attention is intimately linked to memory. However, it is yet to be understood whether SI mediates overt attention during scene encoding, and so explain its detrimental impact on recognition memory. In the current experiment, participants watched 372 photographs belonging to different semantic categories (e.g., a kitchen) with different frequency (4, 20, 40 or 60 images), while being eye-tracked. After 10 minutes, they were presented with the same 372 photographs plus 372 new photographs and asked whether they recognized (or not) each photo (i.e., old/new paradigm). We found that the more the SI, the poorer the recognition performance, especially for old scenes of which memory representations existed. Scenes more widely explored were better recognized, but for increasing SI, participants focused on more local regions of the scene in search for its potentially distinctive details. Attending to the centre of the display, or to scene regions rich in low-level saliency was detrimental to recognition accuracy, and as SI increased participants were more likely to rely on visual saliency. The complexity of maintaining faithful memory representations for increasing SI also manifested in longer fixation durations; in fact, a more successful encoding was also associated with shorter fixations. Our study highlights the interdependence between attention and memory during high-level processing of semantic information.

Applying Eye Movement Modeling Examples to Guide Novices' Attention in the Comprehension of Process Models

Process models are crucial artifacts in many domains, and hence, their proper comprehension is of importance. Process models mediate a plethora of aspects that are needed to be comprehended correctly. Novices especially face difficulties in the comprehension of process models, since the correct comprehension of such models requires process modeling expertise and visual observation capabilities to interpret these models correctly. Research from other domains demonstrated that the visual observation capabilities of experts can be conveyed to novices. In order to evaluate the latter in the context of process model comprehension, this paper presents the results from ongoing research, in which gaze data from experts are used as Eye Movement Modeling Examples (EMMEs) to convey visual observation capabilities to novices. Compared to prior results, the application of EMMEs improves process model comprehension significantly for novices. Novices achieved in some cases similar performances in process model comprehension to experts. The study’s insights highlight the positive effect of EMMEs on fostering the comprehension of process models.

There Is an "Eye" in Team: Exploring the Interplay Between Emotion, Gaze Behavior, and Collective Efficacy in Team Sport Settings

Little is understood about the attentional mechanisms that lead to perceptions of collective efficacy. This paper presents two studies that address this lack of understanding. Study one examined participant's (N = 59) attentional processes relating to positive, neutral, or negative emotional facial photographs, when instructed to select their “most confident” or “least confident” team. Eye gaze metrics of first fixation duration (FFD), fixation duration (FD), and fixation count (FC) were measured alongside individual perceptions of collective efficacy and emotional valence of the teams selected. Participants had shorter FFD, longer FD, and more FC on positive faces when instructed to select their most confident team (p < 0.05). Collective efficacy and emotional valence were significantly greater when participants selected their most confident team (p < 0.05). Study two explored the influence of video content familiarity of team-based observation interventions on attentional processes and collective efficacy in interdependent team-sport athletes (N = 34). When participants were exposed to familiar (own team/sport) and unfamiliar (unknown team/sport) team-based performance video, eye tracking data revealed similar gaze behaviors for the two conditions in terms of areas of interest. However, collective efficacy increased most for the familiar condition. Study one results indicate that the emotional expressions of team members influence both where and for how long we look at potential team members, and that conspecifics' emotional expression impacts on our perceptions of collective efficacy. For Study two, given the apparent greater increase in collective efficacy for the familiar condition, the similar attentional processes evident for familiar and unfamiliar team footage suggests that differences in meaning of the observed content dictates collective efficacy perceptions. Across both studies, the findings indicate the importance of positive emotional vicarious experiences when using team-based observation interventions to improve collective efficacy in teams.

When Natural Behavior Engages Working Memory

Working memory (WM) enables temporary storage and manipulation of information,¹ supporting tasks that require bridging between perception and subsequent behavior. Its properties, such as its capacity, have been thoroughly investigated in highly controlled laboratory tasks.1, 2, 3, 4, 5, 6, 7, 8 Much less is known about the utilization and properties of WM in natural behavior,9, 10, 11 when reliance on WM emerges as a natural consequence of interactions with the environment. We measured the trade-off between reliance on WM and gathering information externally during immersive behavior in an adapted object-copying task.¹² By manipulating the locomotive demands required for task completion, we could investigate whether and how WM utilization changed as gathering information from the environment became more effortful. Reliance on WM was lower than WM capacity measures in typical laboratory tasks. A clear trade-off also occurred. As sampling information from the environment required increasing locomotion and time investment, participants relied more on their WM representations. This reliance on WM increased in a shallow and linear fashion and was associated with longer encoding durations. Participants’ avoidance of WM usage showcases a fundamental dependence on external information during ecological behavior, even if the potentially storable information is well within the capacity of the cognitive system. These foundational findings highlight the importance of using immersive tasks to understand how cognitive processes unfold within natural behavior. Our novel VR approach effectively combines the ecological validity, experimental rigor, and sensitive measures required to investigate the interplay between memory and perception in immersive behavior.

Video Abstract

•

Gaze provides a measure of working-memory (WM) usage during natural behavior

•

Natural reliance on WM is low even when searching for objects externally is effortful

•

WM utilization increases linearly as searching for objects requires more locomotion

•

The trade-off between using WM versus external sampling affects performance

The usability of ventilator maintenance user interface: A comparative evaluation of user task performance, workload, and user experience

Poor usability designed of ventilator user interface can easily lead to human error. In this study, we evaluated the usability design of ventilator maintenance user interface and identified problems related to the usability of user interface that could easily cause human error. Sixteen respiratory therapists participated in this usability study. The usability of the ventilator maintenance user interface was evaluated by participants' task performance (task completion time, task error rate), physiological workload (eye-fixation duration) and perceived workload (NASA-TLX), and user experience (questionnaire). For task performance, task completion time and task error rate showed significant differences. For task completion time, significant difference was found when conducting ventilator self-test (p < 0.001), replace the breathing circuit (p = 0.047), and check battery status (p = 0.005). For task error rate, the three ventilators showed significant difference (p = 0.012), and the Serov I showed a significantly higher task error rate than the Boaray 5000D (p = 0.031). For workload, the Serov I was associated with higher physiological and perceived workloads than other ventilators (p < 0.05). For user experience, the Boaray 5000D received better scores among the ventilators in terms of ease to maintain, friendly to maintain, and willingness to use (p < 0.05, respectively). Our study adds available literature for usability evaluation of ventilator maintenance user interface. The results indicate that the maintenance user interface of the Boaray 5000D performed better than the other two tested ventilators. Moreover, the study results also proved that eye-fixation duration can be a reliable tool for evaluating the usability of ventilator user interface.

Neural correlates of task-related refixation behavior

Eye movement research has shown that attention shifts from the currently fixated location to the next before a saccade is executed. We investigated whether the cost of the attention shift depends on higher-order processing at the time of fixation, in particular on visual working memory load differences between fixations and refixations on task-relevant items. The attention shift is reflected in EEG activity in the saccade-related potential (SRP). In a free viewing task involving visual search and memorization of multiple targets amongst distractors, we compared the SRP in first fixations versus refixations on targets and distractors. The task-relevance of targets implies that more information will be loaded in memory (e.g. both identity and location) than for distractors (e.g. location only). First fixations will involve greater memory load than refixations, since first fixations involve loading of new items, while refixations involve rehearsal of previously visited items. The SRP in the interval preceding the saccade away from a target or distractor revealed that saccade preparation is affected by task-relevance and refixation behavior. For task-relevant items only, we found longer fixation duration and higher SRP amplitudes for first fixations than for refixations over the occipital region and the opposite effect over the frontal region. Our findings provide first neurophysiological evidence that working memory loading of task-relevant information at fixation affects saccade planning.

Component processes in free-viewing visual search: Insights from fixation-aligned pupillary response averaging

Pupil size changes during a visual search may reflect cognitive processes, such as effort and memory accumulation, but methodological confounds and the general lack of literature in this area leave the reliability of findings open to question. We used a novel synthesis of experimental methods and averaging techniques to explore how cognitive processing unfolds during free-viewing visual search for multiple targets. Twenty-seven participants completed 152 searches across two separate 1-hour sessions. The number of targets present (Targets: 0, 1, 2, and 3) in each trial was the main manipulation and the task was to "find all of the targets" and report the total via mouse-click at the end of the trial. Search time lasted for 10 seconds or until the participant purported to have found all of the targets, in which case they could terminate the search via keypress. Whole-trial pupil analysis revealed a significant effect of button pressing as well as a significant main effect of targets for trials that were not self-terminated via button press. Fixation-aligned pupil responses revealed transient modulations in pupil size following initial fixations on targets but not distractors and refixations on both targets and distractors. Owing to rigorous control over experimental confounds and a detailed analysis and correction of eye-movement-related measurement error, we confidently discuss these findings in terms of task-related processing and underlying brain activity.

No Advantage for Separating Overt and Covert Attention in Visual Search

We move our eyes roughly three times every second while searching complex scenes, but covert attention helps to guide where we allocate those overt fixations. Covert attention may be allocated reflexively or voluntarily, and speeds the rate of information processing at the attended location. Reducing access to covert attention hinders performance, but it is not known to what degree the locus of covert attention is tied to the current gaze position. We compared visual search performance in a traditional gaze-contingent display, with a second task where a similarly sized contingent window is controlled with a mouse, allowing a covert aperture to be controlled independently by overt gaze. Larger apertures improved performance for both the mouse- and gaze-contingent trials, suggesting that covert attention was beneficial regardless of control type. We also found evidence that participants used the mouse-controlled aperture somewhat independently of gaze position, suggesting that participants attempted to untether their covert and overt attention when possible. This untethering manipulation, however, resulted in an overall cost to search performance, a result at odds with previous results in a change blindness paradigm. Untethering covert and overt attention may therefore have costs or benefits depending on the task demands in each case.

Refixation patterns reveal memory-encoding strategies in free viewing

We investigated visual working memory encoding across saccadic eye movements, focusing our analysis on refixation behavior. Over 10-s periods, participants performed a visual search for three, four, or five targets and remembered their orientations for a subsequent change-detection task. In 50% of the trials, one of the targets had its orientation changed. From the visual search period, we scored three types of refixations and applied measures for quantifying eye-fixation recurrence patterns. Repeated fixations on the same regions as well as repeated fixation patterns increased with memory load. Correct change detection was associated with more refixations on targets and less on distractors, with increased frequency of recurrence, and with longer intervals between refixations. The results are in accordance with the view that patterns of eye movement are an integral part of visual working memory representation.

Maintaining fixation does not increase demands on working memory relative to free viewing

The comparison of memory performance during free and fixed viewing conditions has been used to demonstrate the involvement of eye movements in memory encoding and retrieval, with stronger effects at encoding than retrieval. Relative to conditions of free viewing, participants generally show reduced memory performance following sustained fixation, suggesting that unrestricted eye movements benefit memory. However, the cognitive basis of the memory reduction during fixed viewing is uncertain, with possible mechanisms including disruption of visual-mnemonic and/or imagery processes with sustained fixation, or greater working memory demands required for fixed relative to free viewing. To investigate one possible mechanism for this reduction, we had participants perform a working memory task-an auditory n-back task-during free and fixed viewing, as well as a repetitive finger tapping condition, included to isolate the effects of motor interference independent of the oculomotor system. As expected, finger tapping significantly interfered with n-back performance relative to free viewing, as indexed by a decrease in accuracy and increase in response times. By contrast, there was no evidence that fixed viewing interfered with n-back performance relative to free viewing. Our findings failed to support a hypothesis of increased working memory load during fixation. They are consistent with the notion that fixation disrupts long-term memory performance through interference with visual processes.

Eye-Tracking Study to Enhance Usability of Molecular Diagnostics Reports in Cancer Precision Medicine

PURPOSE

We conducted usability studies on commercially available molecular diagnostic (MDX) test reports to identify strengths and weaknesses in content and form that drive clinical decision making. Given routine genomic testing in cancer medicine, oncologists must interpret MDX reports as well as evidence concerning clinical utility of biomarkers accurately for treatment or trial selection. This work aims to evaluate effectiveness of MDX reports in facilitating cancer treatment planning.

METHODS

Fourteen clinicians at an academic tertiary care medical facility, with a wide range of experience in oncology and in the use of molecular testing, participated in this study. Three commercially available, widely used, Clinical Laboratory Improvement Amendments (CLIA)-certified, College of American Pathologists (CAP)-accredited test reports (labeled Laboratories A, B, and C) were used. Eye tracking, surveys, and think-aloud protocols were used to collect usability data for these MDX reports focusing on ease of comprehension and actionability.

RESULTS

Clinicians found two primary areas in molecular diagnostic reports most useful for patient care: therapy options with benefit or lack of benefit to patients, including enrolling clinical trials; and pathogenic tumor molecular anomalies detected. Therapeutic implications and therapy classes such as US Food and Drug Administration-approved off-label, on-label, clinical trials were critical for decision making. However, all reports had usability and comprehension issues in these areas and could be improved.

CONCLUSION

Focused usability studies can help drive our understanding of the clinical workflow for use of molecular diagnostic tests in cancer care. This in turn can have major effects on quality of care, outcomes, costs, and patient satisfaction. This study demonstrates the use of specific usability techniques (eye tracking and think-aloud protocols) to help clinical laboratories improve MDX report design in a precision oncology treatment setting.

Investigation of eye movement pattern parameters of individuals with different fluid intelligence

Eye movement studies are subject of interest in human cognition. Cortical activity and cognitive load impress eye movement influentially. Here, we investigated whether fluid intelligence (FI) has any effect on eye movement pattern in a comparative visual search (CVS) task. FI of individuals was measured using the Cattell test, and participants were divided into three groups: low FI, middle FI, and high FI. Eye movements of individuals were then recorded during the CVS task. Eye movement patterns were extracted and compared statistically among the three groups. Our experiment demonstrated that eye movement patterns were significantly different among the three groups. Pearson correlation coefficients between FI and eye movement parameters were also calculated to assess which of the eye movement parameters were most affected by FI. Our findings illustrate that saccade peak velocity had the greatest positive correlation with FI score and the ratio of total fixation duration to total saccade duration had the greatest negative correlation with FI. Next, we extracted 24 features from eye movement patterns and designed: (1) a classifier to categorize individuals and (2) a regression analysis to predict the FI score of individuals. In the best case examined, the classifier categorized subjects with 68.3% accuracy, and the regression predicted FI of individuals with a 0.54 correlation between observed FI and predicted FI. In our investigation, the results have emphasized that imposed loads on low FI individuals is greater than that of high FI individuals in the cognitive load tasks.

Target probability modulates fixation-related potentials in visual search

This study investigated the influence of target probability on the neural response to target detection in free viewing visual search. Participants were asked to indicate the number of targets (one or two) among distractors in a visual search task while EEG and eye movements were co-registered. Target probability was manipulated by varying the set size of the displays between 10, 22, and 30 items. Fixation-related potentials time-locked to first target fixations revealed a pronounced P300 at the centro-parietal cortex with larger amplitudes for set sizes 22 and 30 than for set size 10. With increasing set size, more distractor fixations preceded the detection of the target, resulting in a decreased target probability and, consequently, a larger P300. For distractors, no increase of P300 amplitude with set size was observed. The findings suggest that set size specifically affects target but not distractor processing in overt serial visual search.

Eye movement characteristics reflected fatigue development in both young and elderly individuals

Fatigue can develop during prolonged computer work, particularly in elderly individuals. This study investigated eye movement characteristics in relation to fatigue development. Twenty young and 18 elderly healthy adults were recruited to perform a prolonged functional computer task while their eye movements were recorded. The task lasted 40 minutes involving 240 cycles divided into 12 segments. Each cycle consisted of a sequence involving memorization of a pattern, a washout period, and replication of the pattern using a computer mouse. The participants rated their perceived fatigue after each segment. The mean values of blink duration (BD) and frequency (BF), saccade duration (SCD) and peak velocity (SPV), pupil dilation range (PDR), and fixation duration (FD) along with the task performance based on clicking speed and accuracy, were computed for each task segment. An increased subjective evaluation of fatigue suggested the development of fatigue. BD, BF, and PDR increased whereas SPV and SCD decreased over time in the young and elderly groups. Longer FD, shorter SCD, and lower task performance were observed in the elderly compared with the young group. The present findings provide a viable approach to develop a computational model based on oculometrics to track fatigue development during computer work.

Less imageable words lead to more looks to blank locations during memory retrieval

People revisit spatial locations of visually encoded information when they are asked to retrieve that information, even when the visual image is no longer present. Such “looking at nothing” during retrieval is likely modulated by memory load (i.e., mental effort to maintain and reconstruct information) and the strength of mental representations. We investigated whether words that are more difficult to remember also lead to more looks to relevant, blank locations. Participants were presented four nouns on a two by two grid. A number of lexico-semantic variables were controlled to form high-difficulty and low-difficulty noun sets. Results reveal more frequent looks to blank locations during retrieval of high-difficulty nouns compared to low-difficulty ones. Mixed-effects modelling demonstrates that imagery-related semantic factors (imageability and concreteness) predict looking at nothing during retrieval. Results provide the first direct evidence that looking at nothing is modulated by word difficulty and in particular, word imageability. Overall, the research provides substantial support to the integrated memory account for linguistic stimuli and looking at nothing as a form of mental imagery.

Task modulates functional connectivity networks in free viewing behavior

In free visual exploration, eye-movement is immediately followed by dynamic reconfiguration of brain functional connectivity. We studied the task-dependency of this process in a combined visual search-change detection experiment. Participants viewed two (nearly) same displays in succession. First time they had to find and remember multiple targets among distractors, so the ongoing task involved memory encoding. Second time they had to determine if a target had changed in orientation, so the ongoing task involved memory retrieval. From multichannel EEG recorded during 200 ms intervals time-locked to fixation onsets, we estimated the functional connectivity using a weighted phase lag index at the frequencies of theta, alpha, and beta bands, and derived global and local measures of the functional connectivity graphs. We found differences between both memory task conditions for several network measures, such as mean path length, radius, diameter, closeness and eccentricity, mainly in the alpha band. Both the local and the global measures indicated that encoding involved a more segregated mode of operation than retrieval. These differences arose immediately after fixation onset and persisted for the entire duration of the lambda complex, an evoked potential commonly associated with early visual perception. We concluded that encoding and retrieval differentially shape network configurations involved in early visual perception, affecting the way the visual input is processed at each fixation. These findings demonstrate that task requirements dynamically control the functional connectivity networks involved in early visual perception.