Worth a glance: using eye movements to investigate the cognitive neuroscience of memory

A nonparametric method for detecting fixations and saccades using cluster analysis: removing the need for arbitrary thresholds

BACKGROUND

Eye tracking is an important component of many human and non-human primate behavioral experiments. As behavioral paradigms have become more complex, including unconstrained viewing of natural images, eye movements measured in these paradigms have become more variable and complex as well. Accordingly, the common practice of using acceleration, dispersion, or velocity thresholds to segment viewing behavior into periods of fixations and saccades may be insufficient.

NEW METHOD

Here we propose a novel algorithm, called Cluster Fix, which uses k-means cluster analysis to take advantage of the qualitative differences between fixations and saccades. The algorithm finds natural divisions in 4 state space parameters-distance, velocity, acceleration, and angular velocity-to separate scan paths into periods of fixations and saccades. The number and size of clusters adjusts to the variability of individual scan paths.

RESULTS

Cluster Fix can detect small saccades that were often indistinguishable from noisy fixations. Local analysis of fixations helped determine the transition times between fixations and saccades.

COMPARISON WITH EXISTING METHODS

Because Cluster Fix detects natural divisions in the data, predefined thresholds are not needed.

CONCLUSIONS

A major advantage of Cluster Fix is the ability to precisely identify the beginning and end of saccades, which is essential for studying neural activity that is modulated by or time-locked to saccades. Our data suggest that Cluster Fix is more sensitive than threshold-based algorithms but comes at the cost of an increase in computational time.

In your eyes only: deficits in executive functioning after frontal TMS reflect in eye movements

This study investigated the roles of the right and left dorsolateral prefrontal (rDLPFC, lDLPFC) and the medial frontal cortex (MFC) in executive functioning using a theta burst transcranial magnetic stimulation (TMS) approach. Healthy subjects solved two visual search tasks: a number search task with low cognitive demands, and a number and letter search task with high cognitive demands. To observe how subjects solved the tasks, we assessed their behavior with and without TMS using eye movements when subjects were confronted with specific executive demands. To observe executive functions, we were particularly interested in TMS-induced changes in visual exploration strategies found to be associated with good or bad performance in a control condition without TMS stimulation. TMS left processing time unchanged in both tasks. Inhibition of the rDLPFC resulted in a decrease in anticipatory fixations in the number search task, i.e., a decrease in a good strategy in this low demand task. This was paired with a decrease in stimulus fixations. Together, these results point to a role of the rDLPFC in planning and response selection. Inhibition of the lDLPFC and the MFC resulted in an increase in anticipatory fixations in the number and letter search task, i.e., an increase in the application of a good strategy in this task. We interpret these results as a compensatory strategy to account for TMS-induced deficits in attentional switching when faced with high switching demands. After inhibition of the lDLPFC, an increase in regressive fixations was found in the number and letter search task. In the context of high working memory demands, this strategy appears to support TMS-induced working memory deficits. Combining an experimental TMS approach with the recording of eye movements proved sensitive to discrete decrements of executive functions and allows pinpointing the functional organization of the frontal lobes.

Eye movement analysis and cognitive processing: detecting indicators of conversion to Alzheimer's disease

A great amount of research has been developed around the early cognitive impairments that best predict the onset of Alzheimer's disease (AD). Given that mild cognitive impairment (MCI) is no longer considered to be an intermediate state between normal aging and AD, new paths have been traced to acquire further knowledge about this condition and its subtypes, and to determine which of them have a higher risk of conversion to AD. It is now known that other deficits besides episodic and semantic memory impairments may be present in the early stages of AD, such as visuospatial and executive function deficits. Furthermore, recent investigations have proven that the hippocampus and the medial temporal lobe structures are not only involved in memory functioning, but also in visual processes. These early changes in memory, visual, and executive processes may also be detected with the study of eye movement patterns in pathological conditions like MCI and AD. In the present review, we attempt to explore the existing literature concerning these patterns of oculomotor changes and how these changes are related to the early signs of AD. In particular, we argue that deficits in visual short-term memory, specifically in iconic memory, attention processes, and inhibitory control, may be found through the analysis of eye movement patterns, and we discuss how they might help to predict the progression from MCI to AD. We add that the study of eye movement patterns in these conditions, in combination with neuroimaging techniques and appropriate neuropsychological tasks based on rigorous concepts derived from cognitive psychology, may highlight the early presence of cognitive impairments in the course of the disease.

From perceptual to language-mediated categorization

From at least two months onwards, infants can form perceptual categories. During the first year of life, object knowledge develops from the ability to represent individual object features to representing correlations between attributes and to integrate information from different sources. At the end of the first year, these representations are shaped by labels, opening the way to conceptual knowledge. Here, we review the development of object knowledge and object categorization over the first year of life. We then present an artificial neural network model that models the transition from early perceptual categorization to categories mediated by labels. The model informs a current debate on the role of labels in object categorization by suggesting that although labels do not act as object features they nevertheless affect perceived similarity of perceptually distinct objects sharing the same label. The model presents the first step of an integrated account from early perceptual categorization to language-based concept learning.

It does not look odd to me: perceptual impairments and eye movements in amnesic patients with medial temporal lobe damage

Studies of people with memory impairments have shown that a specific set of brain structures in the medial temporal lobe (MTL) is vital for memory function. However, whether these structures have a role outside of memory remains contentious. Recent studies of amnesic patients with damage to two structures within the MTL, the hippocampus and the perirhinal cortex, indicated that these patients also performed poorly on perceptual tasks. More specifically, they performed worse than controls when discriminating between objects, faces and scenes with overlapping features. In order to investigate whether these perceptual deficits are reflected in their viewing strategies, we tested a group of amnesic patients with MTL damage that included the hippocampus and perirhinal cortex on a series of oddity discrimination tasks in which they had to select an odd item from a visual array. Participants' eye movements were monitored throughout the experiment. Results revealed that patients were impaired on tasks that required them to discriminate between items that shared many features, and tasks that required processing items from different viewpoints. An analysis of their eye movements revealed that they exhibited a similar viewing pattern as controls: they fixated more on the target item on trials answered correctly, but not on trials answered incorrectly. In addition, their impaired performance was not explained by an abnormal viewing-strategy that assessed their use of working memory. These results suggest that the perceptual deficits in the MTL patients are not a consequence of abnormal viewing patterns of the objects and scenes, but instead, could involve an inability to bind information gathered from several fixations into a cohesive percept. These data also support the view that MTL structures are important not only for long-term memory, but are also involved in perceptual tasks.

Shared Visual Attention Reduces Hindsight Bias

Hindsight bias is the tendency to retrospectively think of outcomes as being more foreseeable than they actually were. It is a robust judgment bias and is difficult to correct (or “debias”). In the experiments reported here, we used a visual paradigm in which performers decided whether blurred photos contained humans. Evaluators, who saw the photos unblurred and thus knew whether a human was present, estimated the proportion of participants who guessed whether a human was present. The evaluators exhibited visual hindsight bias in a way that matched earlier data from judgments of historical events surprisingly closely. Using eye tracking, we showed that a higher correlation between the gaze patterns of performers and evaluators (shared attention) is associated with lower hindsight bias. This association was validated by a causal method for debiasing: Showing the gaze patterns of the performers to the evaluators as they viewed the stimuli reduced the extent of hindsight bias.

Procedural learning and associative memory mechanisms contribute to contextual cueing: Evidence from fMRI and eye-tracking

Using a combination of eye tracking and fMRI in a contextual cueing task, we explored the mechanisms underlying the facilitation of visual search for repeated spatial configurations. When configurations of distractors were repeated, greater activation in the right hippocampus corresponded to greater reductions in the number of saccades to locate the target. A psychophysiological interactions analysis for repeated configurations revealed that a strong functional connectivity between this area in the right hippocampus and the left superior parietal lobule early in learning was significantly reduced toward the end of the task. Practice related changes (which we call "procedural learning") in activation in temporo-occipital and parietal brain regions depended on whether or not spatial context was repeated. We conclude that context repetition facilitates visual search through chunk formation that reduces the number of effective distractors that have to be processed during the search. Context repetition influences procedural learning in a way that allows for continuous and effective chunk updating.

Remembering and Voting: Theory and Evidence from Amnesic Patients

One of the most prominent claims to emerge from the field of public opinion is that citizens can vote for candidates whose issue positions best reflect their own beliefs even when they cannot remember previously learned stances associated with the candidates. The current experiment provides a unique and powerful examination of this claim by determining whether individuals with profound amnesia, whose severe memory impairments prevent them from remembering specific issue information associated with any particular candidate, can vote for candidates whose issue positions come closest to their own political views. We report here that amnesic patients, despite not being able to remember any issue information, consistently voted for candidates with favored political positions. Thus, sound voting decisions do not require recall or recognition of previously learned associations between candidates and their issue positions. This result supports a multiple memory systems model of political decision making.

The revealing glance: eye gaze behavior to concealed information

We explored the usefulness of eye fixation durations as a dependent measure in a concealed knowledge test, drawing on Ryan, Hannula, and Cohen (2007), who found eye fixations on a familiar face to be longer than fixations on an unknown face. However, in their study, participants always had to select the known face out of three faces; thus, recognition and response intention could not be differentiated. In the experimental phase of our experiment, participants saw six faces per trial and had to select one of them. We had three conditions: In the first, one of the six faces was a known face, and the participants had to conceal that knowledge and select another face (concealed display); in another, one of the six faces was a known face, and the participants had to select that face (revealed display); or finally, all six faces were unknown, and participants had to select any of the six faces (neutral display). Using fixation durations as the dependent measure, we found a pure and early recognition effect; that is, fixations on the concealed faces (known but not selected) were longer than fixations on the nonselected unknown target faces in the neutral display. In addition, we found a response intention effect; that is, fixation durations on the selected known faces were longer than those on concealed faces (known but not selected).

Relational memory in psychotic bipolar disorder

OBJECTIVES

Recent research has highlighted the phenotypic and genetic overlap of bipolar disorder and schizophrenia. Cognitive deficits in bipolar disorder parallel those seen in schizophrenia, particularly for bipolar disorder patients with a history of psychotic features. Here we explored whether relational memory deficits, which are prominent in schizophrenia, are also present in patients with psychotic bipolar disorder.

METHODS

We tested 25 patients with psychotic bipolar disorder on a relational memory paradigm previously employed to quantify deficits in schizophrenia. During the training, participants learned to associate a set of faces and background scenes. During the testing, participants viewed a single background overlaid by three trained faces and were asked to recall the matching face, which was either present (Match trials) or absent (Non-Match trials). Explicit recognition and eye-movement data were collected and compared to those for 28 schizophrenia patients and 27 healthy subjects from a previously published dataset.

RESULTS

Contrary to our prediction, we found psychotic bipolar disorder patients were less impaired in relational memory than schizophrenia subjects. Bipolar disorder subjects showed eye-movement behavior similar to healthy controls, whereas schizophrenia subjects were impaired relative to both groups. However, bipolar disorder patients with current delusions and/or hallucinations were more impaired than bipolar disorder patients not currently experiencing these symptoms.

CONCLUSIONS

We found that patients with psychotic bipolar disorder had better relational memory performance than schizophrenia patients, indicating that a history of psychotic symptoms does not lead to a significant relational memory deficit.

Adults scan own- and other-race faces differently

It is well established that individuals show an other-race effect (ORE) in face recognition: they recognize own-race faces better than other-race faces. The present study tested the hypothesis that individuals would also scan own- and other-race faces differently. We asked Chinese participants to remember Chinese and Caucasian faces and we tested their memory of the faces over five testing blocks. The participants' eye movements were recorded with the use of an eye tracker. The data were analyzed with an Area of Interest approach using the key AOIs of a face (eyes, nose, and mouth). Also, we used the iMap toolbox to analyze the raw data of participants' fixation on each pixel of the entire face. Results from both types of analyses strongly supported the hypothesis. When viewing target Chinese or Caucasian faces, Chinese participants spent a significantly greater proportion of fixation time on the eyes of other-race Caucasian faces than the eyes of own-race Chinese faces. In contrast, they spent a significantly greater proportion of fixation time on the nose and mouth of Chinese faces than the nose and mouth of Caucasian faces. This pattern of differential fixation, for own- and other-race eyes and nose in particular, was consistent even as participants became increasingly familiar with the target faces of both races. The results could not be explained by the perceptual salience of the Chinese nose or Caucasian eyes because these features were not differentially salient across the races. Our results are discussed in terms of the facial morphological differences between Chinese and Caucasian faces and the enculturation of mutual gaze norms in East Asian cultures.

The hippocampus supports multiple cognitive processes through relational binding and comparison

It has been well established that the hippocampus plays a pivotal role in explicit long-term recognition memory. However, findings from amnesia, lesion and recording studies with non-human animals, eye-movement recording studies, and functional neuroimaging have recently converged upon a similar message: the functional reach of the hippocampus extends far beyond explicit recognition memory. Damage to the hippocampus affects performance on a number of cognitive tasks including recognition memory after short and long delays and visual discrimination. Additionally, with the advent of neuroimaging techniques that have fine spatial and temporal resolution, findings have emerged that show the elicitation of hippocampal responses within the first few 100 ms of stimulus/task onset. These responses occur for novel and previously viewed information during a time when perceptual processing is traditionally thought to occur, and long before overt recognition responses are made. We propose that the hippocampus is obligatorily involved in the binding of disparate elements across both space and time, and in the comparison of such relational memory representations. Furthermore, the hippocampus supports relational binding and comparison with or without conscious awareness for the relational representations that are formed, retrieved and/or compared. It is by virtue of these basic binding and comparison functions that the reach of the hippocampus extends beyond long-term recognition memory and underlies task performance in multiple cognitive domains.

Distinguishing highly confident accurate and inaccurate memory: insights about relevant and irrelevant influences on memory confidence

It is generally believed that accuracy and confidence in one's memory are related, but there are many instances when they diverge. Accordingly it is important to disentangle the factors that contribute to memory accuracy and confidence, especially those factors that contribute to confidence, but not accuracy. We used eye movements to separately measure fluent cue processing, the target recognition experience, and relative evidence assessment on recognition confidence and accuracy. Eye movements were monitored during a face-scene associative recognition task, in which participants first saw a scene cue, followed by a forced-choice recognition test for the associated face, with confidence ratings. Eye movement indices of the target recognition experience were largely indicative of accuracy, and showed a relationship to confidence for accurate decisions. In contrast, eye movements during the scene cue raised the possibility that more fluent cue processing was related to higher confidence for both accurate and inaccurate recognition decisions. In a second experiment we manipulated cue familiarity, and therefore cue fluency. Participants showed higher confidence for cue-target associations for when the cue was more familiar, especially for incorrect responses. These results suggest that over-reliance on cue familiarity and under-reliance on the target recognition experience may lead to erroneous confidence.

Aerobic fitness enhances relational memory in preadolescent children: the FITKids randomized control trial

It is widely accepted that aerobic exercise enhances hippocampal plasticity. Often, this plasticity co-occurs with gains in hippocampal-dependent memory. Cross-sectional work investigating this relationship in preadolescent children has found behavioral differences in higher versus lower aerobically fit participants for tasks measuring relational memory, which is known to be critically tied to hippocampal structure and function. The present study tested whether similar differences would arise in a clinical intervention setting where a group of preadolescent children were randomly assigned to a 9-month after school aerobic exercise intervention versus a wait-list control group. Performance measures included eye-movements as a measure of memory, based on recent work linking eye-movement indices of relational memory to the hippocampus. Results indicated that only children in the intervention increased their aerobic fitness. Compared to the control group, those who entered the aerobic exercise program displayed eye-movement patterns indicative of superior memory for face-scene relations, with no differences observed in memory for individual faces. The results of this intervention study provide clear support for the proposed linkage among the hippocampus, relational memory, and aerobic fitness, as well as illustrating the sensitivity of eye-movement measures as a means of assessing memory.

The hippocampus reevaluated in unconscious learning and memory: at a tipping point?

Classic findings from the neuropsychological literature invariably indicated that performances on tests of memory that can be accomplished without conscious awareness were largely spared in amnesia, while those that required conscious retrieval (e.g., via recognition or recall) of information learned in the very same sessions was devastatingly impaired. Based on reports of such dissociations, it was proposed that one of the fundamental distinctions between memory systems is whether or not they support conscious access to remembered content. Only recently have we come to realize that the putative systemic division of labor between conscious and unconscious memory is not so clean. A primary goal of this review is to examine recent evidence that has been advanced against the view that the hippocampus is selectively critical for conscious memory. Along the way, consideration is given to criticisms that have been levied against these findings, potential explanations for differences in the reported results are proposed, and methodological pitfalls in investigations of unconscious memory are discussed. Ultimately, it is concluded that a tipping point has been reached, and that while conscious recollection depends critically on hippocampal integrity, the reach of the hippocampus extends to unconscious aspects of memory performance when relational memory processing and representation are required.

The eyes know: eye movements as a veridical index of memory

In two experiments, we examined whether observers' eye movements distinguish studied faces from highly similar novel faces. Participants' eye movements were monitored while they viewed three-face displays. Target-present displays contained a studied face and two morphed faces that were visually similar to it; target-absent displays contained three morphed faces that were visually similar to a studied, but not tested, face. On each trial in a test session, participants were instructed to choose the studied face if it was present or a random face if it was not and then to indicate whether the chosen face was studied. Whereas manipulating visual similarity in target-absent displays influenced the rate of false endorsements of nonstudied items as studied, eye movements proved impervious to this manipulation. Studied faces were viewed disproportionately from 1,000 to 2,000 ms after display onset and from 1,000 to 500 ms before explicit identification. Early viewing also distinguished studied faces from faces incorrectly endorsed as studied. Our findings show that eye movements provide a relatively pure index of past experience that is uninfluenced by explicit response strategies, and suggest that eye movement measures may be of practical use in applied settings.

CNTRICS imaging biomarkers final task selection: Long-term memory and reinforcement learning

Functional imaging paradigms hold great promise as biomarkers for schizophrenia research as they can detect altered neural activity associated with the cognitive and emotional processing deficits that are so disabling to this patient population. In an attempt to identify the most promising functional imaging biomarkers for research on long-term memory (LTM), the Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) initiative selected "item encoding and retrieval," "relational encoding and retrieval," and "reinforcement learning" as key LTM constructs to guide the nomination process. This manuscript reports on the outcome of the third CNTRICS biomarkers meeting in which nominated paradigms in each of these domains were discussed by a review panel to arrive at a consensus on which of the nominated paradigms could be recommended for immediate translational development. After briefly describing this decision process, information is presented from the nominating authors describing the 4 functional imaging paradigms that were selected for immediate development. In addition to describing the tasks, information is provided on cognitive and neural construct validity, sensitivity to behavioral or pharmacological manipulations, availability of animal models, psychometric characteristics, effects of schizophrenia, and avenues for future development.

A Flicker Change Detection Task Reveals Object-in-Scene Memory Across Species

Tests of recognition memory in macaques typically assay memory for objects or isolated images, over time spans of seconds to hours from stimulus presentation, and/or require extensive training. Here, we propose a new application of the flicker change detection task that could measure object-in-scene memory days after single-trial exposures. In three experiments, participants searched for a changing object – or “target” – embedded within a scene as their eye movements were tracked. For new targets-in-scenes, the change is difficult to detect and requires extensive search. Once the target is found, however, the change becomes obvious. We reasoned that the decreased times required to find a target in a repeated scene would indicate memory for the target. In humans, targets were found faster when the targets-and-scenes were explicitly remembered than when they were forgotten, or had never been seen before. This led to faster repeated-trial compared to novel-trial search times. Based solely on repeated-trial search times, we were able to select distributions comprised of predominantly remembered or predominantly forgotten trials. Macaques exhibited the same repetition effects as humans, suggesting that remembered trials could be dissociated from novel or forgotten trials using the same procedures we established in humans. Finally, an anterograde amnesic patient with damage that included the medial temporal lobe (MTL) showed no search time differences, suggesting that memory revealed through search times on this task requires MTL integrity. Together, these findings indicate that the time required to locate a changing object reveals object-in-scene memory over long retention intervals in humans and macaques.

Observing degradation of visual representations over short intervals when medial temporal lobe is damaged

Medial temporal lobe (MTL) contributions to the brief maintenance of visual representations were evaluated by studying a group of patients with MTL damage. Eye movements of patients and healthy comparison subjects were tracked while performing a visual search for a target among complex stimuli of varying similarity to that target. Despite the task having no imposed delays, patients were impaired behaviorally, and eye movement measures showed abnormally rapid degradation of target representations in the patients. Eye movement data showed a modulation of the duration of fixations as a function of the similarity of fixated array lures to the target, but the effect was attenuated in patients during long fixation paths away from the sample target. This effect manifested despite patients' shorter searches and more frequent fixations of the sample target. Novel techniques provided unique insight into visual representation without healthy MTL, which may support maintenance of information through hippocampal-dependent relational binding.

Practical aspects of running developmental studies in the MEG

Developmental neuroimaging studies offer a unique opportunity to gain insight into the underpinnings of various cognitive functions by examining age-related changes in brain structure and function. There is an increasing body of neuroimaging literature discussing issues related to testing children in developmental studies (Crone et al. Human Brain Mapping 31:835-837, 2010). These deal with fMRI developmental studies and discuss methods (Luna et al. Human Brain Mapp 31:863-871, 2010), data interpretation (Poldrack Human Brain Mapp 31:872-878, 2010), and theoretical approaches (Karmiloff-Smith Human Brain Mapp 31:934-941, 2010). There has not yet been an equivalent discussion for MEG developmental studies. This paper will address issues specific to data acquisition, analysis, and interpretation for MEG developmental studies.