The relationship between eye movements and subsequent recognition: Evidence from individual differences and amnesia

Distinct eye movement patterns to complex scenes in Alzheimer's disease and Lewy body disease

Background

Alzheimer's disease (AD) and Lewy body disease (LBD), the two most common causes of neurodegenerative dementia with similar clinical manifestations, both show impaired visual attention and altered eye movements. However, prior studies have used structured tasks or restricted stimuli, limiting the insights into how eye movements alter and differ between AD and LBD in daily life.

Objective

We aimed to comprehensively characterize eye movements of AD and LBD patients on naturalistic complex scenes with broad categories of objects, which would provide a context closer to real-world free viewing, and to identify disease-specific patterns of altered eye movements.

Methods

We collected spontaneous viewing behaviors to 200 naturalistic complex scenes from patients with AD or LBD at the prodromal or dementia stage, as well as matched control participants. We then investigated eye movement patterns using a computational visual attention model with high-level image features of object properties and semantic information.

Results

Compared with matched controls, we identified two disease-specific altered patterns of eye movements: diminished visual exploration, which differentially correlates with cognitive impairment in AD and with motor impairment in LBD; and reduced gaze allocation to objects, attributed to a weaker attention bias toward high-level image features in AD and attributed to a greater image-center bias in LBD.

Conclusion

Our findings may help differentiate AD and LBD patients and comprehend their real-world visual behaviors to mitigate the widespread impact of impaired visual attention on daily activities.

Are older adults susceptible to visual distraction when targets and distractors are spatially separated?

Older adults show preserved memory for previously distracting information due to reduced inhibitory control. In some previous studies, targets and distractors overlap both temporally and spatially. We investigated whether age differences in attentional orienting and disengagement affect recognition memory when targets and distractors are spatially separated at encoding. In Experiments 1 and 2, eye movements were recorded while participants completed an incidental encoding task under covert (i.e., restricted viewing) and overt (i.e., free-viewing) conditions, respectively. The encoding task consisted of pairs of target and distractor item-color stimuli presented in separate visual hemifields. Prior to stimulus onset, a central cue indicated the location of the upcoming target. Participants were subsequently tested on their recognition of the items, their location, and the associated color. In Experiment 3, targets were validly cued on 75% of the encoding trials; on invalid trials, participants had to disengage their attention from the distractor and reorient to the target. Associative memory for colors was reduced among older adults across all experiments, though their location memory was only reduced in Experiment 1. In Experiment 2, older and younger adults directed a similar proportion of fixations toward targets and distractors. Explicit recognition of distractors did not differ between age groups in any of the experiments. However, older adults were slower to correctly recognize distractors than false alarm to novel items in Experiment 2, suggesting some implicit memory for distraction. Together, these results demonstrate that older adults may only be vulnerable to encoding visual distraction when viewing behavior is unconstrained.

Chronic Kidney Disease Induces Cognitive Impairment in the Early Stage

OBJECTIVE

Previous research indicates a link between cognitive impairment and chronic kidney disease (CKD), but the underlying factors are not fully understood. This study aimed to investigate the progression of CKD-induced cognitive impairment and the involvement of cognition-related proteins by developing early- and late-stage CKD models in Sprague-Dawley rats.

METHODS

The Morris water maze test and the step-down passive avoidance task were performed to evaluate the cognitive abilities of the rats at 24 weeks after surgery. Histopathologic examinations were conducted to examine renal and hippocampal damage. Real-time PCR, Western blotting analysis, and immunohistochemical staining were carried out to determine the hippocampal expression of brain-derived neurotrophic factor (BDNF), choline acetyltransferase (ChAT), and synaptophysin (SYP).

RESULTS

Compared with the control rats, the rats with early-stage CKD exhibited mild renal damage, while those with late-stage CKD showed significantly increased serum creatinine levels as well as apparent renal and brain damage. The rats with early-stage CKD also demonstrated significantly impaired learning abilities and memory compared with the control rats, with further deterioration observed in the rats with late-stage CKD. Additionally, we observed a significant downregulation of cognition-related proteins in the hippocampus of rats with early-stage CKD, which was further exacerbated with declining renal function as well as worsening brain and renal damage in rats with late-stage CKD.

CONCLUSION

These results suggest the importance of early screening to identify CKD-induced cognitive dysfunction promptly. In addition, the downregulation of cognition-related proteins may play a role in the progression of cognitive dysfunction.

Sex and menstrual cycle influence human spatial navigation strategies and performance

Which facets of human spatial navigation do sex and menstrual cycle influence? To answer this question, a cross-sectional online study of reproductive age women and men was conducted in which participants were asked to demonstrate and self-report their spatial navigation skills and strategies. Participants self-reported their sex and current menstrual phase [early follicular (EF), late follicular/periovulatory (PO), and mid/late luteal (ML)], and completed a series of questionnaires and tasks measuring self-reported navigation strategy use, topographical memory, cognitive map formation, face recognition, and path integration. We found that sex influenced self-reported use of cognitive map- and scene-based strategies, face recognition, and path integration. Menstrual phase moderated the influence of sex: compared to men, women had better face recognition and worse path integration, but only during the PO phase; PO women were also better at path integration in the presence of a landmark compared to EF + ML women and men. These findings provide evidence that human spatial navigation varies with the menstrual cycle and suggest that sensitivity of the entorhinal cortex and longitudinal axis of the hippocampus to differential hormonal effects may account for this variation.

Cortico-ocular Coupling in the Service of Episodic Memory Formation

Encoding of visual information is a necessary requirement for most types of episodic memories. In search for a neural signature of memory formation, amplitude modulation of neural activity has been repeatedly shown to correlate with and suggested to be functionally involved in successful memory encoding. We here report a complementary view on why and how brain activity relates to memory, indicating a functional role of cortico-ocular interactions for episodic memory formation. Recording simultaneous magnetoencephalography and eye tracking in 35 human participants, we demonstrate that gaze variability and amplitude modulations of alpha/beta oscillations (10-20Hz) in visual cortex covary and predict subsequent memory performance between and within participants. Amplitude variation during pre-stimulus baseline was associated with gaze direction variability, echoing the co-variation observed during scene encoding. We conclude that encoding of visual information engages unison coupling between oculomotor and visual areas in the service of memory formation.

"Looking at nothing": An implicit ocular motor index of face recognition in developmental prosopagnosia

Subjects often look towards to previous location of a stimulus related to a task even when that stimulus is no longer visible. In this study we asked whether this effect would be preserved or reduced in subjects with developmental prosopagnosia. Participants learned faces presented in video-clips and then saw a brief montage of four faces, which was replaced by a screen with empty boxes, at which time they indicated whether the learned face had been present in the montage. Control subjects were more likely to look at the blank location where the learned face had appeared, on both hit and miss trials, though the effect was larger on hit trials. Prosopagnosic subjects showed a reduced effect, though still better on hit than on miss trials. We conclude that explicit accuracy and our implicit looking at nothing effect are parallel effects reflecting the strength of the neural activity underlying face recognition.

Episodic memory formation in unrestricted viewing

The brain systems of episodic memory and oculomotor control are tightly linked, suggesting a crucial role of eye movements in memory. But little is known about the neural mechanisms of memory formation across eye movements in unrestricted viewing behavior. Here, we leverage simultaneous eye tracking and EEG recording to examine episodic memory formation in free viewing. Participants memorized multi-element events while their EEG and eye movements were concurrently recorded. Each event comprised elements from three categories (face, object, place), with two exemplars from each category, in different locations on the screen. A subsequent associative memory test assessed participants' memory for the between-category associations that specified each event. We used a deconvolution approach to overcome the problem of overlapping EEG responses to sequential saccades in free viewing. Brain activity was time-locked to the fixation onsets, and we examined EEG power in the theta and alpha frequency bands, the putative oscillatory correlates of episodic encoding mechanisms. Three modulations of fixation-related EEG predicted high subsequent memory performance: (1) theta increase at fixations after between-category gaze transitions, (2) theta and alpha increase at fixations after within-element gaze transitions, (3) alpha decrease at fixations after between-exemplar gaze transitions. Thus, event encoding with unrestricted viewing behavior was characterized by three neural mechanisms, manifested in fixation-locked theta and alpha EEG activity that rapidly turned on and off during the unfolding eye movement sequences. These three distinct neural mechanisms may be the essential building blocks that subserve the buildup of coherent episodic memories during unrestricted viewing behavior.

Eye-movement replay supports episodic remembering

When we bring to mind something we have seen before, our eyes spontaneously unfold in a sequential pattern strikingly similar to that made during the original encounter, even in the absence of supporting visual input. Oculomotor movements of the eye may then serve the opposite purpose of acquiring new visual information; they may serve as self-generated cues, pointing to stored memories. Over 50 years ago Donald Hebb, the forefather of cognitive neuroscience, posited that such a sequential replay of eye movements supports our ability to mentally recreate visuospatial relations during episodic remembering. However, direct evidence for this influential claim is lacking. Here we isolate the sequential properties of spontaneous eye movements during encoding and retrieval in a pure recall memory task and capture their encoding-retrieval overlap. Critically, we show that the fidelity with which a series of consecutive eye movements from initial encoding is sequentially retained during subsequent retrieval predicts the quality of the recalled memory. Our findings provide direct evidence that such scanpaths are replayed to assemble and reconstruct spatio-temporal relations as we remember and further suggest that distinct scanpath properties differentially contribute depending on the nature of the goal-relevant memory.

Damage to the human dentate gyrus impairs the perceptual discrimination of complex, novel objects

The hippocampus (HPC), and the dentate gyrus (DG) subregion in particular, is purported to be a pattern separator, orthogonally representing similar information so that distinct memories may be formed. The HPC may also be involved in complex perceptual discrimination. It is unclear if this role is limited to spatial/scene stimuli or extends to the discrimination of objects. Also unclear is whether the DG itself contributes to pattern separation beyond memory. BL, an individual with bilateral DG lesions, was previously shown to have poor discrimination of similar, everyday objects in memory. Here, we demonstrate that BL's deficit extends to complex perceptual discrimination of novel objects. Specifically, BL was presented with closely matched possible and impossible objects, which give rise to fundamentally different 3D perceptual representations despite being visually similar. BL performed significantly worse than controls when asked to select an odd object (e.g., impossible) amongst three identical counterpart objects (e.g., possible) presented at different rotations. His deficit was also evident in an atypical eye fixation pattern during this task. In contrast, BL's performance was indistinguishable from that of controls on other tasks involving the same objects, indicating that he could visually differentiate the object pairs, that he perceived the objects holistically in 3D, and that he has only a mild weakness in categorizing object possibility. Furthermore, his performance on standardized neuropsychological measures indicated intact mental rotation, visual-spatial attention, and working memory (visual and auditory). Collectively, these results provide evidence that the DG is necessary for complex perceptual discrimination of novel objects, indicating that the DG might function as a generic pattern separator of a wide range of stimuli within high-level perception, and that its role is not limited to memory.

The impact of aging and repetition on eye movements and recognition memory

The modulation of gaze fixations on neural activity in the hippocampus, a region critical for memory, has been shown to be weaker in older adults compared to younger adults. However, as such research has relied on indirect measures of memory, it remains unclear whether the relationship between visual exploration and direct measures of memory is similarly disrupted in aging. The current study tested older and younger adults on a face memory eye-tracking task previously used by our group that showed that recognition memory for faces presented across variable, but not fixed, viewpoints relies on a hippocampal-dependent binding function. Here, we examined how aging influences eye movement measures that reveal the amount (cumulative sampling) and extent (distribution of gaze fixations) of visual exploration. We also examined how aging influences direct (subsequent conscious recognition) and indirect (eye movement repetition effect) expressions of memory. No age differences were found in direct recognition regardless of facial viewpoint. However, the eye movement measures revealed key group differences. Compared to younger adults, older adults exhibited more cumulative sampling, a different distribution of fixations, and a larger repetition effect. Moreover, there was a positive relationship between cumulative sampling and direct recognition in younger adults, but not older adults. Neither age group showed a relationship between the repetition effect and direct recognition. Thus, despite similar direct recognition, age-related differences were observed in visual exploration and in an indirect eye-movement memory measure, suggesting that the two groups may acquire, retain, and use different facial information to guide recognition.

Working memory and active sampling of the environment: Medial temporal contributions

Working memory (WM) refers to the ability to maintain and actively process information-either derived from perception or long-term memory (LTM)-for intelligent thought and action. This chapter focuses on the contributions of the temporal lobe, particularly medial temporal lobe (MTL) to WM. First, neuropsychological evidence for the involvement of MTL in WM maintenance is reviewed, arguing for a crucial role in the case of retaining complex relational bindings between memorized features. Next, MTL contributions at the level of neural mechanisms are covered-with a focus on WM encoding and maintenance, including interactions with ventral temporal cortex. Among WM use processes, we focus on active sampling of environmental information, a key input source to capacity-limited WM. MTL contributions to the bidirectional relationship between active sampling and memory are highlighted-WM control of active sampling and sampling as a way of selecting input to WM. Memory-based sampling studies relying on scene and object inspection, visual-based exploration behavior (e.g., vicarious behavior), and memory-guided visual search are reviewed. The conclusion is that MTL serves an important function in the selection of information from perception and transfer from LTM to capacity-limited WM.

Aging, pattern separation, and categorical perception of faces

Categorical perception (CP) is the phenomenon by which observers view linear changes that occur across a continuum as distinct categories. Although categorical perception is a perceptual phenomenon, it may be subserved by mnemonic processes such as pattern separation. To examine this hypothesis, following standard CP tasks, we assessed younger and older participants' abilities to identify and discriminate between members of pairs of famous or non-famous faces. We hypothesized that if CP is dependent upon neural pattern separation, which declines with aging, discrimination ability as indexed by CP would be compromised in older adults, as was found in our study. Since familiarity promotes pattern separation, CP should be enhanced for famous, as compared to non-famous faces, even in older adults. We found that all participants benefited from familiarity, but younger adults outperformed older adults overall. We next examined the effects of face inversion on CP for both famous and non-famous faces. If pattern separation, and CP, is determined solely by the similarity across physical features, then CP should be similar for upright and inverted faces since these features are perceptually invariant across orientation. If, however, pattern separation, and CP, depends on how stimuli are represented, then orientation may matter as upright and inverted faces are represented holistically or part-based, respectively. We found that inversion disrupted CP in younger adults whereas older adults performed similarly across both conditions, suggesting that face-representation is more part-based in older adults.

Neural Correlates of Subsequent Memory-Related Gaze Reinstatement

Mounting evidence linking gaze reinstatement-the recapitulation of encoding-related gaze patterns during retrieval-to behavioral measures of memory suggests that eye movements play an important role in mnemonic processing. Yet, the nature of the gaze scanpath, including its informational content and neural correlates, has remained in question. In this study, we examined eye movement and neural data from a recognition memory task to further elucidate the behavioral and neural bases of functional gaze reinstatement. Consistent with previous work, gaze reinstatement during retrieval of freely viewed scene images was greater than chance and predictive of recognition memory performance. Gaze reinstatement was also associated with viewing of informationally salient image regions at encoding, suggesting that scanpaths may encode and contain high-level scene content. At the brain level, gaze reinstatement was predicted by encoding-related activity in the occipital pole and BG, neural regions associated with visual processing and oculomotor control. Finally, cross-voxel brain pattern similarity analysis revealed overlapping subsequent memory and subsequent gaze reinstatement modulation effects in the parahippocampal place area and hippocampus, in addition to the occipital pole and BG. Together, these findings suggest that encoding-related activity in brain regions associated with scene processing, oculomotor control, and memory supports the formation, and subsequent recapitulation, of functional scanpaths. More broadly, these findings lend support to scanpath theory's assertion that eye movements both encode, and are themselves embedded in, mnemonic representations.

Encoding and retrieval eye movements mediate age differences in pattern completion

Older adults often mistake new information as 'old', yet the mechanisms underlying this response bias remain unclear. Typically, false alarms by older adults are thought to reflect pattern completion - the retrieval of a previously encoded stimulus in response to partial input. However, other work suggests that age-related retrieval errors can be accounted for by deficient encoding processes. In the present study, we used eye movement monitoring to quantify age-related changes in behavioral pattern completion as a function of eye movements during both encoding and partially cued retrieval. Consistent with an age-related encoding deficit, older adults executed more gaze fixations and more similar eye movements across repeated image presentations than younger adults, and such effects were predictive of subsequent recognition memory. Analysis of eye movements at retrieval further indicated that in response to partial lure cues, older adults reactivated the similar studied image, indexed by the similarity between encoding and retrieval gaze patterns, and did so more than younger adults. Critically, reactivation of encoded image content via eye movements was associated with lure false alarms in older adults, providing direct evidence for a pattern completion bias. Together, these findings suggest that age-related changes in both encoding and retrieval processes, indexed by eye movements, underlie older adults' increased vulnerability to memory errors.

Restricting Visual Exploration Directly Impedes Neural Activity, Functional Connectivity, and Memory

We move our eyes to explore the visual world, extract information, and create memories. The number of gaze fixations-the stops that the eyes make-has been shown to correlate with activity in the hippocampus, a region critical for memory, and with later recognition memory. Here, we combined eyetracking with fMRI to provide direct evidence for the relationships between gaze fixations, neural activity, and memory during scene viewing. Compared to free viewing, fixating a single location reduced: 1) subsequent memory, 2) neural activity along the ventral visual stream into the hippocampus, 3) neural similarity between effects of subsequent memory and visual exploration, and 4) functional connectivity among the hippocampus, parahippocampal place area, and other cortical regions. Gaze fixations were uniquely related to hippocampal activity, even after controlling for neural effects due to subsequent memory. Therefore, this study provides key causal evidence supporting the notion that the oculomotor and memory systems are intrinsically related at both the behavioral and neural level. Individual gaze fixations may provide the basic unit of information on which memory binding processes operate.

Temporal context guides visual exploration during scene recognition

Memories for episodes are temporally structured. Cognitive models derived from list-learning experiments attribute this structure to the retrieval of temporal context information that indicates when a memory occurred. These models predict key features of memory recall, such as the strong tendency to retrieve studied items in the order in which they were first encountered. Can such models explain ecological memory behaviors, such as eye movements during encoding and retrieval of complex visual stimuli? We tested predictions from retrieved-context models using three data sets involving recognition memory and free viewing of complex scenes. Subjects reinstated sequences of eye movements from one scene-viewing episode to the next. Moreover, sequence reinstatement decayed over time and was associated with successful memory. We observed memory-driven reinstatement even after accounting for intrinsic scene properties that produced consistent eye movements. These findings confirm predictions of retrieved-context models, suggesting retrieval of temporal context influences complex behaviors generated during naturalistic memory experiences. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Visual Exploration at Higher Fixation Frequency Increases Subsequent Memory Recall

Only a small proportion of what we see can later be recalled. Up to date it is unknown how far differences in visual exploration during encoding affect the strength of episodic memories. Here, we identified individual gaze characteristics by analyzing eye tracking data in a picture encoding task performed by 967 healthy subjects during fMRI. We found a positive correlation between fixation frequency during visual exploration and subsequent free recall performance. Brain imaging results showed a positive correlation of fixation frequency with activations in regions related to vision and memory, including the medial temporal lobe. To investigate if higher fixation frequency is causally linked to better memory, we experimentally manipulated visual exploration patterns in an independent population of 64 subjects. Doubling the number of fixations within a given exploration time increased subsequent free recall performance by 19%. Our findings provide evidence for a causal relationship between fixation frequency and episodic memory for visual information.

Long-term memory and hippocampal function support predictive gaze control during goal-directed search

Eye movements during visual search change with prior experience for search stimuli. Previous studies measured these gaze effects shortly after initial viewing, typically during free viewing; it remains open whether the effects are preserved across long delays and for goal-directed search, and which memory system guides gaze. In Experiment 1, we analyzed eye movements of healthy adults viewing novel and repeated scenes while searching for a scene-embedded target. The task was performed across different time points to examine the repetition effects in long-term memory, and memory types were grouped based on explicit recall of targets. In Experiment 2, an amnesic person with bilateral extended hippocampal damage and the age-matched control group performed the same task with shorter intervals to determine whether or not the repetition effects depend on hippocampal function. When healthy adults explicitly remembered repeated target-scene pairs, search time and fixation duration decreased, and gaze was directed closer to the target region, than when they forgot targets. These effects were seen even after a one-month delay from their initial viewing, suggesting the effects are associated with long-term, explicit memory. Saccadic amplitude was not strongly modulated by scene repetition or explicit recall of targets. The amnesic person did not show explicit recall or implicit repetition effects, whereas his control group showed similar patterns to those seen in Experiment 1. The results reveal several aspects of gaze control that are influenced by long-term memory. The dependence of gaze effects on medial temporal lobe integrity support a role for this region in predictive gaze control.

Eye movements support behavioral pattern completion

The ability to recall a detailed event from a simple reminder is supported by pattern completion, a cognitive operation performed by the hippocampus wherein existing mnemonic representations are retrieved from incomplete input. In behavioral studies, pattern completion is often inferred through the false endorsement of lure (i.e., similar) items as old. However, evidence that such a response is due to the specific retrieval of a similar, previously encoded item is severely lacking. We used eye movement (EM) monitoring during a partial-cue recognition memory task to index reinstatement of lure images behaviorally via the recapitulation of encoding-related EMs or gaze reinstatement. Participants reinstated encoding-related EMs following degraded retrieval cues and this reinstatement was negatively correlated with accuracy for lure images, suggesting that retrieval of existing representations (i.e., pattern completion) underlies lure false alarms. Our findings provide evidence linking gaze reinstatement and pattern completion and advance a functional role for EMs in memory retrieval.

Representation of viewed and recalled film clips in patterns of brain activity in a person with developmental amnesia

As clear memories transport us back into the past, the brain also revives prior patterns of neural activity, a phenomenon known as neural reactivation. While growing evidence indicates a link between neural reactivation and typical variations in memory performance in healthy individuals, it is unclear how and to what extent reactivation is disrupted by a memory disorder. The current study characterizes neural reactivation in a case of amnesia using Multivoxel Pattern Analysis (MVPA). We tested NC, an individual with developmental amnesia linked to a diencephalic stroke, and 19 young adult controls on a functional magnetic resonance imaging (fMRI) task during which participants viewed and recalled short videos multiple times. An encoding classifier trained and tested to identify videos based on brain activity patterns elicited at perception revealed superior classification in NC. The enhanced consistency in stimulus representation we observed in NC at encoding was accompanied by an absence of multivariate repetition suppression, which occurred over repeated viewing in the controls. Another recall classifier trained and tested to identify videos during mental replay indicated normal levels of classification in NC, despite his poor memory for stimulus content. However, a cross-condition classifier trained on perception trials and tested on mental replay trials-a strict test of reactivation-revealed significantly poorer classification in NC. Thus, while NC's brain activity was consistent and stimulus-specific during mental replay, this specificity did not reflect the reactivation of patterns elicited at perception to the same extent as controls. Fittingly, we identified brain regions for which activity supported stimulus representation during mental replay to a greater extent in NC than in controls. This activity was not modeled on perception, suggesting that compensatory patterns of representation based on generic knowledge can support consistent mental constructs when memory is faulty. Our results reveal several ways in which amnesia impacts distributed patterns of stimulus representation during encoding and retrieval.

Hippocampal theta coordinates memory processing during visual exploration

The hippocampus supports memory encoding and retrieval, which may occur at distinct phases of the theta cycle. These processes dynamically interact over rapid timescales, especially when sensory information conflicts with memory. The ability to link hippocampal dynamics to memory-guided behaviors has been limited by experiments that lack the temporal resolution to segregate encoding and retrieval. Here, we simultaneously tracked eye movements and hippocampal field potentials while neurosurgical patients performed a spatial memory task. Phase-locking at the peak of theta preceded fixations to retrieved locations, indicating that the hippocampus coordinates memory-guided eye movements. In contrast, phase-locking at the trough of theta followed fixations to novel object-locations and predicted intact memory of the original location. Theta-gamma phase amplitude coupling increased during fixations to conflicting visual content, but predicted memory updating. Hippocampal theta thus supports learning through two interleaved processes: strengthening encoding of novel information and guiding exploration based on prior experience.

The intersection between the oculomotor and hippocampal memory systems: empirical developments and clinical implications

Decades of cognitive neuroscience research has shown that where we look is intimately connected to what we remember. In this article, we review findings from human and nonhuman animals, using behavioral, neuropsychological, neuroimaging, and computational modeling methods, to show that the oculomotor and hippocampal memory systems interact in a reciprocal manner, on a moment-to-moment basis, mediated by a vast structural and functional network. Visual exploration serves to efficiently gather information from the environment for the purpose of creating new memories, updating existing memories, and reconstructing the rich, vivid details from memory. Conversely, memory increases the efficiency of visual exploration. We call for models of oculomotor control to consider the influence of the hippocampal memory system on the cognitive control of eye movements, and for models of hippocampal and broader medial temporal lobe function to consider the influence of the oculomotor system on the development and expression of memory. We describe eye movement-based applications for the detection of neurodegeneration and delivery of therapeutic interventions for mental health disorders for which the hippocampus is implicated and memory dysfunctions are at the forefront.

Gradual learning and inflexible strategy use in amnesia: Evidence from case H.C

The value of case studies in informing our understanding of dissociations and interactions in memory was recognized early on by Endel Tulving, whose comprehensive work with the amnesic case K.C. helped to confirm distinctions between episodic and semantic memory. Following in this tradition, we examined memory and the use of cognitive strategies in the developmental amnesic case H.C., a young woman with structural abnormalities in the extended hippocampal system (Rosenbaum et al., 2014). H.C. was tested on two tasks, transitivity and transverse patterning, that each required learning the relations among items, and for the former, also examined the ability to make inferences across sets of relations. H.C. was tested across multiple sessions and demonstrated two seemingly contradictory patterns of performance: evidence of gradual learning, yet an inability to flexibly switch to a cognitive strategy that may otherwise benefit performance. Specifically, on the transitivity task, H.C. showed gradual learning of novel relations that led to successful inferential performance. On transverse patterning, H.C. showed some gradual learning of the relations among the objects across sessions, and expressed knowledge that the task followed 'rock-paper-scissors' rules. However, H.C. did not benefit from a unitization strategy, which had shown previous success with other amnesic cases (D'Angelo et al., 2015; Ryan, Moses, Barense, & Rosenbaum, 2013). H.C.'s over-reliance on 'rock-paper-scissors' rules, even in the face of alternate strategies, is suggestive of an inability to enact cognitive flexibility. Poor performance thus may have resulted from interference from the experimentally presented strategy on her self-imposed strategy. The present findings echo work reported by Tulving in case K.C. (Tulving, Hayman, & Macdonald, 1991). Whereas neurologically intact individuals may rely on the functions of the hippocampal system to rapidly learn new information and resolve interference, some individuals with hippocampal amnesia may learn information gradually, but such learning is particularly prone to interference, resulting in an inability to flexibly adapt to changes in the learning conditions in order to optimize performance.

Gaze direction reveals implicit item and source memory in older adults

This study looked at eye movements in relation to source memory in older adults. Participants first studied images of common objects appearing in different quadrants of a screen. After a delay, they were shown cues one at a time presented in all four quadrants. Participants stated whether or not the cue had been seen before and in which location. Participants also rated level of confidence in their responses. In trials where participants either claimed they have not seen a previously presented cue or placed it in an incorrect location, they looked significantly more at the correct quadrant. The proportion of time looking at the correct quadrants during incorrect responses was not related to confidence ratings. These results suggest that eye gaze during the memory task does not reflect memory retrieval below the threshold of verbal report. They instead point to an implicit form of source memory in humans that is accessible to eye movements but not to verbal responses.

The hippocampus supports deliberation during value-based decisions

Choosing between two items involves deliberation and comparison of the features of each item and its value. Such decisions take more time when choosing between options of similar value, possibly because these decisions require more evidence, but the mechanisms involved are not clear. We propose that the hippocampus supports deliberation about value, given its well-known role in prospection and relational cognition. We assessed the role of the hippocampus in deliberation in two experiments. First, using fMRI in healthy participants, we found that BOLD activity in the hippocampus increased as a function of deliberation time. Second, we found that patients with hippocampal damage exhibited more stochastic choices and longer reaction times than controls, possibly due to their failure to construct value-based or internal evidence during deliberation. Both sets of results were stronger in value-based decisions compared to perceptual decisions.

Scenes facilitate associative memory and integration

How do we form mental links between related items? Forming associations between representations is a key feature of episodic memory and provides the foundation for learning and guiding behavior. Theories suggest that spatial context plays a supportive role in episodic memory, providing a scaffold on which to form associations, but this has mostly been tested in the context of autobiographical memory. We examined the memory boosting effect of spatial stimuli in memory using an associative inference paradigm combined with eye-tracking. Across two experiments, we found that memory was better for associations that included scenes, even indirectly, compared to objects and faces. Eye-tracking measures indicated that these effects may be partly mediated by greater fixations to scenes compared to objects, but did not explain the differences between scenes and faces. These results suggest that scenes facilitate associative memory and integration across memories, demonstrating evidence in support of theories of scenes as a spatial scaffold for episodic memory. A shared spatial context may promote learning and could potentially be leveraged to improve learning and memory in educational settings or for memory-impaired populations.

Eye Movements Actively Reinstate Spatiotemporal Mnemonic Content

Eye movements support memory encoding by binding distinct elements of the visual world into coherent representations. However, the role of eye movements in memory retrieval is less clear. We propose that eye movements play a functional role in retrieval by reinstating the encoding context. By overtly shifting attention in a manner that broadly recapitulates the spatial locations and temporal order of encoded content, eye movements facilitate access to, and reactivation of, associated details. Such mnemonic gaze reinstatement may be obligatorily recruited when task demands exceed cognitive resources, as is often observed in older adults. We review research linking gaze reinstatement to retrieval, describe the neural integration between the oculomotor and memory systems, and discuss implications for models of oculomotor control, memory, and aging.

Object-in-place Memory Predicted by Anterolateral Entorhinal Cortex and Parahippocampal Cortex Volume in Older Adults

The lateral portion of the entorhinal cortex is one of the first brain regions affected by tau pathology, an important biomarker for Alzheimer disease. Improving our understanding of this region's cognitive role may help identify better cognitive tests for early detection of Alzheimer disease. Based on its functional connections, we tested the idea that the human anterolateral entorhinal cortex (alERC) may play a role in integrating spatial information into object representations. We recently demonstrated that the volume of the alERC was related to processing the spatial relationships of the features within an object [Yeung, L. K., Olsen, R. K., Bild-Enkin, H. E. P., D'Angelo, M. C., Kacollja, A., McQuiggan, D. A., et al. Anterolateral entorhinal cortex volume predicted by altered intra-item configural processing. Journal of Neuroscience, 37, 5527–5538, 2017]. In this study, we investigated whether the human alERC might also play a role in processing the spatial relationships between an object and its environment using an eye-tracking task that assessed visual fixations to a critical object within a scene. Guided by rodent work, we measured both object-in-place memory, the association of an object with a given context [Wilson, D. I., Langston, R. F., Schlesiger, M. I., Wagner, M., Watanabe, S., & Ainge, J. A. Lateral entorhinal cortex is critical for novel object-context recognition. Hippocampus, 23, 352–366, 2013], and object-trace memory, the memory for the former location of objects [Tsao, A., Moser, M. B., & Moser, E. I. Traces of experience in the lateral entorhinal cortex. Current Biology, 23, 399–405, 2013]. In a group of older adults with varying stages of brain atrophy and cognitive decline, we found that the volume of the alERC and the volume of the parahippocampal cortex selectively predicted object-in-place memory, but not object-trace memory. These results provide support for the notion that the alERC may integrate spatial information into object representations.

Age-related changes to oscillatory dynamics during maintenance and retrieval in a relational memory task

In aging, structural and/or functional brain changes may precede changes in cognitive performance. We previously showed that despite having hippocampal volumes similar to those of younger adults, older adults showed oscillatory changes during the encoding phase of a short-delay visuospatial memory task that required spatial relations among objects to be bound across time (Rondina et al., 2016). The present work provides a complementary set of analyses to examine age-related changes in oscillatory activity during maintenance and retrieval of those spatial relations in order to provide a comprehensive examination of the neural dynamics that support memory function in aging. Participants were presented with three study objects sequentially. Following a delay (maintenance phase), the objects were re-presented simultaneously and participants had to determine whether the relative spatial relations among the objects had been maintained (retrieval phase). Older adults had similar task accuracy, but slower response times, compared to younger adults. Both groups showed a decrease in theta (2-7Hz), alpha (9-14Hz), and beta (15-30Hz) power during the maintenance phase. During the retrieval phase, younger adults showed theta and beta power increases that predicted greater task accuracy, whereas older adults showed a widespread decrease in each of the three frequency ranges that predicted longer response latencies. Older adults also showed distinct patterns of behaviour-related activity depending on whether the analysis was time-locked to the onset of the stimulus or to the onset of the response during the test phase. These findings suggest that older adults may experience declines in relational binding and/or comparison processes that are reflected in oscillatory changes prior to structural decline.

Awareness of what is learned as a characteristic of hippocampus-dependent memory

We explored the relationship between memory performance and conscious knowledge (or awareness) of what has been learned in memory-impaired patients with hippocampal lesions or larger medial temporal lesions. Participants viewed familiar scenes or familiar scenes where a change had been introduced. Patients identified many fewer of the changes than controls. Across all of the scenes, controls preferentially directed their gaze toward the regions that had been changed whenever they had what we term robust knowledge about the change: They could identify that a change occurred, report what had changed, and indicate where the change occurred. Preferential looking did not occur when they were unaware of the change or had only partial knowledge about it. The patients, overall, did not direct their gaze toward the regions that had been changed, but on the few occasions when they had robust knowledge about the change they (like controls) did exhibit this effect. Patients did not exhibit this effect when they were unaware of the change or had partial knowledge. The findings support the idea that awareness of what has been learned is a key feature of hippocampus-dependent memory.

The Hippocampus Promotes Effective Saccadic Information Gathering in Humans

It is well established that the hippocampus is critical for memory. Recent evidence suggests that one function of hippocampal memory processing is to optimize how people actively explore the world. Here we demonstrate that the link between the hippocampus and exploration extends even to the moment-to-moment use of eye movements during visuospatial memory encoding. In Experiment 1, we examined relationships between study-phase eye movements in healthy individuals and subsequent performance on a spatial reconstruction test. In addition to quantitative measures of viewing behaviors (e.g., how many fixations or saccades were deployed during study), we used the information-theoretic measure of entropy to assess the amount of randomness or disorganization in participants' scanning behaviors. We found that the use of scanpaths during study that were lower in entropy (e.g., more organized, less random) predicted more accurate spatial reconstruction both within and between participants. Scanpath entropy was a better predictor of reconstruction accuracy than were the quantitative measures of viewing. In Experiment 2, we found that individuals with hippocampal amnesia tended to engage in viewing patterns that were higher in entropy (less organized) relative to healthy comparisons. These findings reveal a critical role of the hippocampus in guiding eye movement exploration to optimize visuospatial relational memory.

Age-related changes in the relationship between visual exploration and hippocampal activity

Deciphering the mechanisms underlying age-related memory declines remains an important goal in cognitive neuroscience. Recently, we observed that visual sampling behavior predicted activity within the hippocampus, a region critical for memory. In younger adults, increases in the number of gaze fixations were associated with increases in hippocampal activity (Liu et al., 2017). This finding suggests a close coupling between the oculomotor and memory system. However, the extent to which this coupling is altered with aging has not been investigated. In this study, we gave older adults the same face processing task used in Liu et al. (2017) and compared their visual exploration behavior and neural activation in the hippocampus and the fusiform face area (FFA) to those of younger adults. Compared to younger adults, older adults showed an increase in visual exploration as indexed by the number of gaze fixations. However, the relationship between visual exploration and neural responses in the hippocampus and FFA was weaker than that of younger adults. Older adults also showed weaker responses to novel faces and a smaller repetition suppression effect in the hippocampus and FFA compared to younger adults. All together, this study provides novel evidence that the capacity to bind visually sampled information, in real-time, into coherent representations along the ventral visual stream and the medial temporal lobe declines with aging.

How the Brain's Navigation System Shapes Our Visual Experience

We explore the environment not only by navigating, but also by viewing our surroundings with our eyes. Here we review growing evidence that the mammalian hippocampal formation, extensively studied in the context of navigation and memory, mediates a representation of visual space that is stably anchored to the external world. This visual representation puts the hippocampal formation in a central position to guide viewing behavior and to modulate visual processing beyond the medial temporal lobe (MTL). We suggest that vision and navigation share several key computational challenges that are solved by overlapping and potentially common neural systems, making vision an optimal domain to explore whether and how the MTL supports cognitive operations beyond navigation.

Eye movements support the link between conscious memory and medial temporal lobe function

When individuals select the recently studied (and familiar) item in a multiple-choice memory test, they direct a greater proportion of viewing time toward the to-be-selected item when their choice is correct than when their choice is incorrect. Thus, for both correct and incorrect choices, individuals indicate that the chosen item is old, but viewing time nevertheless distinguishes between old and new items. What kind of memory supports this preferential viewing effect? We recorded eye movements while participants made three-alternative, forced-choice recognition memory judgments for scenes. In experiment 1 (n = 30), the magnitude of the preferential viewing effect was strongly correlated with measures of conscious, declarative memory: recognition accuracy as well as the difference in confidence ratings and in response times for correct and incorrect choices. In four analyses that minimized the contribution of declarative memory in order to detect a possible contribution from other processes, the preferential viewing effect was absent. In experiment 2, five memory-impaired patients with medial temporal lobe lesions exhibited a diminished preferential viewing effect. These patients also exhibited poor recognition accuracy and reduced differences in confidence ratings and response times for correct and incorrect choices. We propose that the preferential viewing effect is a phenomenon of conscious, declarative memory and is dependent on the medial temporal lobe. The findings support the link between medial temporal lobe function and declarative memory. When the effects of experience depend on the medial temporal lobe, the effects reflect conscious memory.

Visual Sampling Predicts Hippocampal Activity

Eye movements serve to accumulate information from the visual world, contributing to the formation of coherent memory representations that support cognition and behavior. The hippocampus and the oculomotor network are well connected anatomically through an extensive set of polysynaptic pathways. However, the extent to which visual sampling behavior is related to functional responses in the hippocampus during encoding has not been studied directly in human neuroimaging. In the current study, participants engaged in a face processing task while brain responses were recorded with fMRI and eye movements were monitored simultaneously. The number of gaze fixations that a participant made on a given trial was correlated significantly with hippocampal activation such that more fixations were associated with stronger hippocampal activation. Similar results were also found in the fusiform face area, a face-selective perceptual processing region. Notably, the number of fixations was associated with stronger hippocampal activation when the presented faces were novel, but not when the faces were repeated. Increases in fixations during viewing of novel faces also led to larger repetition-related suppression in the hippocampus, indicating that this fixation-hippocampal relationship may reflect the ongoing development of lasting representations. Together, these results provide novel empirical support for the idea that visual exploration and hippocampal binding processes are inherently linked.

SIGNIFICANCE STATEMENT

The hippocampal and oculomotor networks have each been studied extensively for their roles in the binding of information and gaze function, respectively. Despite the evidence that individuals with amnesia whose damage includes the hippocampus show alterations in their eye movement patterns and recent findings that the two systems are anatomically connected, it has not been demonstrated whether visual exploration is related to hippocampal activity in neurologically intact adults. In this combined fMRI-eye-tracking study, we show how hippocampal responses scale with the number of gaze fixations made during viewing of novel, but not repeated, faces. These findings provide new evidence suggesting that the hippocampus plays an important role in the binding of information, as sampled by gaze fixations, during visual exploration.

An Eye-Movement Study of relational Memory in Adults with Autism Spectrum Disorder

Persons with Autism Spectrum Disorder (ASD) demonstrate good memory for single items but difficulties remembering contextual information related to these items. Recently, we found compromised explicit but intact implicit retrieval of object-location information in ASD (Ring et al. Autism Res 8(5):609–619, 2015). Eye-movement data collected from a sub-sample of the participants are the focus of the current paper. At encoding, trial-by-trial viewing durations predicted subsequent retrieval success only in typically developing (TD) participants. During retrieval, TD compared to ASD participants looked significantly longer at previously studied object-locations compared to alternative locations. These findings extend similar observations recently reported by Cooper et al. (Cognition 159:127–138, 2017a) and demonstrate that eye-movement data can shed important light on the source and nature of relational memory difficulties in ASD.

A Closer Look at the Hippocampus and Memory

Current interpretations of hippocampal memory function are blind to the fact that viewing behaviors are pervasive and complicate the relationships among perception, behavior, memory, and brain activity. For example, hippocampal activity and associative memory demands increase with stimulus complexity. Stimulus complexity also strongly modulates viewing. Associative processing and viewing thus are often confounded, rendering interpretation of hippocampal activity ambiguous. Similar considerations challenge many accounts of hippocampal function. To explain relationships between memory and viewing, we propose that the hippocampus supports the online memory demands necessary to guide visual exploration. The hippocampus thus orchestrates memory-guided exploration that unfolds over time to build coherent memories. This new perspective on hippocampal function harmonizes with the fact that memory formation and exploratory viewing are tightly intertwined.

Anterolateral Entorhinal Cortex Volume Predicted by Altered Intra-Item Configural Processing

Recent functional imaging studies have proposed that the human entorhinal cortex (ERC) is subdivided into functionally distinct anterolateral (alERC) and posteromedial (pmERC) subregions. The alERC overlaps with regions that are affected earliest by Alzheimer's disease pathology, yet its cognitive function remains poorly understood. Previous human fMRI studies have focused on its role in object memory, but rodent studies on the putatively homologous lateral entorhinal cortex suggest that it also plays an important role in representing spatial properties of objects. To investigate the cognitive effects of human alERC volume differences, we developed an eye-tracking-based task to evaluate intra-item configural processing (i.e., processing the arrangement of an object's features) and used manual segmentation based on a recently developed protocol to delineate the alERC/pmERC and other medial temporal lobe (MTL) subregions. In a group of older adult men and women at varying stages of brain atrophy and cognitive decline, we found that intra-item configural processing, regardless of an object's novelty, was strongly predicted by alERC volume, but not by the volume of any other MTL subregion. These results provide the first evidence that the human alERC plays a role in supporting a distinct aspect of object processing, namely attending to the arrangement of an object's component features.SIGNIFICANCE STATEMENT Alzheimer's disease pathology appears earliest in brain regions that overlap with the anterolateral entorhinal cortex (alERC). However, the cognitive role of the alERC is poorly understood. Previous human studies treat the alERC as an extension of the neighboring perirhinal cortex, supporting object memory. Animal studies suggest that the alERC may support the spatial properties of objects. In a group of older adult humans at the earliest stages of cognitive decline, we show here that alERC volume selectively predicted configural processing (attention to the spatial arrangement of an object's parts). This is the first study to demonstrate a cognitive role related to alERC volume in humans. This task can be adapted to serve as an early detection method for Alzheimer's disease pathology.

When eye movements express memory for old and new scenes in the absence of awareness and independent of hippocampus

Eye movements can reflect memory. For example, participants make fewer fixations and sample fewer regions when viewing old versus new scenes (the repetition effect). It is unclear whether the repetition effect requires that participants have knowledge (awareness) of the old–new status of the scenes or if it can occur independent of knowledge about old–new status. It is also unclear whether the repetition effect is hippocampus-dependent or hippocampus-independent. A complication is that testing conscious memory for the scenes might interfere with the expression of unconscious (unaware), experience-dependent eye movements. In experiment 1, 75 volunteers freely viewed old and new scenes without knowledge that memory for the scenes would later be tested. Participants then made memory judgments and confidence judgments for each scene during a surprise recognition memory test. Participants exhibited the repetition effect regardless of the accuracy or confidence associated with their memory judgments (i.e., the repetition effect was independent of their awareness of the old–new status of each scene). In experiment 2, five memory-impaired patients with medial temporal lobe damage and six controls also viewed old and new scenes without expectation of memory testing. Both groups exhibited the repetition effect, even though the patients were impaired at recognizing which scenes were old and which were new. Thus, when participants viewed scenes without expectation of memory testing, eye movements associated with old and new scenes reflected unconscious, hippocampus-independent memory. These findings are consistent with the formulation that, when memory is expressed independent of awareness, memory is hippocampus-independent.