Eyes on Memory: Pupillometry in Encoding and Retrieval

This review critically examines the contributions of pupillometry to memory research, primarily focusing on its enhancement of our understanding of memory encoding and retrieval mechanisms mainly investigated with the recognition memory paradigm. The evidence supports a close link between pupil response and memory formation, notably influenced by the type of novelty detected. This proposal reconciles inconsistencies in the literature regarding pupil response patterns that may predict successful memory formation, and highlights important implications for encoding mechanisms. The review also discusses the pupil old/new effect and its significance in the context of recollection and in reflecting brain signals related to familiarity or novelty detection. Additionally, the capacity of pupil response to serve as a true memory signal and to distinguish between true and false memories is evaluated. The evidence provides insights into the nature of false memories and offers a novel understanding of the cognitive mechanisms involved in memory distortions. When integrated with rigorous experimental design, pupillometry can significantly refine theoretical models of memory encoding and retrieval. Furthermore, combining pupillometry with neuroimaging and pharmacological interventions is identified as a promising direction for future research.

Is implicit memory associated with the hippocampus?

According to the traditional memory-systems view, the hippocampus is critical during explicit (conscious) long-term memory, whereas other brain regions support implicit (nonconscious) memory. In the last two decades, some fMRI studies have reported hippocampal activity during implicit memory tasks. The aim of the present discussion paper was to identify whether any implicit memory fMRI studies have provided convincing evidence that the hippocampus is associated with nonconscious processes without being confounded by conscious processes. Experimental protocol and analysis parameters included the stimulus type(s), task(s), measures of subjective awareness, explicit memory accuracy, the relevant fMRI contrast(s) or analysis, and confound(s). A systematic review was conducted to identify implicit memory studies that reported fMRI activity in the hippocampus. After applying exclusion criteria, 13 articles remained for analysis. We found that there were no implicit memory fMRI studies where subjective awareness was absent, explicit memory performance was at chance, and there were no confounds that could have driven the observed hippocampal activity. The confounds included explicit memory (including false memory), imbalanced attentional states between conditions (yielding activation of the default-mode network), imbalanced stimuli between conditions, and differential novelty. As such, not a single fMRI study provided convincing evidence that implicit memory was associated with the hippocampus. Neuropsychological evidence was also considered, and implicit memory deficits were caused by factors known to disrupt brain regions beyond the hippocampus, such that the behavioral effects could not be attributed to this region. The present results indicate that implicit memory is not associated with the hippocampus.

Pupil response patterns distinguish true from false memories

Memory is reconstructive and error-prone, which make memory illusions very common in everyday life. However, studying memory illusions can provide valuable insights into how memory works. Pupil response has emerged, in recent years, as an indicator of memory encoding and retrieval, however its validity as a measure of memory success is debated. In this study, we explored whether pupil response patterns can differentiate true from false memories and whether variations in the temporal dynamics of pupil response can elucidate the mechanisms underlying false memory creation. The Deese-Roediger-McDermott (DRM) paradigm was employed to generate false memories in two separate experiments involving visual and auditory stimuli. Pupil responses effectively differentiated true from false memories based on variations in pupil amplitude at different temporal components. This discrimination remained consistent across both experiments, with slightly stronger effects in the auditory condition, aligning with the more pronounced false memory effects in this condition. Notably, differential pupil responses between true and false memories varied based on the type of memory involved at recognition. These findings provide valuable insights into the cognitive processes underlying memory distortions, with implications for theoretical frameworks and real-world contexts.

Can pupillometry distinguish accurate from inaccurate familiarity?

Pupillometry, the measurement of pupil diameter, has become increasingly popular as a tool to investigate human memory. It has long been accepted that the pupil is able to distinguish familiar from completely novel items, a phenomenon known as "pupil old/new effect". Surprisingly, most pupillometric studies on the pupil old/new effect tend to disregard the possibility that the pupillary response to familiarity memory may not be entirely exclusive. Here, we investigated whether the pupillary response to old items correctly judged familiar (hits; accurate familiarity) can be differentiated from the pupillary response to new items wrongly judged familiar (false alarms; inaccurate familiarity). We found no evidence that the two processes could be isolated, as both accurate and inaccurate familiarity showed nearly identical mean and across-time pupillary responses. However, both familiarity hits and false alarms showed pupillary responses unequivocally distinct from those observed during either recollection or novelty detection, which suggests that the pupil measure of familiarity hits and/or false alarms was sufficiently sensitive. The pupillary response to false alarms may have been partially driven by perceptual fluency, since novel objects incorrectly judged to be old (i.e., false alarms) showed a higher degree of similarity to studied images than items correctly judged as novel (i.e., correct rejections). Thus, our results suggest that pupil dilation may not be able to distinguish accurate from inaccurate familiarity using standard recognition memory paradigms, and they also suggest that the pupillary response during familiarity feelings may also partly reflect perceptual fluency.

Expecting Questions Modulates Cognitive Effort in a Syntactic Processing Task: Evidence From Pupillometry

Purpose Pupillary responses captured via pupillometry (measurement of pupillary dilation and constriction during the performance of a cognitive task) are psychophysiological indicators of cognitive effort, attention, arousal, and resource engagement. Pupillometry may be a promising tool for enhancing our understanding of the relationship between cognition and language in people with and without aphasia. Interpretation of pupillary responses is complex. This study was designed as a stepping-stone for future pupillometric studies involving people with aphasia. Asking comprehension questions is common in language processing research involving people with and without aphasia. However, the influence of comprehension questions on pupillometric indices of task engagement (tonic responses) and cognitive effort (task-evoked responses of the pupil [TERPs]) is unknown. We tested whether asking comprehension questions influenced pupillometric results of adults without aphasia during a syntactic processing task. Method Forty adults without aphasia listened to easy (canonical) and difficult (noncanonical) sentences in two conditions: one that contained an explicit comprehension task (question condition) and one that did not (no-question condition). The influence of condition and canonicity on pupillary responses was examined. Results The influence of canonicity was only significant in the question condition: TERPs for difficult sentences were larger than TERPs for easy sentences. Tonic responses did not differ between conditions. Conclusions Although participants had similar levels of attentiveness in both conditions, increases in indices of cognitive effort during syntactic processing were significant only when participants expected comprehension questions. Results contribute to a body of evidence indicating the importance of task design and careful linguistic stimulus control when using pupillometry to study language processing. Supplemental Material https://doi.org/10.23641/asha.13480368.

Do eye movements enhance visual memory retrieval?

When remembering an object at a given location, participants tend to return their gaze to that location even after the object has disappeared, known as Looking-at-Nothing (LAN). However, it is unclear whether LAN is associated with better memory performance. Previous studies reporting beneficial effects of LAN have often not systematically manipulated or assessed eye movements. We asked 20 participants to remember the location and identity of eight objects arranged in a circle, shown for 5 s. Participants were prompted to judge whether a location statement (e.g., "Star Right") was correct or incorrect, or referred to a previously unseen object. During memory retrieval, participants either fixated in the screen center or were free to move their eyes. Results reveal no difference in memory accuracy and response time between free-viewing and fixation while a LAN effect was found for saccades during free viewing, but not for microsaccades during fixation. Memory performance was better in those free-viewing trials in which participants made a saccade to the critical location, and scaled with saccade accuracy. These results indicate that saccade kinematics might be related to both memory performance and memory retrieval processes, but the strength of their link would differ between individuals and task demands.

Gaze direction reveals implicit item and source memory in older adults

This study looked at eye movements in relation to source memory in older adults. Participants first studied images of common objects appearing in different quadrants of a screen. After a delay, they were shown cues one at a time presented in all four quadrants. Participants stated whether or not the cue had been seen before and in which location. Participants also rated level of confidence in their responses. In trials where participants either claimed they have not seen a previously presented cue or placed it in an incorrect location, they looked significantly more at the correct quadrant. The proportion of time looking at the correct quadrants during incorrect responses was not related to confidence ratings. These results suggest that eye gaze during the memory task does not reflect memory retrieval below the threshold of verbal report. They instead point to an implicit form of source memory in humans that is accessible to eye movements but not to verbal responses.

The Pupillary Response to the Unknown: Novelty Versus Familiarity

Object recognition is a type of perception that enables observers to recognize familiar shapes and categorize them into real-world identities. In this preregistered study, we aimed to determine whether pupil size changes occur during the perception and recognition of identifiable objects. We compared pupil size changes for familiar objects, nonobjects, and random noise. Nonobjects and noise produced greater pupil dilation than familiar objects. Contrary to previous evidence showing greater pupil dilation to stimuli with more perceptual and affective content, these results indicate a greater pupil dilation to stimuli that are unidentifiable. This is consistent with the relative salience of novelty compared to familiarity at the physiological level driving the pupil response.

Patients with hippocampal amnesia successfully integrate gesture and speech

During conversation, people integrate information from co-speech hand gestures with information in spoken language. For example, after hearing the sentence, "A piece of the log flew up and hit Carl in the face" while viewing a gesture directed at the nose, people tend to later report that the log hit Carl in the nose (information only in gesture) rather than in the face (information in speech). The cognitive and neural mechanisms that support the integration of gesture with speech are unclear. One possibility is that the hippocampus - known for its role in relational memory and information integration - is necessary for integrating gesture and speech. To test this possibility, we examined how patients with hippocampal amnesia and healthy and brain-damaged comparison participants express information from gesture in a narrative retelling task. Participants watched videos of an experimenter telling narratives that included hand gestures that contained supplementary information. Participants were asked to retell the narratives and their spoken retellings were assessed for the presence of information from gesture. For features that had been accompanied by supplementary gesture, patients with amnesia retold fewer of these features overall and fewer retellings that matched the speech from the narrative. Yet their retellings included features that contained information that had been present uniquely in gesture in amounts that were not reliably different from comparison groups. Thus, a functioning hippocampus is not necessary for gesture-speech integration over short timescales. Providing unique information in gesture may enhance communication for individuals with declarative memory impairment, possibly via non-declarative memory mechanisms.

Pupillary response indexes the metrical hierarchy of unattended rhythmic violations

The perception of music is a complex interaction between what we hear and our interpretation. This is reflected in beat perception, in which a listener infers a regular pulse from a musical rhythm. Although beat perception is a fundamental human ability, it is still unknown whether attention to the music is necessary to establish the perception of stronger and weaker beats, or meter. In addition, to what extent beat perception is dependent on musical expertise is still a matter of debate. Here, we address these questions by measuring the pupillary response to omissions at different metrical positions in drum rhythms, while participants attended to another task. We found that the omission of the salient first beat elicited a larger pupil dilation than the omission of the less-salient second beat. This result shows that participants not only detected the beat without explicit attention to the music, but also perceived a metrical hierarchy of stronger and weaker beats. This suggests that hierarchical beat perception is an automatic process that requires no or minimal attentional resources. In addition, we found no evidence for the hypothesis that hierarchical beat perception is affected by musical expertise, suggesting that elementary beat perception might be independent from musical expertise. Finally, our results show that pupil dilation reflects surprise without explicit attention, demonstrating that the pupil is an accessible index to signatures of unattentive processing.

Pupil old/new effects reflect stimulus encoding and decoding in short-term memory

We conducted five pupil old/new experiments to examine whether pupil old/new effects can be linked to familiarity and/or recollection processes of recognition memory. In Experiments 1-3, we elicited robust pupil old/new effects for legal words and pseudowords (Experiment 1), positive and negative words (Experiment 2), and low-frequency and high-frequency words (Experiment 3). Importantly, unlike for old/new effects in ERPs, we failed to find any effects of long-term memory representations on pupil old/new effects. In Experiment 4, using the words and pseudowords from Experiment 1, participants made lexical decisions instead of old/new decisions. Pupil old/new effects were restricted to legal words. Additionally requiring participants to make speeded responses (Experiment 5) led to a complete absence of old/new effects. Taken together, these data suggest that pupil old/new effects do not map onto familiarity and recollection processes of recognition memory. They rather seem to reflect strength of memory traces in short-term memory, with little influence of long-term memory representations. Crucially, weakening the memory trace through manipulations in the experimental task significantly reduces pupil/old new effects.

The pupillary response discriminates between subjective and objective familiarity and novelty

The pupil response discriminates between old and new stimuli, with old stimuli characterized by larger pupil dilation patterns than new stimuli. We sought to explore the cause of the pupil old/new effect and discount the effect of targetness, effort, recollection retrieval, and complexity of the recognition decision. Two experiments are reported in which the pupil response and the eye fixation patterns were measured, while participants identified novel and familiar object stimuli, in two separate tasks, emphasizing either novelty or familiarity detection. In Experiment 1, familiarity and novelty decisions were taken using a rating scale, while in Experiment 2 a simpler yes/no decision was used. In both experiments, we found that detection of target familiar stimuli resulted in greater pupil dilation than the detection of target novel stimuli, while the duration of the first fixation discriminated between familiar and novel stimuli as early as within 320 ms after stimulus onset. Importantly, the pupil response distinguished between the objective (during an earlier temporal component) and the subjective (during a later temporal component) status of the stimulus for misses and false alarms. In the light of previous findings, we suggest that the pupil and fixation old/new effects reflect the distinct neural and cognitive mechanisms involved in the familiarity and novelty decisions. The findings also have important implications for the use of pupil dilation and eye movement patterns to explore explicit and implicit memory processes.

Vocal acoustic analysis as a biometric indicator of information processing: implications for neurological and psychiatric disorders

Vocal expression reflects an integral component of communication that varies considerably within individuals across contexts and is disrupted in a range of neurological and psychiatric disorders. There is reason to suspect that variability in vocal expression reflects, in part, the availability of "on-line" resources (e.g., working memory, attention). Thus, understanding vocal expression is a potentially important biometric index of information processing, not only across but within individuals over time. A first step in this line of research involves establishing a link between vocal expression and information processing systems in healthy adults. The present study employed a dual attention experimental task where participants provided natural speech while simultaneously engaged in a baseline, medium or high nonverbal processing-load task. Objective, automated, and computerized analysis was employed to measure vocal expression in 226 adults. Increased processing load resulted in longer pauses, fewer utterances, greater silence overall and less variability in frequency and intensity levels. These results provide compelling evidence of a link between information processing resources and vocal expression, and provide important information for the development of an automated, inexpensive and uninvasive biometric measure of information processing.

The pupil as an indicator of unconscious memory: Introducing the pupil priming effect

We explored whether object behavioral priming and pupil changes occur in the absence of recognition memory. Experiment 1 found behavioral priming for unrecognized objects (Ms) regardless of whether they had been encoded perceptually or conceptually. Using the same perceptual encoding task, Experiment 2 showed greater pupil dilation for Ms than for correct rejections of unstudied objects (CRs) when reaction times were matched. In Experiment 3, there was relatively less pupil dilation for Ms than for similarly matched CRs when objects had been encoded conceptually. Mean/peak pupil dilation for CRs, but not Ms, increased in Experiment 3, in which novelty expectation was also reduced, and the pupillary time course for both Ms and CRs was distinct in the two experiments. These findings indicate that both behavioral and pupil memory occur for studied, but unrecognized stimuli, and suggest that encoding and novelty expectation modulate pupillary memory responses.

Task-evoked pupillometry provides a window into the development of short-term memory capacity

The capacity to keep multiple items in short-term memory (STM) improves over childhood and provides the foundation for the development of multiple cognitive abilities. The goal of this study was to measure the extent to which age differences in STM capacity are related to differences in task engagement during encoding. Children (n = 69, mean age = 10.6 years) and adults (n = 54, mean age = 27.5 years) performed two STM tasks: the forward digit span test from the Wechsler Intelligence Scale for Children (WISC) and a novel eyetracking digit span task designed to overload STM capacity. Building on prior research showing that task-evoked pupil dilation can be used as a real-time index of task engagement, we measured changes in pupil dilation while participants encoded long sequences of digits for subsequent recall. As expected, adults outperformed children on both STM tasks. We found similar patterns of pupil dilation while children and adults listened to the first six digits on our STM overload task, after which the adults' pupils continued to dilate and the children's began to constrict, suggesting that the children had reached their cognitive limits and that they had begun to disengage from the task. Indeed, the point at which pupil dilation peaked at encoding was a significant predictor of WISC forward span, and this relationship held even after partialing out recall performance on the STM overload task. These findings indicate that sustained task engagement at encoding is an important component of the development of STM.

Pupil Dilation Reflects the Creation and Retrieval of Memories

It has long been known that pupils-the apertures that allow light into the eyes-dilate and constrict not only in response to changes in ambient light but also in response to emotional changes and arousing stimuli (e.g., Fontana, 1765). Charles Darwin (1872) related changes in pupil diameter to fear and other "emotions" in animals. For decades, pupillometry has been used to study cognitive processing across many domains, including perception, language, visual search, and short-term memory. Historically, such studies have examined the pupillary reflex as a correlate of attentional demands imposed by different tasks or stimuli-pupils typically dilate as cognitive demand increases. Because the neural mechanisms responsible for such task-evoked pupillary reflexes (TEPRs) implicate a role for memory processes, recent studies have examined pupillometry as a tool for investigating the cognitive processes underlying the creation of new episodic memories and their later retrieval. Here, we review the historical antecedents of current pupillometric research and discuss several recent studies linking pupillary dilation to the on-line consumption of cognitive resources in long-term-memory tasks. We conclude by discussing the future role of pupillometry in memory research and several methodological considerations that are important when designing pupillometric studies.

Pupillometry

The measurement of pupil diameter in psychology (in short, “pupillometry”) has just celebrated 50 years. The method established itself after the appearance of three seminal studies ( Hess & Polt, 1960 , 1964 ; Kahneman & Beatty, 1966 ). Since then, the method has continued to play a significant role within the field, and pupillary responses have been successfully used to provide an estimate of the “intensity” of mental activity and of changes in mental states, particularly changes in the allocation of attention and the consolidation of perception. Remarkably, pupillary responses provide a continuous measure regardless of whether the participant is aware of such changes. More recently, research in neuroscience has revealed a tight correlation between the activity of the locus coeruleus (i.e., the “hub” of the noradrenergic system) and pupillary dilation. As we discuss in this short review, these neurophysiological findings provide new important insights to the meaning of pupillary responses for mental activity. Finally, given that pupillary responses can be easily measured in a noninvasive manner, occur from birth, and can occur in the absence of voluntary, conscious processes, they constitute a very promising tool for the study of preverbal (e.g., infants) or nonverbal participants (e.g., animals, neurological patients).

Recognition Memory Strength is Predicted by Pupillary Responses at Encoding While Fixation Patterns Distinguish Recollection from Familiarity

Thirty-five healthy participants incidentally encoded a set of man-made and natural object pictures, while their pupil response and eye movements were recorded. At retrieval, studied and new stimuli were rated as novel, familiar (strong, moderate, or weak), or recollected. We found that both pupil response and fixation patterns at encoding predict later recognition memory strength. The extent of pupillary response accompanying incidental encoding was found to be predictive of subsequent memory. In addition, the number of fixations was also predictive of later recognition memory strength, suggesting that the accumulation of greater visual detail, even for single objects, is critical for the creation of a strong memory. Moreover, fixation patterns at encoding distinguished between recollection and familiarity at retrieval, with more dispersed fixations predicting familiarity and more clustered fixations predicting recollection. These data reveal close links between the autonomic control of pupil responses and eye movement patterns on the one hand and memory encoding on the other. Moreover, the data illustrate quantitative as well as qualitative differences in the incidental visual processing of stimuli, which are differentially predictive of the strength and the kind of memory experienced at recognition.

Pupillary Stroop effects

We recorded the pupil diameters of participants performing the words' color-naming Stroop task (i.e., naming the color of a word that names a color). Non-color words were used as baseline to firmly establish the effects of semantic relatedness induced by color word distractors. We replicated the classic Stroop effects of color congruency and color incongruency with pupillary diameter recordings: relative to non-color words, pupil diameters increased for color distractors that differed from color responses, while they reduced for color distractors that were identical to color responses. Analyses of the time courses of pupil responses revealed further differences between color-congruent and color-incongruent distractors, with the latter inducing a steep increase of pupil size and the former a relatively lower increase. Consistent with previous findings that have demonstrated that pupil size increases as task demands rise, the present results indicate that pupillometry is a robust measure of Stroop interference, and it represents a valuable addition to the cognitive scientist's toolbox.

Experience-dependent eye movements reflect hippocampus-dependent (aware) memory

We investigated the relationship between experience-dependent eye movements, hippocampus-dependent memory, and aware memory. We measured eye movements in young adults, older adults, and memory-impaired patients with damage to the medial temporal lobe as they viewed 120 novel scenes and 120 previously viewed scenes. Participants indicated if each scene was old or new and also gave a confidence rating for the memory judgment. Young adults and older adults explored old scenes less than they explored new scenes, but the patients did not. For the young and older adults, this effect was observed only when participants were aware of the scene's familiar or novel status. In a second experiment, young adults viewed scenes that were either new, had been viewed previously, or had been viewed previously but had been changed (i.e., an object within the scene was either added or removed). The only instructions were to pay attention to the scenes and view each scene as it appeared, and there was no expectation that memory would be tested. Directly after the first altered scene was presented, participants were asked to classify the scene as new, old, or old but changed. Participants who were aware of the manipulation preferentially viewed the changed region, but participants who were unaware did not. These findings suggest that experience-dependent eye movements reflect hippocampus-dependent (and aware) memory, even when participants have no expectation that memory is being tested; and they are consistent with the view that awareness of what is learned is a fundamental characteristic of hippocampus-dependent memory.