Effort during visual search and counting: Insights from pupillometry

The pupillary light response reveals the focus of covert visual attention

The pupillary light response is often assumed to be a reflex that is not susceptible to cognitive influences. In line with recent converging evidence, we show that this reflexive view is incomplete, and that the pupillary light response is modulated by covert visual attention: Covertly attending to a bright area causes a pupillary constriction, relative to attending to a dark area under identical visual input. This attention-related modulation of the pupillary light response predicts cuing effects in behavior, and can be used as an index of how strongly participants attend to a particular location. Therefore, we suggest that pupil size may offer a new way to continuously track the focus of covert visual attention, without requiring a manual response from the participant. The theoretical implication of this finding is that the pupillary light response is neither fully reflexive, nor under complete voluntary control, but is instead best characterized as a stereotyped response to a voluntarily selected target. In this sense, the pupillary light response is similar to saccadic and smooth pursuit eye movements. Together, eye movements and the pupillary light response maximize visual acuity, stabilize visual input, and selectively filter visual information as it enters the eye.

Pupillary response predicts multiple object tracking load, error rate, and conscientiousness, but not inattentional blindness

Research on inattentional blindness (IB) has uncovered few individual difference measures that predict failures to detect an unexpected event. Notably, no clear relationship exists between primary task performance and IB. This is perplexing as better task performance is typically associated with increased effort and should result in fewer spare resources to process the unexpected event. We utilized a psychophysiological measure of effort (pupillary response) to explore whether differences in effort devoted to the primary task (multiple object tracking) are related to IB. Pupillary response was sensitive to tracking load and differences in primary task error rates. Furthermore, pupillary response was a better predictor of conscientiousness than primary task errors; errors were uncorrelated with conscientiousness. Despite being sensitive to task load, individual differences in performance and conscientiousness, pupillary response did not distinguish between those who noticed the unexpected event and those who did not. Results provide converging evidence that effort and primary task engagement may be unrelated to IB.

Eye-tracking, autonomic, and electrophysiological correlates of emotional face processing in adolescents with autism spectrum disorder

Individuals with autism spectrum disorder (ASD) often have difficulty with social-emotional cues. This study examined the neural, behavioral, and autonomic correlates of emotional face processing in adolescents with ASD and typical development (TD) using eye-tracking and event-related potentials (ERPs) across two different paradigms. Scanning of faces was similar across groups in the first task, but the second task found that face-sensitive ERPs varied with emotional expressions only in TD. Further, ASD showed enhanced neural responding to non-social stimuli. In TD only, attention to eyes during eye-tracking related to faster face-sensitive ERPs in a separate task; in ASD, a significant positive association was found between autonomic activity and attention to mouths. Overall, ASD showed an atypical pattern of emotional face processing, with reduced neural differentiation between emotions and a reduced relationship between gaze behavior and neural processing of faces.

The "subjective" pupil old/new effect: is the truth plain to see?

Human memory is an imperfect process, prone to distortion and errors that range from minor disturbances to major errors that can have serious consequences on everyday life. In this study, we investigated false remembering of manipulatory verbs using an explicit recognition task and pupillometry. Our results replicated the "classical" pupil old/new effect as well as data in false remembering literature that show how items must be recognize as old in order for the pupil size to increase (e.g., "subjective" pupil old/new effect), even though these items do not necessarily have to be truly old. These findings support the strength-of-memory trace account that affirms that pupil dilation is related to experience rather than to the accuracy of recognition. Moreover, behavioral results showed higher rates of true and false recognitions for manipulatory verbs and a consequent larger pupil diameter, supporting the embodied view of language.

Motivation modulates visual attention: evidence from pupillometry

Increasing evidence suggests that action planning does not only affect the preparation and execution of overt actions but also "works back" to tune the perceptual system toward action-relevant information. We investigated whether the amount of this impact of action planning on perceptual selection varies as a function of motivation for action, which was assessed online by means of pupillometry (Experiment 1) and visual analog scales (VAS, Experiment 2). Findings replicate the earlier observation that searching for size-defined targets is more efficient in the context of grasping than in the context of pointing movements (Wykowska et al., 2009). As expected, changes in tonic pupil size (reflecting changes in effort and motivation) across the sessions, as well as changes in motivation-related scores on the VAS were found to correlate with changes in the size of the action-perception congruency effect. We conclude that motivation and effort might play a crucial role in how much participants prepare for an action and activate action codes. The degree of activation of action codes in turn influences the observed action-related biases on perception.

Pupil dilation co-varies with memory strength of individual traces in a delayed response paired-associate task

Studies on cognitive effort have shown that pupil dilation is a reliable indicator of memory load. However, it is conceivable that there are other sources of effort involved in memory that also affect pupil dilation. One of these is the ease with which an item can be retrieved from memory. Here, we present the results of an experiment in which we studied the way in which pupil dilation acts as an online marker for memory processing during the retrieval of paired associates while reducing confounds associated with motor responses. Paired associates were categorized into sets containing either 4 or 7 items. After learning the paired associates once, pupil dilation was measured during the presentation of the retrieval cue during four repetitions of each set. Memory strength was operationalized as the number of repetitions (frequency) and set-size, since having more items per set results in a lower average recency. Dilation decreased with increased memory strength, supporting the hypothesis that the amplitude of the evoked pupillary response correlates positively with retrieval effort. Thus, while many studies have shown that “memory load” influences pupil dilation, our results indicate that the task-evoked pupillary response is also sensitive to the experimentally manipulated memory strength of individual items. As these effects were observed well before the response had been given, this study also suggests that pupil dilation can be used to assess an item’s memory strength without requiring an overt response.

Pupil-BLAH-metry: cognitive effort in speech planning reflected by pupil dilation

In reading research, a longstanding question is whether any stages of lexical processing require central attention, and whether such potential demands are frequency-sensitive. In the present study, we examined the allocation of cognitive effort in lexical processing by examining pupil dilations and naming latencies in a modified delayed naming procedure. In this dual-task/change procedure, participants read words and waited for various delays before being signaled to issue a response. On most trials (80%), participants issued a standard naming response. On the remaining trials, they were cued to abandon the original speech plan, saying "blah" instead, thereby equating production across different words. Using feature-matched low- and high-frequency words, we observed the differences in pupil dilations as a function of word frequency. Indeed, frequency-sensitive cognitive demands were seen in word processing, even after naming responses were issued. The results suggest that word perception and/or speech planning requires the frequency-sensitive allocation of cognitive resources.

Perception while watching movies: Effects of physical screen size and scene type

Over the last decade, television screens and display monitors have increased in size considerably, but has this improved our televisual experience? Our working hypothesis was that the audiences adopt a general strategy that "bigger is better." However, as our visual perceptions do not tap directly into basic retinal image properties such as retinal image size (C. A. Burbeck, 1987), we wondered whether object size itself might be an important factor. To test this, we needed a task that would tap into the subjective experiences of participants watching a movie on different-sized displays with the same retinal subtense. Our participants used a line bisection task to self-report their level of "presence" (i.e., their involvement with the movie) at several target locations that were probed in a 45-min section of the movie "The Good, The Bad, and The Ugly." Measures of pupil dilation and reaction time to the probes were also obtained. In Experiment 1, we found that subjective ratings of presence increased with physical screen size, supporting our hypothesis. Face scenes also produced higher presence scores than landscape scenes for both screen sizes. In Experiment 2, reaction time and pupil dilation results showed the same trends as the presence ratings and pupil dilation correlated with presence ratings, providing some validation of the method. Overall, the results suggest that real-time measures of subjective presence might be a valuable tool for measuring audience experience for different types of (i) display and (ii) audiovisual material.

Systematic influence of gaze position on pupil size measurement: analysis and correction

Cognitive effort is reflected in pupil dilation, but the assessment of pupil size is potentially susceptible to changes in gaze position. This study exemplarily used sentence reading as a stand-in for paradigms that assess pupil size in tasks during which changes in gaze position are unavoidable. The influence of gaze position on pupil size was first investigated by an artificial eye model with a fixed pupil size. Despite its fixed pupil size, the systematic measurements of the artificial eye model revealed substantial gaze-position-dependent changes in the measured pupil size. We evaluated two functions and showed that they can accurately capture and correct the gaze-dependent measurement error of pupil size recorded during a sentence-reading and an effortless z-string-scanning task. Implications for previous studies are discussed, and recommendations for future studies are provided.

Pupil Dilation Reflects the Creation and Retrieval of Memories

It has long been known that pupils-the apertures that allow light into the eyes-dilate and constrict not only in response to changes in ambient light but also in response to emotional changes and arousing stimuli (e.g., Fontana, 1765). Charles Darwin (1872) related changes in pupil diameter to fear and other "emotions" in animals. For decades, pupillometry has been used to study cognitive processing across many domains, including perception, language, visual search, and short-term memory. Historically, such studies have examined the pupillary reflex as a correlate of attentional demands imposed by different tasks or stimuli-pupils typically dilate as cognitive demand increases. Because the neural mechanisms responsible for such task-evoked pupillary reflexes (TEPRs) implicate a role for memory processes, recent studies have examined pupillometry as a tool for investigating the cognitive processes underlying the creation of new episodic memories and their later retrieval. Here, we review the historical antecedents of current pupillometric research and discuss several recent studies linking pupillary dilation to the on-line consumption of cognitive resources in long-term-memory tasks. We conclude by discussing the future role of pupillometry in memory research and several methodological considerations that are important when designing pupillometric studies.

Memory strength and specificity revealed by pupillometry

Voice-specificity effects in recognition memory were investigated using both behavioral data and pupillometry. Volunteers initially heard spoken words and nonwords in two voices; they later provided confidence-based old/new classifications to items presented in their original voices, changed (but familiar) voices, or entirely new voices. Recognition was more accurate for old-voice items, replicating prior research. Pupillometry was used to gauge cognitive demand during both encoding and testing: enlarged pupils revealed that participants devoted greater effort to encoding items that were subsequently recognized. Further, pupil responses were sensitive to the cue match between encoding and retrieval voices, as well as memory strength. Strong memories, and those with the closest encoding-retrieval voice matches, resulted in the highest peak pupil diameters. The results are discussed with respect to episodic memory models and Whittlesea's (1997) SCAPE framework for recognition memory.

Recognition Memory Strength is Predicted by Pupillary Responses at Encoding While Fixation Patterns Distinguish Recollection from Familiarity

Thirty-five healthy participants incidentally encoded a set of man-made and natural object pictures, while their pupil response and eye movements were recorded. At retrieval, studied and new stimuli were rated as novel, familiar (strong, moderate, or weak), or recollected. We found that both pupil response and fixation patterns at encoding predict later recognition memory strength. The extent of pupillary response accompanying incidental encoding was found to be predictive of subsequent memory. In addition, the number of fixations was also predictive of later recognition memory strength, suggesting that the accumulation of greater visual detail, even for single objects, is critical for the creation of a strong memory. Moreover, fixation patterns at encoding distinguished between recollection and familiarity at retrieval, with more dispersed fixations predicting familiarity and more clustered fixations predicting recollection. These data reveal close links between the autonomic control of pupil responses and eye movement patterns on the one hand and memory encoding on the other. Moreover, the data illustrate quantitative as well as qualitative differences in the incidental visual processing of stimuli, which are differentially predictive of the strength and the kind of memory experienced at recognition.

The Pupil Reflects Motor Preparation for Saccades - Even before the Eye Starts to Move

The eye produces saccadic eye movements whose reaction times are perhaps the shortest in humans. Saccade latencies reflect ongoing cortical processing and, generally, shorter latencies are supposed to reflect advanced motor preparation. The dilation of the eye's pupil is reported to reflect cortical processing as well. Eight participants made saccades in a gap and overlap paradigm (in pure and mixed blocks), which we used in order to produce a variety of different saccade latencies. Saccades and pupil size were measured with the EyeLink II. The pattern in pupil dilation resembled that of a gap effect: for gap blocks, pupil dilations were larger compared to overlap blocks; mixing gap and overlap trials reduced the pupil dilation for gap trials thereby inducing a switching cost. Furthermore, saccade latencies across all tasks predicted the magnitude of pupil dilations post hoc: the longer the saccade latency the smaller the pupil dilation before the eye actually began to move. In accordance with observations for manual responses, we conclude that pupil dilations prior to saccade execution reflect advanced motor preparations and therefore provide valid indicator qualities for ongoing cortical processes.

Estimation of mental effort in learning visual search by measuring pupil response

Perceptual learning refers to the improvement of perceptual sensitivity and performance with training. In this study, we examined whether learning is accompanied by a release from mental effort on the task, leading to automatization of the learned task. For this purpose, we had subjects conduct a visual search for a target, defined by a combination of orientation and spatial frequency, while we monitored their pupil size. It is well known that pupil size reflects the strength of mental effort invested in a task. We found that pupil size increased rapidly as the learning proceeded in the early phase of training and decreased at the later phase to a level half of its maximum value. This result does not support the simple automatization hypothesis. Instead, it suggests that the mental effort and behavioral performance reflect different aspects of perceptual learning. Further, mental effort would be continued to be invested to maintain good performance at a later stage of training.

Reaction time distributions constrain models of visual search

Many experiments have investigated visual search for simple stimuli like colored bars or alphanumeric characters. When eye movements are not a limiting factor, these tasks tend to produce roughly linear functions relating reaction time (RT) to the number of items in the display (set size). The slopes of the RTxset size functions for different searches fall on a continuum from highly efficient (slopes near zero) to inefficient (slopes>25-30ms/item). Many theories of search can produce the correct pattern of mean RTs. Producing the correct RT distributions is more difficult. In order to guide future modeling, we have collected a very large data set (about 112,000 trials) on three tasks: an efficient color feature search, an inefficient search for a 2 among 5s, and an intermediate colorxorientation conjunction search. The RT distributions have interesting properties. For example, target absent distributions overlap target present more than would be expected if the decision to end search were based on a simple elapsed time threshold. Other qualitative properties of the RT distributions falsify some classes of model. For example, normalized RT distributions do not change shape as set size changes as a standard self-terminating model predicts that they should.

Neural correlates of fast pupil dilation in nonhuman primates: relation to behavioral performance and cognitive workload

Pupil dilation in humans has been previously shown to correlate with cognitive workload, whereby increased frequency of dilation is associated with increased degree of difficulty of a task. It has been suggested that frontal oculomotor brain areas control cognitively related pupil dilations, but this has not been confirmed due to lack of animal models of cognitive workload and task-related pupil dilation. This is the first report of a wavelet analysis applied to continuous measures of pupil size used to detect the onset of abrupt pupil dilations and the frequency of those dilations in nonhuman primates (NHPs) performing a trial-unique delayed-match-to-sample (DMS) task. A unique finding shows that electrophysiological recordings in the same animals revealed firing of neurons in frontal cortex correlated to different components of pupil dilation during task performance. It is further demonstrated that the frequency of fast pupil dilations (but not rate of eye movements) correlated with cognitive workload during task performance. Such correlations suggest that frontal neuron encoding of pupil dilation provides critical feedback to other brain areas involved in the processing of complex visual information.

Pupillometry as a measure of cognitive effort in younger and older adults

Two experiments examined the effectiveness of the pupillary response as a measure of cognitive load in younger and older adults. Experiment 1 measured the change in pupil size of younger and older adults while they listened to spoken digit lists that varied in length and retained them briefly for recall. In Experiment 2 changes in relative pupil size were measured while younger and older adults listened to sentences for later recall that varied in syntactic complexity and sentence length. Both age groups' pupil sizes were sensitive to the size of the memory set in Experiment 1 and sentence length in Experiment 2, with the older adults showing a larger effect of the memory load on a normalized measure of pupil size relative to the younger adults. By contrast, only the younger adults showed a difference in the pupillary response to a change in syntactic complexity, even with an adjustment for the reduced reactivity of the older pupil.

Deficits in cross-race face learning: insights from eye movements and pupillometry

The own-race bias (ORB) is a well-known finding wherein people are better able to recognize and discriminate own-race faces, relative to cross-race faces. In 2 experiments, participants viewed Asian and Caucasian faces, in preparation for recognition memory tests, while their eye movements and pupil diameters were continuously monitored. In Experiment 1 (with Caucasian participants), systematic differences emerged in both measures as a function of depicted race: While encoding cross-race faces, participants made fewer (and longer) fixations, they preferentially attended to different sets of features, and their pupils were more dilated, all relative to own-race faces. Also, in both measures, a pattern emerged wherein some participants reduced their apparent encoding effort to cross-race faces over trials. In Experiment 2 (with Asian participants), the authors observed the same patterns, although the ORB favored the opposite set of faces. Taken together, the results suggest that the ORB appears during initial perceptual encoding. Relative to own-race face encoding, cross-race encoding requires greater effort, which may reduce vigilance in some participants.

Infant cognition: going full factorial with pupil dilation

The violation-of-expectation (VOE) paradigm and related methods are the main tools used to study high-level cognition in preverbal infants. Infants' differential looking to conceptually implausible/impossible events has been used as an index of early cognitive competence in many areas, including object knowledge, physics, language, and number. However, an event's plausibility is commonly confounded with its perceptual novelty or familiarity, leading to a variety of interpretations for looking time data (Bogartz, Shinskey & Speaker, 1997). This illustrative study demonstrates the value of factorial designs, in which perceptual (novelty-familiarity) and conceptual (possible-impossible) variables are independently and jointly explored. It also introduces pupil dilation as a viable and complementary dependent measure to study infant cognition. We show that pupil data can assist in the interpretation of otherwise equivocal looking time data. The discussion focuses on methodological considerations in infancy research.

Ocular accommodation and cognitive demand: an additional indicator besides pupil size and cardiovascular measures?

BACKGROUND

The aim of the present study was to assess accommodation as a possible indicator of changes in the autonomic balance caused by altered cognitive demand. Accounting for accommodative responses from a human factors perspective may be motivated by the interest of designing virtual image displays or by establishing an autonomic indicator that allows for remote measurement at the human eye. Heart period, pulse transit time, and the pupillary response were considered as reference for possible closed-loop accommodative effects. Cognitive demand was varied by presenting monocularly numbers at a viewing distance of 5 D (20 cm) which had to be read, added or multiplied; further, letters were presented in a "n-back" task.

RESULTS

Cardiovascular parameters and pupil size indicated a change in autonomic balance, while error rates and reaction time confirmed the increased cognitive demand during task processing. An observed decrease in accommodation could not be attributed to the cognitive demand itself for two reasons: (1) the cognitive demand induced a shift in gaze direction which, for methodological reasons, accounted for a substantial part of the observed accommodative changes. (2) Remaining effects disappeared when the correctness of task processing was taken into account.

CONCLUSION

Although the expectation of accommodation as possible autonomic indicator of cognitive demand was not confirmed, the present results are informative for the field of applied psychophysiology noting that it seems not to be worthwhile to include closed-loop accommodation in future studies. From a human factors perspective, expected changes of accommodation due to cognitive demand are of minor importance for design specifications - of, for example, complex visual displays.

Sex differences in a virtual water maze: an eye tracking and pupillometry study

Sex differences in human spatial navigation are well known. However, the exact strategies that males and females employ in order to navigate successfully around the environment are unclear. While some researchers propose that males prefer environment-centred (allocentric) and females prefer self-centred (egocentric) navigation, these findings have proved difficult to replicate. In the present study we examined eye movements and physiological measures of memory (pupillometry) in order to compare visual scanning of spatial orientation using a human virtual analogue of the Morris Water Maze task. Twelve women and twelve men (average age=24 years) were trained on a visible platform and had to locate an invisible platform over a series of trials. On all but the first trial, participants' eye movements were recorded for 3s and they were asked to orient themselves in the environment. While the behavioural data replicated previous findings of improved spatial performance for males relative to females, distinct sex differences in eye movements were found. Males tended to explore consistently more space early on while females demonstrated initially longer fixation durations and increases in pupil diameter usually associated with memory processing. The eye movement data provides novel insight into differences in navigational strategies between the sexes.

Investigating dimensional organization in scripts using the pupillary response

Scripts are mental representations of activities in memory and are thought to be organized dimensionally in a temporal dimension. We investigated the cognitive strategies during the processing of temporal order of an event sequence to gain insight into the organization of scripts. Subjects were presented with triplets of script events (A - B - C). Fifty percent of the items included sequence violations at different positions within the triplet (late: A - C - B, or early: C - A - B). Reaction times indicate that subjects use an economical strategy by comparing the relative temporal positions of event pairs (e.g., A vs. B and if necessary B vs. C) and only attend to information that is necessary. Pupil data and error rates indicate that the temporal information of the complete sequence affects the decision process even if the first event pair indicates that temporal order has been violated. Results are seen as evidence of a dimensional structure of scripts.