Effort during visual search and counting: Insights from pupillometry

Pupil diameter differentiates expertise in dental radiography visual search

Expert behavior is characterized by rapid information processing abilities, dependent on more structured schemata in long-term memory designated for their domain-specific tasks. From this understanding, expertise can effectively reduce cognitive load on a domain-specific task. However, certain tasks could still evoke different gradations of load even for an expert, e.g., when having to detect subtle anomalies in dental radiographs. Our aim was to measure pupil diameter response to anomalies of varying levels of difficulty in expert and student dentists’ visual examination of panoramic radiographs. We found that students’ pupil diameter dilated significantly from baseline compared to experts, but anomaly difficulty had no effect on pupillary response. In contrast, experts’ pupil diameter responded to varying levels of anomaly difficulty, where more difficult anomalies evoked greater pupil dilation from baseline. Experts thus showed proportional pupillary response indicative of increasing cognitive load with increasingly difficult anomalies, whereas students showed pupillary response indicative of higher cognitive load for all anomalies when compared to experts.

Putting effort into infant cognition

Working memory allows for the manipulation of information in support of ongoing tasks, providing a workspace for cognitive processes such as learning, reasoning, and decision making. How well working memory works depends, in part, on effort. Someone who pays attention at the right time and place will have better memory, and performance. In adult cognitive research studies, participants' devotion of maximal task-focused effort is often taken for granted, but in infant studies researchers cannot make that assumption. Here we showcase how pupillometry can provide an easy-to-obtain physiological measure of cognitive effort, allowing us to better understand infants' emerging abilities. In our work, we used pupillometry to measure trial-by-trial fluctuations of effort, establishing that, just as in adults, it influences how well infants could encode information in visual working memory. We hope that by using physiological measures such as pupil dilation, there will be a renewed effort to investigate the interaction between infants' attentive states and cognition.

What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures

Interpretation of volumetric medical images represents a rapidly growing proportion of the workload in radiology. However, relatively little is known about the strategies that best guide search behavior when looking for abnormalities in volumetric images. Although there is extensive literature on two-dimensional medical image perception, it is an open question whether the conclusions drawn from these images can be generalized to volumetric images. Importantly, volumetric images have distinct characteristics (e.g., scrolling through depth, smooth-pursuit eye-movements, motion onset cues, etc.) that should be considered in future research. In this manuscript, we will review the literature on medical image perception and discuss relevant findings from basic science that can be used to generate predictions about expertise in volumetric image interpretation. By better understanding search through volumetric images, we may be able to identify common sources of error, characterize the optimal strategies for searching through depth, or develop new training and assessment techniques for radiology residents.

The Ontogenesis of Action Syntax

Language and action share similar organizational principles. Both are thought to be hierarchical and recursive in nature. Here we address the relationship between language and action from developmental and neurophysiological perspectives. We discuss three major aspects: The extent of the analogy between language and action; the necessity to extend research on the yet largely neglected aspect of action syntax; the positive contribution of a developmental approach to this topic. We elaborate on the claim that adding an ontogenetic approach will help to obtain a comprehensive picture about both the interplay between language and action and its development, and to answer the question whether the underlying mechanisms of detecting syntactic violations of action sequences are similar to or different from the processing of language syntactic violations.

Pupil size and search performance in low and high perceptual load

The ability to focus on a task while disregarding irrelevant information is an example of selective attention. The perceptual-load hypothesis argues that the occurrence of early or late selection mechanisms is determined by task-relevant perceptual load. Additionally, evidence shows that pupil size serves as proxy of locus coeruleus-norepinephrine (LC-NE) activity, a system associated with cognitive and attentional mediation. Here, we assessed pupil baseline (and pupil dilation) as predictors of load-related early and late selection performance. Participants were asked to search for a target in conditions of high and low perceptual load, while ignoring irrelevant stimuli. The results showed that pupil baseline size, measured prior trial onset, significantly predicted the upcoming search efficiency only in low perceptual load, when-according to the perceptual-load hypothesis-all perceptual information receives attentional resources. In addition, pupil dilation was linked to the time course of perceptual processing and predicted response times in both perceptual load conditions, an association that was enhanced in high load. Thus, this study relates attentional selection mechanisms, as defined by the perceptual-load theory, with pupil-related LC-NE activity. Because pupil baseline predicted attentional performance in low load but not in high load, this suggests that different attentional mechanisms are involved, one in which the LC-NE system plays a key role (low load) and one in which it is less relevant (high load). This suggests that the degree with which LC-NE influences behavioral performance is related to the perceptual load of the task at hand.

Focused attention predicts visual working memory performance in 13-month-old infants: A pupillometric study

Attention turns looking, into seeing. Yet, little developmental research has examined the interface of attention and visual working memory (VWM), where what is seen is maintained for use in ongoing visual tasks. Using the task-evoked pupil response - a sensitive, real-time, involuntary measure of focused attention that has been shown to correlate with VWM performance in adults and older children - we examined the relationship between focused attention and VWM in 13-month-olds. We used a Delayed Match Retrieval paradigm, to test infants' VWM for object-location bindings - what went where - while recording anticipatory gaze responses and pupil dilation. We found that infants with greater focused attention during memory encoding showed significantly better memory performance. As well, trials that ended in a correct response had significantly greater pupil response during memory encoding than incorrect trials. Taken together, this shows that pupillometry can be used as a measure of focused attention in infants, and a means to identify those individuals, or moments, where cognitive effort is maximized.

Learning Choreography: An Investigation of Motor Imagery, Attentional Effort, and Expertise in Modern Dance

The study of choreography in dance offers researchers an intriguing window on the relationship between expertise, imagination, and attention in the creative process of learning new movements. The present study investigated an unresolved issue in this field - namely, the effects of expertise on motor imagery (MI; or the mental rehearsal of actions without engaging in the actual movements involved) and attentional effort (as measured by pupil dilation) on dancers while they engaged in the processes of learning, performing, and imagining a dance movement. Participants were 18 female dancers (mean age = 23, SD = 5.85) comprising three experience levels (i.e., novice, intermediate and expert performers) in this field. Data comprised these participants' MI scores as well as their pupil dilation while they learned, performed, and imagined a 15 s piece of choreography. In addition, the time taken both to perform and to imagine the choreography were recorded. Results showed no significant effect of dance expertise on MI but some differences between beginners and intermediate dancers in attentional effort (pupil dilation) at the start of the performance and the imagined movement conditions. Specifically, the beginners had the highest pupil dilation, with the experts having the second highest, while intermediates had the lowest dilation. Further analysis suggested that the novice dancers' pupil dilation at the start of the performance may have been caused, in part, by the initial mental effort required to assess the cognitive demands of the dance task.

Eye vergence responses to novel and familiar stimuli in young children

Eye vergence is the slow movement of both eyes in opposite directions enabling binocular vision. Recently, it was suggested that vergence could be involved in orienting visual attention and memory having a role in cognitive processing of sensory information. In the present study, we assessed whether such vergence responses are observed in early childhood. We measured eye vergence responses in 43 children (12-37 months of age) while looking at novel and repeated object images. Based on previous research, we hypothesized that visual attention and Visual Short-Term Memory (VSMT) would be evidenced by differential vergence responses for both experimental conditions, i.e. repeated (familiar) vs. novel items. The results show that attention related vergence is present in early childhood and that responses to repeated images differ from the ones to novel items. Our current findings suggest that vergence mechanisms could be linking visual attention with short-term memory recognition.

Pupil dilation as an index of effort in cognitive control tasks: A review

Pupillometry research has experienced an enormous revival in the last two decades. Here we briefly review the surge of recent studies on task-evoked pupil dilation in the context of cognitive control tasks with the primary aim being to evaluate the feasibility of using pupil dilation as an index of effort exertion, rather than task demand or difficulty. Our review shows that across the three cognitive control domains of updating, switching, and inhibition, increases in task demands typically leads to increases in pupil dilation. Studies show a diverging pattern with respect to the relationship between pupil dilation and performance and we show how an effort account of pupil dilation can provide an explanation of these findings. We also discuss future directions to further corroborate this account in the context of recent theories on cognitive control and effort and their potential neurobiological substrates.

Quantitative Evaluations of the Effects of Noise on Mental Workloads Based on Pupil Dilation during Laparoscopic Surgery

Noise can exert undeniable pressure on human minds, especially during tasks that require high precision and attention, such as those performed during surgery. To investigate whether auditory stimuli increases mental loads during laparoscopic surgery, we examined the effects of operating room (OR) noises and music by measuring mean changes in pupil sizes and subjectively assessing performances during surgery. We recruited 24 subjects with varying laparoscopic surgery experience levels to perform complete appendectomy using a laparoscopic simulator. Wearable eye trackers were worn by all subjects to monitor pupil sizes during surgery, and surgical tasks were performed under conditions of silence, background OR noise, and music. National Aeronautics and Space Administration-Task Load Index scores and performance parameters were also recorded during surgical tasks. Noise distractions were associated with significant increases in pupil sizes compared with those observed in silence, and the related increases in mental loads may have affected surgical performance. However, more experienced operators had smaller changes in pupil sizes because of auditory disturbances than moderately experienced surgeons. Noise stimulation in the OR increases surgeon's mental workload and performance. Auditory regulation of the OR may be better standardized using data from studies of the effects of acoustic stimulation in the OR, and mental stresses during surgery could be considered in a more humane manner. Further investigations are necessary to determine the cognitive consequences of various auditory stimuli.

Set similarity modulates object tracking in dynamic environments

Based on the observation that sports teams rely on colored jerseys to define group membership, we examined how grouping by similarity affected observers' abilities to track a "ball" target passed between 20 colored circle "players" divided into two color "teams" of 10 players each, or five color teams of four players each. Observers were more accurate and exerted less effort (indexed by pupil diameter) when their task was to count the number of times any player gained possession of the ball versus when they had to count only the possessions by a given color team, especially when counting the possessions of one team when players were grouped into fewer teams of more individual members each. Overall, results confirm previous reports of costs for segregating a larger set into smaller subsets and suggest that grouping by similarity facilitates processing at the set level.

Pupillary responses to a cognitive effort task in schizophrenia

Effort-based decision making paradigms are increasingly utilized to gain insight into the nature of motivation deficits. Although these tasks are being used to assess effort and motivation in schizophrenia, little work has been done to confirm that effort-based decision making tasks validly manipulate effort. In the current study, we adapted the effort component a cognitive effort-based decision making task (the Deck Choice Effort Task) for use with pupillometric assessment. We sought to confirm with psychophysiology that cognitive effort is manipulated. We also examined correlations between physiological indicators of effort exertion and cognition and negative symptoms. The results confirmed manipulation of cognitive effort: there was a significant difference in pupillary responses between easy and difficult task conditions. Pupillary responses were also correlated with cognitive ability, and with negative symptoms when controlling for cognition. Thus, our findings offer physiological validation of an effort manipulation included in a cognitive effort-based decision making task for schizophrenia.

Safe and sensible preprocessing and baseline correction of pupil-size data

Measurement of pupil size (pupillometry) has recently gained renewed interest from psychologists, but there is little agreement on how pupil-size data is best analyzed. Here we focus on one aspect of pupillometric analyses: baseline correction, i.e., analyzing changes in pupil size relative to a baseline period. Baseline correction is useful in experiments that investigate the effect of some experimental manipulation on pupil size. In such experiments, baseline correction improves statistical power by taking into account random fluctuations in pupil size over time. However, we show that baseline correction can also distort data if unrealistically small pupil sizes are recorded during the baseline period, which can easily occur due to eye blinks, data loss, or other distortions. Divisive baseline correction (corrected pupil size = pupil size/baseline) is affected more strongly by such distortions than subtractive baseline correction (corrected pupil size = pupil size - baseline). We discuss the role of baseline correction as a part of preprocessing of pupillometric data, and make five recommendations: (1) before baseline correction, perform data preprocessing to mark missing and invalid data, but assume that some distortions will remain in the data; (2) use subtractive baseline correction; (3) visually compare your corrected and uncorrected data; (4) be wary of pupil-size effects that emerge faster than the latency of the pupillary response allows (within ±220 ms after the manipulation that induces the effect); and (5) remove trials on which baseline pupil size is unrealistically small (indicative of blinks and other distortions).

Pupillary response to complex interdependent tasks: A cognitive-load theory perspective

Pupil dilation is known to indicate cognitive load. In this study, we looked at the average pupillary responses of a cohort of 29 undergraduate students during graphical problem solving. Three questions were asked, based on the same graphical input. The questions were interdependent and comprised multiple steps. We propose a novel way of analyzing pupillometry data for such tasks on the basis of eye fixations, a commonly used eyetracking parameter. We found that pupil diameter increased during the solution process. However, pupil diameter did not always reflect the expected cognitive load. This result was studied within a cognitive-load theory model. Higher-performing students showed evidence of germane load and schema creation, indicating use of the interdependent nature of the tasks to inform their problem-solving process. However, lower-performing students did not recognize the interdependent nature of the tasks and solved each problem independently, which was expressed in a markedly different pupillary response pattern. We discuss the import of our findings for instructional design.

Association between pupil dilation and implicit processing prior to object recognition via insight

Insight refers to the sudden conscious shift in the perception of a situation following a period of unconscious processing. The present study aimed to investigate the implicit neural mechanisms underlying insight-based recognition, and to determine the association between these mechanisms and the extent of pupil dilation. Participants were presented with ambiguous, transforming images comprised of dots, following which they were asked to state whether they recognized the object and their level of confidence in this statement. Changes in pupil dilation were not only characterized by the recognition state into the ambiguous object but were also associated with prior awareness of object recognition, regardless of meta-cognitive confidence. Our findings indicate that pupil dilation may represent the level of implicit integration between memory and visual processing, despite the lack of object awareness, and that this association may involve noradrenergic activity within the locus coeruleus-noradrenergic (LC-NA) system.

Vergence responses to face stimuli in young children

In line with a possible role of vergence in orienting visual attention, we sought to study this eye movement in young children to provide insights on face perception. For this purpose, we measured the modulation in the angle of eye vergence in a sample of 39 children aged 6-36 months and compared it when presenting static human face images versus their scrambled version. We observed enhanced vergence responses to faces compared with scrambled images. Our data suggest a role of eye vergence in face processing.

Foggy windows: Pupillary responses during task preparation

We investigated pupil dilation in 96 subjects during task preparation and during a post-trial interval in a visual search task and an auditory working memory task. Completely informative difficulty cues (easy, medium, or hard) were presented right before task preparation to examine whether pupil dilation indicated advance mobilisation of attentional resources; functional magnetic resonance imaging (fMRI) studies have argued for the existence of such task preparation, and the literature shows that pupil dilation tracks attentional effort during task performance. We found, however, little evidence for such task preparation. In the working memory task, pupil size was identical across cues, and although pupil dilation in the visual search task tracked the cue, pupil dilation predicted subsequent performance in neither task. Pupil dilation patterns in the post-trial interval were more consistent with an effect of emotional reactivity. Our findings suggest that the mobilisation of attentional resources in the service of the task does not occur during the preparatory interval, but is delayed until the task itself is initiated.

Pre-stimulus pupil dilation and the preparatory control of attention

Task preparation involves multiple component processes, including a general evaluative process that signals the need for adjustments in control, and the engagement of task-specific control settings. Here we examined the dynamics of these different mechanisms in preparing the attentional control system for visual search. We explored preparatory activity using pupil dilation, a well-established measure of task demands and effortful processing. In an initial exploratory experiment, participants were cued at the start of each trial to search for either a salient color singleton target (an easy search task) or a low-salience shape singleton target (a difficult search task). Pupil dilation was measured during the preparation period from cue onset to search display onset. Mean dilation was larger in preparation for the difficult shape target than the easy color target. In two additional experiments, we sought to vary effects of evaluative processing and task-specific preparation separately. Experiment 2 showed that when the color and shape search tasks were matched for difficulty, the shape target no longer evoked larger dilations, and the pattern of results was in fact reversed. In Experiment 3, we manipulated difficulty within a single feature dimension, and found that the difficult search task evoked larger dilations. These results suggest that pupil dilation reflects expectations of difficulty in preparation for a search task, consistent with the activity of an evaluative mechanism. We did not find consistent evidence for relationship between pupil dilation and search performance (accuracy and response timing), suggesting that pupil dilation during search preparation may not be strongly linked to ongoing task-specific preparation.

Applying fractal analysis to pupil dilation for measuring complexity in a process monitoring task

This laboratory experiment was designed to use fractal dimension as a new method to analyze pupil dilation to evaluate the level of complexity in a multitasking environment. By using the eye-head integrated tracking system, we collected both pupil responses and head positions while participants conducted both process monitoring task and Multi-Attribute Task Battery (MATB-II) tasks. There was a significant effect of scenario complexity on a composite index of multitasking performance (Low Complexity » High Complexity). The fractal dimension of pupil dilation was also significantly influenced by complexity. The results clearly showed that the correlation between pupil dilation and multitasking performance was stronger when the pupil data was analyzed by using the fractal dimension method. The participants showed a higher fractal dimension when they performed a low complexity multitasking scenario. The findings of this research help us to advance our understanding of how to evaluate the complexity level of real-world applications by using pupillary responses.

The World (of Warcraft) through the eyes of an expert

Negative correlations between pupil size and the tendency to look at salient locations were found in recent studies (e.g., Mathôt et al., 2015). It is hypothesized that this negative correlation might be explained by the mental effort put by participants in the task that leads in return to pupil dilation. Here we present an exploratory study on the effect of expertise on eye-movement behavior. Because there is no available standard tool to evaluate WoW players' expertise, we built an off-game questionnaire testing players' knowledge about WoW and acquired skills through completed raids, highest rated battlegrounds, Skill Points, etc. Experts (N = 4) and novices (N = 4) in the massively multiplayer online role-playing game World of Warcraft (WoW) viewed 24 designed video segments from the game that differ in regards with their content (i.e, informative locations) and visual complexity (i.e, salient locations). Consistent with previous studies, we found a negative correlation between pupil size and the tendency to look at salient locations (experts, r =  - .17, p < .0001, and novices, r =  - .09, p < .0001). This correlation has been interpreted in terms of mental effort: People are inherently biased to look at salient locations (sharp corners, bright lights, etc.), but are able (i.e., experts) to overcome this bias if they invest sufficient mental effort. Crucially, we observed that this correlation was stronger for expert WoW players than novice players (Z =  - 3.3, p = .0011). This suggests that experts learned to improve control over eye-movement behavior by guiding their eyes towards informative, but potentially low-salient areas of the screen. These findings may contribute to our understanding of what makes an expert an expert.

Working memory load predicts visual search efficiency: Evidence from a novel pupillary response paradigm

An observer's pupil dilates and constricts in response to variables such as ambient and focal luminance, cognitive effort, the emotional stimulus content, and working memory load. The pupil's memory load response is of particular interest, as it might be used for estimating observers' memory load while they are performing a complex task, without adding an interruptive and confounding memory test to the protocol. One important task in which working memory's involvement is still being debated is visual search, and indeed a previous experiment by Porter, Troscianko, and Gilchrist (Quarterly Journal of Experimental Psychology, 60, 211-229, 2007) analyzed observers' pupil sizes during search to study this issue. These authors found that pupil size increased over the course of the search, and they attributed this finding to accumulating working memory load. However, since the pupil response is slow and does not depend on memory load alone, this conclusion is rather speculative. In the present study, we estimated working memory load in visual search during the presentation of intermittent fixation screens, thought to induce a low, stable level of arousal and cognitive effort. Using standard visual search and control tasks, we showed that this paradigm reduces the influence of non-memory-related factors on pupil size. Furthermore, we found an early increase in working memory load to be associated with more efficient search, indicating a significant role of working memory in the search process.

Eye Behavior Associated with Internally versus Externally Directed Cognition

What do our eyes do when we are focused on internal representations such as during imagination or planning? Evidence from mind wandering research suggests that spontaneous shifts from externally directed cognition (EDC) to internally directed cognition (IDC) involves oculomotor changes indicative of visual disengagement. In the present study, we investigated potential differences in eye behavior between goal-directed forms of IDC and EDC. To this end, we manipulated the focus of attention (internal versus external) in two demanding cognitive tasks (anagram and sentence generation). IDC was associated with fewer and longer fixations and higher variability in pupil diameter and eye vergence compared to EDC, suggesting reduced visual scanning and higher spontaneous eye activity. IDC was further related to longer blinks, lower microsaccade frequency, and a lower angle of eye vergence. These latter changes appear conducive to attenuate visual input and thereby shield ongoing internal processes from external distraction. Together, these findings suggest that IDC is accompanied by characteristic eye behavior that reflects a decoupling of attention from external events and serves gating out visual input.

EEG and Eye Tracking Signatures of Target Encoding during Structured Visual Search

EEG and eye tracking variables are potential sources of information about the underlying processes of target detection and storage during visual search. Fixation duration, pupil size and event related potentials (ERPs) locked to the onset of fixation or saccade (saccade-related potentials, SRPs) have been reported to differ dependent on whether a target or a non-target is currently fixated. Here we focus on the question of whether these variables also differ between targets that are subsequently reported (hits) and targets that are not (misses). Observers were asked to scan 15 locations that were consecutively highlighted for 1 s in pseudo-random order. Highlighted locations displayed either a target or a non-target stimulus with two, three or four targets per trial. After scanning, participants indicated which locations had displayed a target. To induce memory encoding failures, participants concurrently performed an aurally presented math task (high load condition). In a low load condition, participants ignored the math task. As expected, more targets were missed in the high compared with the low load condition. For both conditions, eye tracking features distinguished better between hits and misses than between targets and non-targets (with larger pupil size and shorter fixations for missed compared with correctly encoded targets). In contrast, SRP features distinguished better between targets and non-targets than between hits and misses (with average SRPs showing larger P300 waveforms for targets than for non-targets). Single trial classification results were consistent with these averages. This work suggests complementary contributions of eye and EEG measures in potential applications to support search and detect tasks. SRPs may be useful to monitor what objects are relevant to an observer, and eye variables may indicate whether the observer should be reminded of them later.

Pupil Sizes Scale with Attentional Load and Task Experience in a Multiple Object Tracking Task

Previous studies have related changes in attentional load to pupil size modulations. However, studies relating changes in attentional load and task experience on a finer scale to pupil size modulations are scarce. Here, we investigated how these changes affect pupil sizes. To manipulate attentional load, participants covertly tracked between zero and five objects among several randomly moving objects on a computer screen. To investigate effects of task experience, the experiment was conducted on three consecutive days. We found that pupil sizes increased with each increment in attentional load. Across days, we found systematic pupil size reductions. We compared the model fit for predicting pupil size modulations using attentional load, task experience, and task performance as predictors. We found that a model which included attentional load and task experience as predictors had the best model fit while adding performance as a predictor to this model reduced the overall model fit. Overall, results suggest that pupillometry provides a viable metric for precisely assessing attentional load and task experience in visuospatial tasks.

The Mechanisms Underlying the ASD Advantage in Visual Search

A number of studies have demonstrated that individuals with autism spectrum disorders (ASDs) are faster or more successful than typically developing control participants at various visual-attentional tasks (for reviews, see Dakin and Frith in Neuron 48:497-507, 2005; Simmons et al. in Vis Res 49:2705-2739, 2009). This "ASD advantage" was first identified in the domain of visual search by Plaisted et al. (J Child Psychol Psychiatry 39:777-783, 1998). Here we survey the findings of visual search studies from the past 15 years that contrasted the performance of individuals with and without ASD. Although there are some minor caveats, the overall consensus is that-across development and a broad range of symptom severity-individuals with ASD reliably outperform controls on visual search. The etiology of the ASD advantage has not been formally specified, but has been commonly attributed to 'enhanced perceptual discrimination', a superior ability to visually discriminate between targets and distractors in such tasks (e.g. O'Riordan in Cognition 77:81-96, 2000). As well, there is considerable evidence for impairments of the attentional network in ASD (for a review, see Keehn et al. in J Child Psychol Psychiatry 37:164-183, 2013). We discuss some recent results from our laboratory that support an attentional, rather than perceptual explanation for the ASD advantage in visual search. We speculate that this new conceptualization may offer a better understanding of some of the behavioral symptoms associated with ASD, such as over-focusing and restricted interests.

Children view own-age faces qualitatively differently to other-age faces

Like most own-group biases in face recognition, the own-age bias (OAB) is thought to be based either on perceptual expertise or socio-cognitive motivational mechanisms [Wolff, N., Kemter, K., Schweinberger, S. R., & Wiese, H. (2013). What drives social in-group biases in face recognition memory? ERP evidence from the own-gender bias. Social Cognitive and Affective Neuroscience. doi:10.1093/scan/nst024]. The present study employed a recognition paradigm with eye-tracking in order to assess whether participants actively viewed faces of their own-age differently to that of other-age faces. The results indicated a significant OAB (superior recognition for own-age relative to other-age faces), provided that they were upright, indicative of expertise being employed for the recognition of own-age faces. However, the eye-tracking results indicate that viewing other-age faces was qualitatively different to the viewing of own-age faces, with more nose fixations for other-age faces. These results are interpreted as supporting the socio-cognitive model of the OAB.

Cross-Modal and Intra-Modal Characteristics of Visual Function and Speech Perception Performance in Postlingually Deafened, Cochlear Implant Users

Evidence of visual-auditory cross-modal plasticity in deaf individuals has been widely reported. Superior visual abilities of deaf individuals have been shown to result in enhanced reactivity to visual events and/or enhanced peripheral spatial attention. The goal of this study was to investigate the association between visual-auditory cross-modal plasticity and speech perception in post-lingually deafened, adult cochlear implant (CI) users. Post-lingually deafened adults with CIs (N = 14) and a group of normal hearing, adult controls (N = 12) participated in this study. The CI participants were divided into a good performer group (good CI, N = 7) and a poor performer group (poor CI, N = 7) based on word recognition scores. Visual evoked potentials (VEP) were recorded from the temporal and occipital cortex to assess reactivity. Visual field (VF) testing was used to assess spatial attention and Goldmann perimetry measures were analyzed to identify differences across groups in the VF. The association of the amplitude of the P1 VEP response over the right temporal or occipital cortex among three groups (control, good CI, poor CI) was analyzed. In addition, the association between VF by different stimuli and word perception score was evaluated. The P1 VEP amplitude recorded from the right temporal cortex was larger in the group of poorly performing CI users than the group of good performers. The P1 amplitude recorded from electrodes near the occipital cortex was smaller for the poor performing group. P1 VEP amplitude in right temporal lobe was negatively correlated with speech perception outcomes for the CI participants (r = -0.736, P = 0.003). However, P1 VEP amplitude measures recorded from near the occipital cortex had a positive correlation with speech perception outcome in the CI participants (r = 0.775, P = 0.001). In VF analysis, CI users showed narrowed central VF (VF to low intensity stimuli). However, their far peripheral VF (VF to high intensity stimuli) was not different from the controls. In addition, the extent of their central VF was positively correlated with speech perception outcome (r = 0.669, P = 0.009). Persistent visual activation in right temporal cortex even after CI causes negative effect on outcome in post-lingual deaf adults. We interpret these results to suggest that insufficient intra-modal (visual) compensation by the occipital cortex may cause negative effects on outcome. Based on our results, it appears that a narrowed central VF could help identify CI users with poor outcomes with their device.

Greater Pupil Size in Response to Emotional Faces as an Early Marker of Social-Communicative Difficulties in Infants at High Risk for Autism

When scanning faces, individuals with autism spectrum disorder (ASD) have shown reduced visual attention (e.g., less time on eyes) and atypical autonomic responses (e.g., heightened arousal). To understand how these differences might explain sub-clinical variability in social functioning, 9-month-olds, with or without a family history of ASD, viewed emotionally-expressive faces, and gaze and pupil diameter (a measure of autonomic activation) were recorded using eye-tracking. Infants at high-risk for ASD with no subsequent clinical diagnosis (HRA-) and low-risk controls (LRC) showed similar face scanning and attention to eyes and mouth. Attention was overall greater to eyes than mouth, but this varied as a function of the emotion presented. HRA- showed significantly larger pupil size than LRC. Correlations between scanning at 9 months, pupil size at 9 months, and 18-month social-communicative behavior, revealed positive associations between pupil size and attention to both face and eyes at 9 months in LRC, and a negative association between 9-month pupil size and 18-month social-communicative behavior in HRA-. The present findings point to heightened autonomic arousal in HRA-. Further, with greater arousal relating to worse social-communicative functioning at 18 months, this work points to a mechanism by which unaffected siblings might develop atypical social behavior.

Detection of Changes in Surgical Difficulty

Background. Assessing the workload of surgeons requires technology to continuously monitor surgeons’ behaviors without interfering with their performance. We investigated the feasibility of using eye-tracking to reveal surgeons’ response to increasing task difficulty. Methods. A controlled study was conducted in a simulated operating room, where 14 subjects were required to perform a laparoscopic procedure that includes 9 subtasks. The subtasks could be divided into 3 types with different levels of task difficulty, calculated by the index of task difficulty (ID) proposed by Fitts in 1954. Pupillary responses of subjects in performing the procedure were recorded using Tobii eye-tracking equipment. Peak pupil dilation and movement time were compared between subtasks with different IDs as well as between fast moving and slow aiming phases within each subtask. Results. When the task difficulty was increased, task completion time increased. Meanwhile, the subjects’ peak pupil size also increased. As the entire procedure was performed continuously, we found that pupil responses were not only affected by the ID in the current subtask but also influenced by subtasks before and after. Discussion. Decomposing a surgical procedure into meaningful subtasks and examining the surgeon’s pupil response to each subtask enables us to identify the challenging steps within a continuous surgical procedure. Psychomotor evidence on surgeon’s performance may lead to an innovation for designing a task-specific training curriculum.

Validation of the NIMH-ChEFS adolescent face stimulus set in an adolescent, parent, and health professional sample

Attention to faces is a fundamental psychological process in humans, with atypical attention to faces noted across several clinical disorders. Although many clinical disorders onset in adolescence, there is a lack of well-validated stimulus sets containing adolescent faces available for experimental use. Further, the images comprising most available sets are not controlled for high- and low-level visual properties. Here, we present a cross-site validation of the National Institute of Mental Health Child Emotional Faces Picture Set (NIMH-ChEFS), comprised of 257 photographs of adolescent faces displaying angry, fearful, happy, sad, and neutral expressions. All of the direct facial images from the NIMH-ChEFS set were adjusted in terms of location of facial features and standardized for luminance, size, and smoothness. Although overall agreement between raters in this study and the original development-site raters was high (89.52%), this differed by group such that agreement was lower for adolescents relative to mental health professionals in the current study. These results suggest that future research using this face set or others of adolescent/child faces should base comparisons on similarly-aged validation data. Copyright © 2015 John Wiley & Sons, Ltd.

Cognitive effort and pupil dilation in controlled and automatic processes

The Five Digits Test (FDT) is a Stroop paradigm test that aims to evaluate executive functions. It is composed of four parts, two of which are related to automatic and two of which are related to controlled processes. It is known that pupillary diameter increases as the task's cognitive demand increases. In the present study, we evaluated whether the pupillary diameter could distinguish cognitive effort between automated and controlled cognitive processing during the FDT as the task progressed. As a control task, we used a simple reading paradigm with a similar visual aspect as the FDT. We then divided each of the four parts into two blocks in order to evaluate the differences between the first and second half of the task. Results indicated that, compared to a control task, the FDT required higher cognitive effort for each consecutive part. Moreover, the first half of every part of the FDT induced dilation more than the second. The differences in pupil dilation during the first half of the four FDT parts were statistically significant between the parts 2 and 4 (p=0.023), and between the parts 3 and 4 (p=0.006). These results provide further evidence that cognitive effort and pupil diameter can distinguish controlled from automatic processes.

Can you see what I am talking about? Human speech triggers referential expectation in four-month-old infants

Infants' sensitivity to selectively attend to human speech and to process it in a unique way has been widely reported in the past. However, in order to successfully acquire language, one should also understand that speech is a referential, and that words can stand for other entities in the world. While there has been some evidence showing that young infants can make inferences about the communicative intentions of a speaker, whether they would also appreciate the direct relationship between a specific word and its referent, is still unknown. In the present study we tested four-month-old infants to see whether they would expect to find a referent when they hear human speech. Our results showed that compared to other auditory stimuli or to silence, when infants were listening to speech they were more prepared to find some visual referents of the words, as signalled by their faster orienting towards the visual objects. Hence, our study is the first to report evidence that infants at a very young age already understand the referential relationship between auditory words and physical objects, thus show a precursor in appreciating the symbolic nature of language, even if they do not understand yet the meanings of words.

Pupillometry reveals increased pupil size during indirect request comprehension

Fluctuations in pupil size have been shown to reflect variations in processing demands during lexical and syntactic processing in language comprehension. An issue that has not received attention is whether pupil size also varies due to pragmatic manipulations. In two pupillometry experiments, we investigated whether pupil diameter was sensitive to increased processing demands as a result of comprehending an indirect request versus a direct statement. Adult participants were presented with 120 picture-sentence combinations that could be interpreted either as an indirect request (a picture of a window with the sentence "it's very hot here") or as a statement (a picture of a window with the sentence "it's very nice here"). Based on the hypothesis that understanding indirect utterances requires additional inferences to be made on the part of the listener, we predicted a larger pupil diameter for indirect requests than statements. The results of both experiments are consistent with this expectation. We suggest that the increase in pupil size reflects additional processing demands for the comprehension of indirect requests as compared to statements. This research demonstrates the usefulness of pupillometry as a tool for experimental research in pragmatics.

The pupillary response discriminates between subjective and objective familiarity and novelty

The pupil response discriminates between old and new stimuli, with old stimuli characterized by larger pupil dilation patterns than new stimuli. We sought to explore the cause of the pupil old/new effect and discount the effect of targetness, effort, recollection retrieval, and complexity of the recognition decision. Two experiments are reported in which the pupil response and the eye fixation patterns were measured, while participants identified novel and familiar object stimuli, in two separate tasks, emphasizing either novelty or familiarity detection. In Experiment 1, familiarity and novelty decisions were taken using a rating scale, while in Experiment 2 a simpler yes/no decision was used. In both experiments, we found that detection of target familiar stimuli resulted in greater pupil dilation than the detection of target novel stimuli, while the duration of the first fixation discriminated between familiar and novel stimuli as early as within 320 ms after stimulus onset. Importantly, the pupil response distinguished between the objective (during an earlier temporal component) and the subjective (during a later temporal component) status of the stimulus for misses and false alarms. In the light of previous findings, we suggest that the pupil and fixation old/new effects reflect the distinct neural and cognitive mechanisms involved in the familiarity and novelty decisions. The findings also have important implications for the use of pupil dilation and eye movement patterns to explore explicit and implicit memory processes.

How do infants recognize joint attention?

The emergence of joint attention is still a matter of vigorous debate. It involves diverse hypotheses ranging from innate modules dedicated to intention reading to more neuro-constructivist approaches. The aim of this study was to assess whether 12-month-old infants are able to recognize a "joint attention" situation when observing such a social interaction. Using a violation-of-expectation paradigm, we habituated infants to a "joint attention" video and then compared their looking time durations between "divergent attention" videos and "joint attention" ones using a 2 (familiar or novel perceptual component)×2 (familiar or novel conceptual component) factorial design. These results were enriched with measures of pupil dilation, which are considered to be reliable measures of cognitive load. Infants looked longer at test events that involved novel speaker and divergent attention but no changes in infants' pupil dilation were observed in any conditions. Although looking time data suggest that infants may appreciate discrepancies from expectations related to joint attention behavior, in the absence of clear evidence from pupillometry, the results show no demonstration of understanding of joint attention, even at a tacit level. Our results suggest that infants may be sensitive to relevant perceptual variables in joint attention situations, which would help scaffold social cognitive development. This study supports a gradual, learning interpretation of how infants come to recognize, understand, and participate in joint attention.

Costs of storing colour and complex shape in visual working memory: Insights from pupil size and slow waves

We investigated the impact of perceptual processing demands on visual working memory of coloured complex random polygons during change detection. Processing load was assessed by pupil size (Exp. 1) and additionally slow wave potentials (Exp. 2). Task difficulty was manipulated by presenting different set sizes (1, 2, 4 items) and by making different features (colour, shape, or both) task-relevant. Memory performance in the colour condition was better than in the shape and both condition which did not differ. Pupil dilation and the posterior N1 increased with set size independent of type of feature. In contrast, slow waves and a posterior P2 component showed set size effects but only if shape was task-relevant. In the colour condition slow waves did not vary with set size. We suggest that pupil size and N1 indicates different states of attentional effort corresponding to the number of presented items. In contrast, slow waves reflect processes related to encoding and maintenance strategies. The observation that their potentials vary with the type of feature (simple colour versus complex shape) indicates that perceptual complexity already influences encoding and storage and not only comparison of targets with memory entries at the moment of testing.

Parents' empathic perspective taking and altruistic behavior predicts infants' arousal to others' emotions

Empathy emerges in children’s overt behavior around the middle of the second year of life. Younger infants, however, exhibit arousal in response to others’ emotional displays, which is considered to be a precursor to fully developed empathy. The goal of the present study was to investigate individual variability in infants’ arousal toward others’ emotional displays, as indexed by 12- and 15-month-old infants’ (n = 49) pupillary changes in response to another infant’s emotions, and to determine whether such variability is linked to parental empathy and prosociality, as indexed via self-report questionnaires. We found that increases in infants’ pupil dilation in response to others’ emotional displays were associated with aspects of parental empathy and prosociality. Specifically, infants who exhibited the greatest arousal in response to others’ emotions had parents who scored highly on empathic perspective taking and self-reported altruism. These relations may have been found because arousal toward others’ emotions shares certain characteristics with empathic and prosocial dispositions. Together, these results demonstrate the presence of early variability in a precursor to mature empathic responding in infancy, which is meaningfully linked to parents’ empathic dispositions and prosocial behaviors.

Task constraints distinguish perspective inferences from perspective use during discourse interpretation in a false belief task

Interpreting other peoples' actions relies on an understanding of their current mental states (e.g. beliefs, desires and intentions). In this paper, we distinguish between listeners' ability to infer others' perspectives and their explicit use of this knowledge to predict subsequent actions. In a visual-world study, two groups of participants (passive observers vs. active participants) watched short videos, depicting transfer events, where one character ('Jane') either held a true or false belief about an object's location. We tracked participants' eye-movements around the final visual scene, time-locked to related auditory descriptions (e.g. "Jane will look for the chocolates in the container on the left".). Results showed that active participants had already inferred the character's belief in the 1s preview period prior to auditory onset, before it was possible to use this information to predict an outcome. Moreover, they used this inference to correctly anticipate reference to the object's initial location on false belief trials at the earliest possible point (i.e. from "Jane" onwards). In contrast, passive observers only showed evidence of a belief inference from the onset of "Jane", and did not show reliable use of this inference to predict Jane's behaviour on false belief trials until much later, when the location ("left/right") was auditorily available. These results show that active engagement in a task activates earlier inferences about others' perspectives, and drives immediate use of this information to anticipate others' actions, compared to passive observers, who are susceptible to influences from egocentric or reality biases. Finally, we review evidence that using other peoples' perspectives to predict their behaviour is more cognitively effortful than simply using one's own.

Fixation duration surpasses pupil size as a measure of memory load in free viewing

Oculomotor behavior reveals, not only the acquisition of visual information at fixation, but also the accumulation of information in memory across subsequent fixations. Two candidate measures were considered as indicators of such dynamic visual memory load: fixation duration and pupil size. While recording these measures, we displayed an arrangement of 3, 4 or 5 targets among distractors. Both occurred in various orientations. Participants searched for targets and reported whether in a subsequent display one of them had changed orientation. We determined to what extent fixation duration and pupil size indicate dynamic memory load, as a function of the number of targets fixated during the search. We found that fixation duration reflects the number of targets, both when this number is within and above the limit of working memory capacity. Pupil size reflects the number of targets only when it exceeds the capacity limit. Moreover, the duration of fixations on successive targets but not on distractors increases whereas pupil size does not. The increase in fixation duration with number of targets both within and above working memory capacity suggests that in free viewing fixation duration is sensitive to actual memory load as well as to processing load, whereas pupil size is indicative of processing load only. Two alternative models relating visual attention and working memory are considered relevant to these results. We discuss the results as supportive of a model which involves a temporary buffer in the interaction of attention and working memory.

Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload

While studies exist that compare different physiological variables with respect to their association with mental workload, it is still largely unclear which variables supply the best information about momentary workload of an individual and what is the benefit of combining them. We investigated workload using the n-back task, controlling for body movements and visual input. We recorded EEG, skin conductance, respiration, ECG, pupil size and eye blinks of 14 subjects. Various variables were extracted from these recordings and used as features in individually tuned classification models. Online classification was simulated by using the first part of the data as training set and the last part of the data for testing the models. The results indicate that EEG performs best, followed by eye related measures and peripheral physiology. Combining variables from different sensors did not significantly improve workload assessment over the best performing sensor alone. Best classification accuracy, a little over 90%, was reached for distinguishing between high and low workload on the basis of 2 min segments of EEG and eye related variables. A similar and not significantly different performance of 86% was reached using only EEG from single electrode location Pz.