Cognitive workload estimation using physiological measures: a review

Estimating cognitive workload levels is an emerging research topic in the cognitive neuroscience domain, as participants' performance is highly influenced by cognitive overload or underload results. Different physiological measures such as Electroencephalography (EEG), Functional Magnetic Resonance Imaging, Functional near-infrared spectroscopy, respiratory activity, and eye activity are efficiently used to estimate workload levels with the help of machine learning or deep learning techniques. Some reviews focus only on EEG-based workload estimation using machine learning classifiers or multimodal fusion of different physiological measures for workload estimation. However, a detailed analysis of all physiological measures for estimating cognitive workload levels still needs to be discovered. Thus, this survey highlights the in-depth analysis of all the physiological measures for assessing cognitive workload. This survey emphasizes the basics of cognitive workload, open-access datasets, the experimental paradigm of cognitive tasks, and different measures for estimating workload levels. Lastly, we emphasize the significant findings from this review and identify the open challenges. In addition, we also specify future scopes for researchers to overcome those challenges.

Combined Effects of Moderate Hypoxia and Sleep Restriction on Mental Workload

Aircraft pilots face a high mental workload (MW) under environmental constraints induced by high altitude and sometimes sleep restriction (SR). Our aim was to assess the combined effects of hypoxia and sleep restriction on cognitive and physiological responses to different MW levels using the Multi-Attribute Test Battery (MATB)-II with an additional auditory Oddball-like task. Seventeen healthy subjects were subjected in random order to three 12-min periods of increased MW level (low, medium, and high): sleep restriction (SR, <3 h of total sleep time (TST)) vs. habitual sleep (HS, >6 h TST), hypoxia (HY, 2 h, FIO2 = 13.6%, ~3500 m vs. normoxia, NO, FIO2 = 21%). Following each MW level, participants completed the NASA-TLX subjective MW scale. Increasing MW decreases performance on the MATB-II Tracking task (p = 0.001, MW difficulty main effect) and increases NASA-TLX (p = 0.001). In the combined HY/SR condition, MATB-II performance was lower, and the NASA-TLX score was higher compared with the NO/HS condition, while no effect of hypoxia alone was observed. In the accuracy of the auditory task, there is a significant interaction between hypoxia and MW difficulty (F(2-176) = 3.14, p = 0.04), with lower values at high MW under hypoxic conditions. Breathing rate, pupil size, and amplitude of pupil dilation response (PDR) to auditory stimuli are associated with increased MW. These parameters are the best predictors of increased MW, independently of physiological constraints. Adding ECG, SpO2, or electrodermal conductance does not improve model performance. In conclusion, hypoxia and sleep restriction have an additive effect on MW. Physiological and electrophysiological responses must be taken into account when designing a MW predictive model and cross-validation.

Glossiness perception and its pupillary response

Recent studies have revealed that pupillary response changes depend on perceptual factors such as subjective brightness caused by optical illusions and luminance. However, the manner in which the perceptual factor that is derived from the glossiness perception of object surfaces affects the pupillary response remains unclear. We investigated the relationship between the glossiness perception and pupillary response through a glossiness rating experiment that included recording the pupil diameter. We prepared general object images (original) and randomized images (shuffled) that comprised the same images with randomized small square regions as stimuli. The image features were controlled by matching the luminance histogram. The observers were asked to rate the perceived glossiness of the stimuli presented for 3,000 ms and the changes in their pupil diameters were recorded. Images with higher glossiness ratings constricted the pupil size more than those with lower glossiness ratings at the peak constriction of the pupillary responses during the stimulus duration. The linear mixed-effects model demonstrated that the glossiness rating, image category (original/shuffled), variance of the luminance histogram, and stimulus area were most effective in predicting the pupillary responses. These results suggest that the illusory brightness obtained by the image regions of high-glossiness objects, such as specular highlights, induce pupil constriction.

From pre-processing to advanced dynamic modeling of pupil data

The pupil of the eye provides a rich source of information for cognitive scientists, as it can index a variety of bodily states (e.g., arousal, fatigue) and cognitive processes (e.g., attention, decision-making). As pupillometry becomes a more accessible and popular methodology, researchers have proposed a variety of techniques for analyzing pupil data. Here, we focus on time series-based, signal-to-signal approaches that enable one to relate dynamic changes in pupil size over time with dynamic changes in a stimulus time series, continuous behavioral outcome measures, or other participants' pupil traces. We first introduce pupillometry, its neural underpinnings, and the relation between pupil measurements and other oculomotor behaviors (e.g., blinks, saccades), to stress the importance of understanding what is being measured and what can be inferred from changes in pupillary activity. Next, we discuss possible pre-processing steps, and the contexts in which they may be necessary. Finally, we turn to signal-to-signal analytic techniques, including regression-based approaches, dynamic time-warping, phase clustering, detrended fluctuation analysis, and recurrence quantification analysis. Assumptions of these techniques, and examples of the scientific questions each can address, are outlined, with references to key papers and software packages. Additionally, we provide a detailed code tutorial that steps through the key examples and figures in this paper. Ultimately, we contend that the insights gained from pupillometry are constrained by the analysis techniques used, and that signal-to-signal approaches offer a means to generate novel scientific insights by taking into account understudied spectro-temporal relationships between the pupil signal and other signals of interest.

Sensory manipulation as a countermeasure to robot teleoperation delays: system and evidence

In the realm of robotics and automation, robot teleoperation, which facilitates human-machine interaction in distant or hazardous settings, has surged in significance. A persistent issue in this domain is the delays between command issuance and action execution, causing negative repercussions on operator situational awareness, performance, and cognitive load. These delays, particularly in long-distance operations, are difficult to mitigate even with the most advanced computing advancements. Current solutions mainly revolve around machine-based adjustments to combat these delays. However, a notable lacuna remains in harnessing human perceptions for an enhanced subjective teleoperation experience. This paper introduces a novel approach of sensory manipulation for induced human adaptation in delayed teleoperation. Drawing from motor learning and rehabilitation principles, it is posited that strategic sensory manipulation, via altered sensory stimuli, can mitigate the subjective feeling of these delays. The focus is not on introducing new skills or adapting to novel conditions; rather, it leverages prior motor coordination experience in the context of delays. The objective is to reduce the need for extensive training or sophisticated automation designs. A human-centered experiment involving 41 participants was conducted to examine the effects of modified haptic cues in teleoperations with delays. These cues were generated from high-fidelity physics engines using parameters from robot-end sensors or physics engine simulations. The results underscored several benefits, notably the considerable reduction in task time and enhanced user perceptions about visual delays. Real-time haptic feedback, or the anchoring method, emerged as a significant contributor to these benefits, showcasing reduced cognitive load, bolstered self-confidence, and minimized frustration. Beyond the prevalent methods of automation design and training, this research underscores induced human adaptation as a pivotal avenue in robot teleoperation. It seeks to enhance teleoperation efficacy through rapid human adaptation, offering insights beyond just optimizing robotic systems for delay compensations.

Dysregulated noradrenergic response is associated with symptom severity in individuals with schizophrenia

Introduction

The locus coeruleus-noradrenaline (LC-NA) system is involved in a wide range of cognitive functions and may be altered in schizophrenia. A non-invasive method to indirectly measure LC activity is task-evoked pupillary response. Individuals with schizophrenia present reduced pupil dilation compared to healthy subjects, particularly when task demand increases. However, the extent to which alteration in LC activity contributes to schizophrenia symptomatology remains largely unexplored. We aimed to investigate the association between symptomatology, cognition, and noradrenergic response in individuals with schizophrenia.

Methods

We assessed task-evoked pupil dilation during a pro- and antisaccade task in 23 individuals with schizophrenia and 28 healthy subjects.

Results

Both groups showed similar preparatory pupil dilation during prosaccade trials, but individuals with schizophrenia showed significantly lower pupil dilation compared to healthy subjects in antisaccade trials. Importantly, reduced preparatory pupil dilation for antisaccade trials was associated with worse general symptomatology in individuals with schizophrenia.

Discussion

Our findings suggest that changes in LC-NA activity - measured by task-evoked pupil dilation - when task demand increases is associated with schizophrenia symptoms. Interventions targeting the modulation of noradrenergic responses may be suitable candidates to reduce schizophrenia symptomatology.

Learning progress mediates the link between cognitive effort and task engagement

While a substantial body of work has shown that cognitive effort is aversive and costly, a separate line of research on intrinsic motivation suggests that people spontaneously seek challenging tasks. According to one prominent account of intrinsic motivation, the learning progress motivation hypothesis, the preference for difficult tasks reflects the dynamic range that these tasks yield for changes in task performance (Kaplan & Oudeyer, 2007). Here we test this hypothesis, by asking whether greater engagement with intermediately difficult tasks, indexed by subjective ratings and objective pupil measurements, is a function of trial-wise changes in performance. In a novel paradigm, we determined each individual's capacity for task performance and used difficulty levels that are low, intermediately challenging or high for that individual. We demonstrated that challenging tasks resulted in greater liking and engagement scores compared with easy tasks. Pupil size tracked objective task difficulty, where challenging tasks were associated with greater pupil responses than easy tasks. Most importantly, pupil responses were predicted by trial-to-trial changes in average accuracy as well as learning progress (derivative of average accuracy), while greater pupil responses also predicted greater subjective engagement scores. Together, these results substantiate the learning progress motivation hypothesis stating that the link between task engagement and cognitive effort is mediated the dynamic range for changes in task performance.

Effects of reading contextualized physics problems among men and women: A psychophysiological approach

To counteract declining interest in science, contextualizing course material has been suggested, despite little evidence supporting this strategy. We assessed how reading physics problems in different contexts-none, technical, or humanistic-impacted performance and implicit cognitive and affective situational interest (SI) among undergraduate men and women (n = 60). We hypothesized that contextualized problems would increase cognitive SI, boosting performance. We also investigated existing hypotheses that this influence would be stronger when contexts matched stereotypical gender interests. Pupillometric and electroencephalographic data served to indicate cognitive SI, while electrodermal activity (EDA) and valence were measures of affective SI. Significantly higher valence was observed in decontextualized than humanistic problems (p = 0.003) specifically among men (p < 0.001). Greater EDA (p = 0.019) and decontextualized problems (p < 0.001) yielded greater performance than contextualized problems for all participants. Results emphasize the importance of affective SI and of avoiding gender biases in curricular development. This study encourages caution if implementing contextualization.

The contribution of temporal analysis of pupillometry measurements to cognitive research

Reaction time (RT) is one of the most frequently used measures to detect cognitive processes. When tasks require more cognitive processes/resources, reaction is slower. However, RTs may provide only restricted information regarding the temporal characteristics of cognitive processes. Pupils respond reflexively to light but also to cognitive activation. The more cognitive resources a task requires, the more the pupil dilates. However, despite being able to use temporal changes in pupil size (advanced devices measure changes in pupil diameter with sampling rates of above 1000 samples per second), most past studies using pupil dilation have not investigated temporal changes in pupil response. In the current paper, we discuss the advantage of the temporal approach to analyze pupil changes compared to a more traditional perspective, specifically, singular value methods such as mean value and peak amplitude value. Using data from two recent studies conducted in our laboratory, we demonstrate the differences in findings arising from the various analyses. In particular, we focus on the advantage of temporal analysis in detecting hidden effects, investigating temporal characterizations of the effects, and validating the experimental manipulation.

The pupillometry of the possible: an investigation of infants' representation of alternative possibilities

Contrasting possibilities has a fundamental adaptive value for prediction and learning. Developmental research, however, has yielded controversial findings. Some data suggest that preschoolers might have trouble in planning actions that take into account mutually exclusive possibilities, while other studies revealed an early understanding of alternative future outcomes based on infants' looking behaviour. To better understand the origin of such abilities, here we use pupil dilation as a potential indicator of infants' representation of possibilities. Ten- and 14-month-olds were engaged in an object-identification task by watching video animations where three different objects with identical top parts moved behind two screens. Importantly, a target object emerged from one of the screens but remained in partial occlusion, revealing only its top part, which was compatible with a varying number of possible identities. Just as adults' pupil diameter grows monotonically with the amount of information held in memory, we expected that infants' pupil size would increase with the number of alternatives sustained in memory as candidate identities for the partially occluded object. We found that pupil diameter increased with the object's potential identities in 14- but not in 10-month-olds. We discuss the implications of these results for the foundation of humans' capacities to represent alternatives. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

Gamma-Band Modulation in Parietal Area as the Electroencephalographic Signature for Performance in Auditory-Verbal Working Memory: An Exploratory Pilot Study in Hearing and Unilateral Cochlear Implant Children

This pilot study investigates the neurophysiological patterns of visual and auditory verbal working memory (VWM) in unilateral cochlear implant users (UCIs). We compared the task-related electroencephalogram (EEG) power spectral density of 7- to 13-year-old UCIs (n = 7) with a hearing control group (HC, n = 10) during the execution of a three-level n-back task with auditory and visual verbal (letters) stimuli. Performances improved as memory load decreased regardless of sensory modality (SM) and group factors. Theta EEG activation over the frontal area was proportionally influenced by task level; the left hemisphere (LH) showed greater activation in the gamma band, suggesting lateralization of VWM function regardless of SM. However, HCs showed stronger activation patterns in the LH than UCIs regardless of SM and in the parietal area (PA) during the most challenging audio condition. Linear regressions for gamma activation in the PA suggest the presence of a pattern-supporting auditory VWM only in HCs. Our findings seem to recognize gamma activation in the PA as the signature of effective auditory VWM. These results, although preliminary, highlight this EEG pattern as a possible cause of the variability found in VWM outcomes in deaf children, opening up new possibilities for interdisciplinary research and rehabilitation intervention.

Evaluation of children’s cognitive load in processing and storage of their spatial working memory

Working memory performance affects children’s learning. This study examined objective (task performance), subjective (self-report), and physiological (pupil dilation) cognitive load (CL) while children completed a spatial working memory complex span task. Frist, 80 Taiwanese 11-year-olds (40 boys) who participated in Experiment 1 confirmed the suitability of the materials. Then, 72 Taiwanese 11-year-olds (35 boys) were assigned to high and low complexity groups to participate in Experiment 2 to test the study hypothesis. Children had to recall at the end of a dual-task list and answer two questions regarding the difficulty and mental effort involved in processing and storage. Their pupil diameters were recorded using an eye-tracker. Two-way mixed ANOVA found that the processing requirements and memory load reduced storage and aggravated the subjective CL of storage; the subjective CL of processing was higher under highly complex conditions. Stepwise regression analysis indicated that subjective CL of processing predicted memory performance in low CL conditions, and physiological CL of processing predicted it in high CL conditions.

Action-based predictions affect visual perception, neural processing, and pupil size, regardless of temporal predictability

Sensory consequences of one's own action are often perceived as less intense, and lead to reduced neural responses, compared to externally generated stimuli. Presumably, such sensory attenuation is due to predictive mechanisms based on the motor command (efference copy). However, sensory attenuation has also been observed outside the context of voluntary action, namely when stimuli are temporally predictable. Here, we aimed at disentangling the effects of motor and temporal predictability-based mechanisms on the attenuation of sensory action consequences. During fMRI data acquisition, participants (N = 25) judged which of two visual stimuli was brighter. In predictable blocks, the stimuli appeared temporally aligned with their button press (active) or aligned with an automatically generated cue (passive). In unpredictable blocks, stimuli were presented with a variable delay after button press/cue, respectively. Eye tracking was performed to investigate pupil-size changes and to ensure proper fixation. Self-generated stimuli were perceived as darker and led to less neural activation in visual areas than their passive counterparts, indicating sensory attenuation for self-generated stimuli independent of temporal predictability. Pupil size was larger during self-generated stimuli, which correlated negatively with the blood oxygenation level dependent (BOLD) response: the larger the pupil, the smaller the BOLD amplitude in visual areas. Our results suggest that sensory attenuation in visual cortex is driven by action-based predictive mechanisms rather than by temporal predictability. This effect may be related to changes in pupil diameter. Altogether, these results emphasize the role of the efference copy in the processing of sensory action consequences.

Applying functional near-infrared spectroscopy and eye-tracking in a naturalistic educational environment to investigate physiological aspects that underlie the cognitive effort of children during mental rotation tests

Spatial cognition is related to academic achievement in science, technology, engineering, and mathematics (STEM) domains. Neuroimaging studies suggest that brain regions' activation might be related to the general cognitive effort while solving mental rotation tasks (MRT). In this study, we evaluate the mental effort of children performing MRT tasks by measuring brain activation and pupil dilation. We use functional near-infrared spectroscopy (fNIRS) concurrently to collect brain hemodynamic responses from children's prefrontal cortex (PFC) and an Eye-tracking system to measure pupil dilation during MRT. Thirty-two healthy students aged 9-11 participated in this experiment. Behavioral measurements such as task performance on geometry problem-solving tests and MRT scores were also collected. The results were significant positive correlations between the children's MRT and geometry problem-solving test scores. There are also significant positive correlations between dorsolateral PFC (dlPFC) hemodynamic signals and visuospatial task performances (MRT and geometry problem-solving scores). Moreover, we found significant activation in the amplitude of deoxy-Hb variation on the dlPFC and that pupil diameter increased during the MRT, suggesting that both physiological responses are related to mental effort processes during the visuospatial task. Our findings indicate that children with more mental effort under the task performed better. The multimodal approach to monitoring students' mental effort can be of great interest in providing objective feedback on cognitive resource conditions and advancing our comprehension of the neural mechanisms that underlie cognitive effort. Hence, the ability to detect two distinct mental states of rest or activation of children during the MRT could eventually lead to an application for investigating the visuospatial skills of young students using naturalistic educational paradigms.

Eye-Tracking in Interpreting Studies: A Review of Four Decades of Empirical Studies

It has been four decades since eye-tracking was first used in interpreting studies, and recent years has witnessed a growing interest in the application of this method, which holds great potential for offering a look into the “black box” of interpreting processing. However, little attention has been paid to comprehensively illustrating what has been done, what can be done, and what needs to be done with this method in this discipline. With this in view, this paper sets out to understand contributions of previous studies—key themes discussed, eye-tracking measures used, their limitations and implications, and future directions. To this end, we conduct a review of a total of 26 empirical papers from peer-reviewed journals within a time span of 4 decades ranging from 1981 to 2021. This study, as the first attempt of its kind at a comprehensive review on using eye-tracking in interpreting studies, should have implications for researchers, educators, and practitioners.

Dual-task interference while receiving information on a head mounted display and manual tracking with and without auditory warnings

BACKGROUND

Dual-task interference is a concern when users attempt to use head mounted displays (HMD) while performing seated manual control tasks. Auditory warnings of information displayed on the HMD may or may not assist users while multi-tasking.

OBJECTIVE

In this study, we investigated the dual-task interference of words, to be remembered later, presented on a HMD and motor control in a manual tracking task. The word presentations were warned or not warned with auditory signals and the impact of audio warnings were examined.

METHODS

Participants were required to perform five tasks: (1) a compensatory tracking task performed alone, (2) a word memory task performed alone, (3) a tracking and word memory task, (4) a word memory task with words preceded by audial signals, and (5) a tracking and word memory task with words preceded by audial signals. In addition, in the dual-tasks, half of the word presentations were paired with directions changes in the tracking task to test for immediate perceptual interference.

RESULTS

There were significant dual task costs for both mean tracking error and later word recall. Additionally, participants took significantly longer to respond to motion changes paired with word presentations than motion changes not paired with word presentations. However, the impact of auditory warnings on tracking performance was nuanced.

CONCLUSION

Even with an in field of view transparent HMD momentary and sustained cognitive dual-task interference remains. Reaction times are affected most in the worst case scenario, when task critical events occur at the same time as a text message.

APPLICATION

The use of HMDs in time critical manual control tasks (such as operating machinery) should be limited. The use of audial warnings to alert operators to information displayed on an HMD requires further research; it may disrupt appropriate or natural task ordering.

Classifying the difficulty levels of working memory tasks by using pupillary response

Knowing the difficulty of a given task is crucial for improving the learning outcomes. This paper studies the difficulty level classification of memorization tasks from pupillary response data. Developing a difficulty level classifier from pupil size features is challenging because of the inter-subject variability of pupil responses. Eye-tracking data used in this study was collected while students solved different memorization tasks divided as low-, medium-, and high-level. Statistical analysis shows that values of pupillometric features (as peak dilation, pupil diameter change, and suchlike) differ significantly for different difficulty levels. We used a wrapper method to select the pupillometric features that work the best for the most common classifiers; Support Vector Machine (SVM), Decision Tree (DT), Linear Discriminant Analysis (LDA), and Random Forest (RF). Despite the statistical difference, experiments showed that a random forest classifier trained with five features obtained the best F1-score (82%). This result is essential because it describes a method to evaluate the cognitive load of a subject performing a task using only pupil size features.

The effect of background speech on attentive sound processing: A pupil dilation study

Listening to task-irrelevant speech while performing a cognitive task can involuntarily deviate our attention and lead to decreases in performance. One explanation for the impairing effect of irrelevant speech is that semantic processing can consume attentional resources. In the present study, we tested this assumption by measuring performance in a non-linguistic attentional task while participants were exposed to meaningful (native) and non-meaningful (foreign) speech. Moreover, based on the tight relation between pupillometry and attentional processes, we also registered changes in pupil diameter size to quantify the effect of meaningfulness upon attentional allocation. To these aims, we recruited 41 native German speakers who had neither received formal instruction in French nor had extensive informal contact with this language. The focal task consisted of an auditory oddball task. Participants performed a duration discrimination task containing frequently repeated standard sounds and rarely presented deviant sounds while a story was read in German or (non-meaningful) French in the background. Our results revealed that, whereas effects of language meaningfulness on attention were not detectable at the behavioural level, participants' pupil dilated more in response to the sounds of the auditory task when background speech was played in non-meaningful French compared to German, independent of sound type. In line with the initial hypothesis, this suggested that semantic processing of the native language required attentional resources, which lead to fewer resources devoted to the processing of the sounds of the focal task. Our results highlight the potential of the pupil dilation response for the investigation of subtle cognitive processes that might not surface when only behaviour is measured.

Arousal-based pupil modulation is dictated by luminance

Pupillometry has become a standard measure for assessing arousal state. However, environmental factors such as luminance, a primary dictator of pupillary responses, often vary across studies. To what degree does luminance interact with arousal-driven pupillary changes? Here, we parametrically assessed luminance-driven pupillary responses across a wide-range of luminances, while concurrently manipulating cognitive arousal using auditory math problems of varying difficulty. At the group-level, our results revealed that the modulatory effect of cognitive arousal on pupil size interacts multiplicatively with luminance, with the largest effects occurring at low and mid-luminances. However, at the level of individuals, there were qualitatively distinct individual differences in the modulatory effect of cognitive arousal on luminance-driven pupillary responses. Our findings suggest that pupillometry as a measure for assessing arousal requires more careful consideration: there are ranges of luminance levels that are more ideal in observing pupillary differences between arousal conditions than others.

High fluid intelligence is characterized by flexible allocation of attentional resources: Evidence from EEG

Recently, the integrated control hypothesis (Lu et al., 2020) was proposed to explain the relationship between fluid intelligence (Gf) and attentional resource allocation. This hypothesis suggested that individuals with higher Gf tend to flexibly and adaptively allocate their limited resources according to the task type and task difficulty rather than simply exert more or fewer resources in any condition. To examine this hypothesis, the present study used electroencephalogram (EEG) indicators (i.e., frontal theta-ERS and parietal-occipital alpha-ERD) as the measurements of participants' resource allocation during the exploration task and exploitation task with different difficulties. The results found that higher Gf individuals tend to allocate fewer resources in all difficulty levels in the exploitation task compared to average Gf participants. In contrast, in the exploration task, higher Gf participants would allocate more resources in the medium- and high-difficulty levels than average Gf participants, but this phenomenon was only found in males. These findings provided supportive evidence for the integrated control hypothesis that flexible and adaptive attentional control ability are important characteristics of human intelligence.

Neurophysiological Verbal Working Memory Patterns in Children: Searching for a Benchmark of Modality Differences in Audio/Video Stimuli Processing

Exploration of specific brain areas involved in verbal working memory (VWM) is a powerful but not widely used tool for the study of different sensory modalities, especially in children. In this study, for the first time, we used electroencephalography (EEG) to investigate neurophysiological similarities and differences in response to the same verbal stimuli, expressed in the auditory and visual modality during the n-back task with varying memory load in children. Since VWM plays an important role in learning ability, we wanted to investigate whether children elaborated the verbal input from auditory and visual stimuli through the same neural patterns and if performance varies depending on the sensory modality. Performance in terms of reaction times was better in visual than auditory modality (p = 0.008) and worse as memory load increased regardless of the modality (p < 0.001). EEG activation was proportionally influenced by task level and was evidenced in theta band over the prefrontal cortex (p = 0.021), along the midline (p = 0.003), and on the left hemisphere (p = 0.003). Differences in the effects of the two modalities were seen only in gamma band in the parietal cortices (p = 0.009). The values of a brainwave-based engagement index, innovatively used here to test children in a dual-modality VWM paradigm, varied depending on n-back task level (p = 0.001) and negatively correlated (p = 0.002) with performance, suggesting its computational effectiveness in detecting changes in mental state during memory tasks involving children. Overall, our findings suggest that auditory and visual VWM involved the same brain cortical areas (frontal, parietal, occipital, and midline) and that the significant differences in cortical activation in theta band were more related to memory load than sensory modality, suggesting that VWM function in the child's brain involves a cross-modal processing pattern.

Investigating the Effect of Real-Time Center of Pressure (CoP) Feedback Training on the Swing Phase of Lower Limb Kinematics in Transfemoral Prostheses with SACH foot

Transfemoral amputee often encounters reduced toe clearance resulting in trip-related falls. Swing phase joint angles have been shown to influence the toe clearance therefore, training intervention that targets shaping the swing phase joint angles can potentially enhance toe clearance. The focus of this study was to investigate the effect of the shift in the location of the center of pressure (CoP) during heel strike on modulation of the swing phase joint angles in able-bodied participants (n=6) and transfemoral amputees (n=3). We first developed a real-time CoP-based visual feedback system such that participants could shift the CoP during treadmill walking. Next, the kinematic data were collected during two different walking sessions- baseline (without feedback) and feedback (shifting the CoP anteriorly/posteriorly at heel strike to match the target CoP location). Primary swing phase joint angle adaptations were observed with feedback such that during the mid-swing phase, posterior CoP shift feedback significantly increases (p<0.05) the average hip and knee flexion angle by 11.55 degrees and 11.86 degrees respectively in amputees, whereas a significant increase (p<0.05) in ankle dorsiflexion, hip and knee flexion angle by 3.60 degrees, 3.22 degrees, and 1.27 degrees respectively compared to baseline was observed in able-bodied participants. Moreover, an opposite kinematic adaptation was seen during anterior CoP shift feedback. Overall, results confirm a direct correlation between the CoP shift and the modulation in the swing phase lower limb joint angles.

Effects of presentation modality and duration on children's strategy use: A study in computational estimation

Two experiments were run to determine how presentation modality and duration influence children's arithmetic performance and strategy selection. Third and fourth graders were asked to find estimates for two-digit addition problems (e.g., 52 + 39). Children were tested in three conditions: (1) time-unlimited visual, (2) time-limited visual, or (3) time-limited auditory conditions. Moreover, we assessed children's working-memory updating and arithmetic fluency. Children were told which strategy to use on each problem to assess arithmetic performance while executing strategies, in Experiment 1, and were asked to choose the best strategy of three available strategies to assess strategy selection, in Experiment 2. Presentation modality influenced strategy execution (i.e., children were faster and more accurate in problems under visual than auditory conditions) but only in children with low updating abilities. In contrast, presentation modality had no effect on children's strategy selection. Presentation duration had an effect on both strategy execution and strategy selection with time-limited presentation leading to a decline in children's performance. Interestingly, specifically in children with low updating abilities, time-limited presentation led to poorer performance. Hence, efficient updating seemed to compensate for detrimental effects of auditory in comparison to visual and time-limited in comparison to time-unlimited presentation. These findings have important implications for determining conditions under which children execute strategies most efficiently and select the best strategy on each problem most often, as well as for understanding mechanisms underlying strategic behaviour.

Neurophysiological indicators of internal attention: An fMRI-eye-tracking coregistration study

Many goal-directed, as well as spontaneous everyday activities (e.g., planning, mind-wandering), rely on an internal focus of attention. This fMRI-eye-tracking coregistration study investigated brain mechanisms and eye behavior related to internally versus externally directed cognition. Building on an established paradigm, we manipulated internal attention demands within tasks utilizing conditional stimulus masking. Internally directed cognition involved bilateral activation of the lingual gyrus and inferior parietal lobe areas as well as wide-spread deactivation of visual networks. Moreover, internally directed cognition was related to greater pupil diameter, pupil diameter variance, blink duration, fixation disparity variance, and smaller amounts of microsaccades. FMRI-eye-tracking covariation analyses further revealed that larger pupil diameter was related to increased activation of basal ganglia and lingual gyrus. It can be concluded that internally and externally directed cognition are characterized by distinct neurophysiological signatures. The observed neurophysiological differences indicate that internally directed cognition is associated with reduced processing of task-irrelevant information and increased mental load. These findings shed further light on the interplay between neural and perceptual mechanisms contributing to an internal focus of attention.

Musical and Bodily Predictors of Mental Effort in String Quartet Music: An Ecological Pupillometry Study of Performers and Listeners

Music performance can be cognitively and physically demanding. These demands vary across the course of a performance as the content of the music changes. More demanding passages require performers to focus their attention more intensity, or expend greater “mental effort.” To date, it remains unclear what effect different cognitive-motor demands have on performers' mental effort. It is likewise unclear how fluctuations in mental effort compare between performers and perceivers of the same music. We used pupillometry to examine the effects of different cognitive-motor demands on the mental effort used by performers and perceivers of classical string quartet music. We collected pupillometry, motion capture, and audio-video recordings of a string quartet as they performed a rehearsal and concert (for live audience) in our lab. We then collected pupillometry data from a remote sample of musically-trained listeners, who heard the audio recordings (without video) that we captured during the concert. We used a modelling approach to assess the effects of performers' bodily effort (head and arm motion; sound level; performers' ratings of technical difficulty), musical complexity (performers' ratings of harmonic complexity; a score-based measure of harmonic tension), and expressive difficulty (performers' ratings of expressive difficulty) on performers' and listeners' pupil diameters. Our results show stimulating effects of bodily effort and expressive difficulty on performers' pupil diameters, and stimulating effects of expressive difficulty on listeners' pupil diameters. We also observed negative effects of musical complexity on both performers and listeners, and negative effects of performers' bodily effort on listeners, which we suggest may reflect the complex relationships that these features share with other aspects of musical structure. Looking across the concert, we found that both of the quartet violinists (who exchanged places halfway through the concert) showed more dilated pupils during their turns as 1st violinist than when playing as 2nd violinist, suggesting that they experienced greater arousal when “leading” the quartet in the 1st violin role. This study shows how eye tracking and motion capture technologies can be used in combination in an ecological setting to investigate cognitive processing in music performance.

Pupillary dilation elicited by attending to two disks with different luminance

Pupils become smaller when people attend to a bright disk as compared to a dark disk. However, people can divide their attention into several distinct positions, which is referred to as divided attention, and pupillary responses under such conditions have not been investigated. In this study, we examined how pupils would respond when people attended to two disks presented at two distinct positions by conducting three experiments. We found that the pupillary response when attending to two disks with different luminance was larger than when attending to a single brighter disk and was comparable to that when attending to a single darker disk, whereas the pupillary response when attending to two disks with identical luminance was not larger than when attending to a single disk (irrespective of the disk luminance). Furthermore, we found that the magnitude of pupillary dilation was determined by the magnitude of the luminance difference between two disks. These results make a useful contribution to the literature on human pupillary responses.

Top-down effect on pupillary response: Evidence from shape from shading

Shaded 2D images often create an illusion of depth, due to the shading information and assumptions regarding the location of the light source. Specifically, 2D images that are lighter on top usually appear convex while images that are darker on top, usually appear concave, reflecting the assumption that light is coming from above. The process of recovering the 3D shape of a shaded image is called Shape from Shading. Here we examined whether the pupil responds to the illusion of depth in a shape from shading task. In three experiments we show that pupil size is affected by the percept of depth, so that it dilates more when participants perceive the stimulus as concave, compared to when they perceive it as convex. This only happens if participants make a judgment regarding the shape of the stimulus or when they view it passively but are aware of the different shapes. No differences in pupil size were found with passive viewing if participants were not aware of the illusion, suggesting that some aspects of shape from shading require attention. All stimuli were equiluminant, and the percept of depth was created by manipulating the orientation of the shading, so that changes in pupil size could not be accounted by changes in the amount of light in the image. We posit, and confirmed it in a behavioral control experiment, that the perception of depth is translated to a subjective perception of darkness, due to the "darker is deeper" heuristic and conclude that the pupillary physiological response reflects the subjective perception of light.

How Does Cognitive Effort Influence the Tactical Behavior of Soccer Players?

In the present study, we aimed to investigate the association between soccer players' cognitive effort and their tactical behavior. We assessed 52 young male soccer players from a first division Brazilian club, using FUT-SAT to evaluate tactical behavior efficiency and Mobile Eye Tracking-XG software and a video test protocol to measure pupillary behavior and cognitive effort. Following data collection, statistical analyses were performed using the Kolmogorov-Smirnov normality test, and linear regression. We found a high inverse association between cognitive effort and tactical behavior efficiency; players with less cognitive effort during the task displayed higher values of tactical behavior efficiency on the field. We concluded that sustaining less cognitive effort in game situations helped players realize better tactical behavior and enabled better performance.

Pupillary Responses for Cognitive Load Measurement to Classify Difficulty Levels in an Educational Video Game: Empirical Study

Background

A learning task recurrently perceived as easy (or hard) may cause poor learning results. Gamer data such as errors, attempts, or time to finish a challenge are widely used to estimate the perceived difficulty level. In other contexts, pupillometry is widely used to measure cognitive load (mental effort); hence, this may describe the perceived task difficulty.

Objective

This study aims to assess the use of task-evoked pupillary responses to measure the cognitive load measure for describing the difficulty levels in a video game. In addition, it proposes an image filter to better estimate baseline pupil size and to reduce the screen luminescence effect.

Methods

We conducted an experiment that compares the baseline estimated from our filter against that estimated from common approaches. Then, a classifier with different pupil features was used to classify the difficulty of a data set containing information from students playing a video game for practicing math fractions.

Results

We observed that the proposed filter better estimates a baseline. Mauchly’s test of sphericity indicated that the assumption of sphericity had been violated (χ²₁₄=0.05; P=.001); therefore, a Greenhouse-Geisser correction was used (ε=0.47). There was a significant difference in mean pupil diameter change (MPDC) estimated from different baseline images with the scramble filter (F_5,78=30.965; P<.001). Moreover, according to the Wilcoxon signed rank test, pupillary response features that better describe the difficulty level were MPDC (z=−2.15; P=.03) and peak dilation (z=−3.58; P<.001). A random forest classifier for easy and hard levels of difficulty showed an accuracy of 75% when the gamer data were used, but the accuracy increased to 87.5% when pupillary measurements were included.

Conclusions

The screen luminescence effect on pupil size is reduced with a scrambled filter on the background video game image. Finally, pupillary response data can improve classifier accuracy for the perceived difficulty of levels in educational video games.

Adolescent girls' physiological reactivity to real-world peer feedback: A pilot study to validate a Peer Expressed Emotion task

Peer feedback becomes highly salient during adolescence, especially for girls. The way in which adolescents react to social feedback is associated with psychosocial adjustment and mental health. Consequently, researchers are increasingly interested in understanding the physiological and neural underpinnings of adolescent response to feedback by simulating the experience of rejection and acceptance using computer-based paradigms. However, these paradigms typically use nonfamiliar peers and the facade of internet chatrooms or games to present artificial peer feedback. The current study piloted the use of a novel and potentially more ecologically valid peer expressed emotion paradigm in which participants listen to prerecorded clips of ostensible personalized feedback made by their close friend. Physiological data measuring autonomic nervous system response were collected as an index of emotional reactivity/arousal and cognitive-affective processing. Results show promising preliminary evidence validating the task for future use. Participants (N = 18 girls, aged 11-17 years) reported feeling more positive following praise, relative to critical and neutral feedback, and reported feeling more upset following criticism, relative to praise and neutral feedback. Girls exhibited greater pupillary dilation, skin conductance levels (N = 17), and/or heart rate (N = 17) while listening to affectively charged, peer feedback compared with neutral yet personally relevant statements. Girls also exhibited variable physiological response when listening to praising versus critical feedback. Findings from this pilot study validate the use of this novel Peer Expressed Emotion task for the investigation of adolescents' emotional and physiological reactivity in response to real-world peer evaluation. However, it is important to recognize that this study provides only preliminary findings and that future research is needed to replicate the results in larger samples.

Effects and mechanisms of information saliency in enhancing value-based decision-making in younger and older adults

Aging attenuates frontostriatal network functioning, which could lead to deficits in value computation when decision-making involves uncertainty. Although it has been shown that visually enhancing information saliency of outcome probability can improve decision-making in old age, mechanisms of this effect are still unclear. In the present study, the saliency of outcome probability was increased using a color-coding scheme as a decision aid in a mixed lottery choice task, and spontaneous eye-blink rate and pupillary responses were assessed in younger and older adults. Older adults showed lower value sensitivity than younger adults; however, increasing information saliency benefitted choice behaviors in both age groups. Furthermore, the decision aid reduced pupil size during decision-making in both age groups, suggesting that enhancing information saliency decreases cognitive demands of value computation. Baseline value sensitivity was negatively correlated with benefit of enhancing information saliency only in older adults. As value representations in older decision makers are less distinctive at baseline, they may have required more environmental compensation than younger adults.

Effects of blocked vs. interleaved administration mode on saccade preparatory set revealed using pupillometry

Eye movements have been used extensively to assess information processing and cognitive function. However, significant variability in saccade performance has been observed, which could arise from methodological variations across different studies. For example, prosaccades and antisaccades have been studied using either a blocked or interleaved design, which has a significant influence on error rates and latency. This is problematic as it makes it difficult to compare saccade performance across studies and may limit the ability to use saccades as a behavioural assay to assess neurocognitive function. Thus, the current study examined how administration mode influences saccade related preparatory activity by employing pupil size as a non-invasive proxy for neural activity related to saccade planning and execution. Saccade performance and pupil dynamics were examined in eleven participants as they completed pro- and antisaccades in blocked and interleaved paradigms. Results showed that administration mode significantly modulated saccade performance and preparatory activity. Reaction times were longer for both pro- and antisaccades in the interleaved condition, compared to the blocked condition (p < 0.05). Prosaccade pupil dilations were larger in the interleaved condition (p < 0.05), while antisaccade pupil dilations did not significantly differ between administration modes. Additionally, ROC analysis provided preliminary evidence that pupil size can effectively predict saccade directional errors prior to saccade onset. We propose that task-evoked pupil dilations reflect an increase in preparatory activity for prosaccades and the corresponding cognitive demands associated with interleaved administration mode. Overall, the results highlight the importance that administration mode plays in the design of neurocognitive tasks.

Physiological investigation of cognitive load in real-life train travelers during information processing

Management of travelers' cognitive load is crucial for efficient information processing for optimal railway operations. We investigated variations in travelers' cognitive load with different expertise levels, in a field study. We aimed to assess the use of three eye metrics: pupil diameter, saccade amplitude and gaze of fixation duration for cognitive load analysis between expert and novice travelers in a Mass Transit train station: Saint-Michel Notre Dame. Physiological measures of emotional activity through skin conductance responses were also investigated, together with subjective measures of mental load using NASA-Task Load Index. Our results followed our expectations, where novice travelers expressed higher cognitive load than expert travelers, characterized by wider pupil diameter, larger amplitude of saccade and longer gaze duration, as well as higher electrodermal activity and NASA-TLX. Additional observations showed that experts used hierarchical and symmetrical scan paths, with more intense exposure on relevant information, characterized by brighter heat maps. This difference in behavior showed a clear difference in strategies for information retrieval at different expertise levels. Metrics of eye tracking device, together with electrodermal activity, proved to be potent in cognitive load analysis of train travelers, and helped to provide insights for real-life information processing.

Oculometric signature of switch into awareness? Pupil size predicts sudden insight whereas microsaccades predict problem-solving via analysis

According to the Gestalt theorists, restructuring is an essential component of insight problem-solving, contributes to the "Aha!" experience, and is similar to the perceptual switch experienced when reinterpreting ambiguous figures. Previous research has demonstrated that pupil diameter increases during the perceptual switch of ambiguous figures, and indexes norepeinephrine functioning mediated by the locus coeruleus. In this study, we investigated if pupil diameter similarly predicts the switch into awareness people experience when solving a problem via insight. Additionally, we explored eye movement dynamics during the same task to investigate if the problem-solving strategies used are linked to specific oculomotor behaviors. In 38 participants, pupil diameter increased about 500 msec prior to solution only in trials for which subjects report having an insight. In contrast, participants increased their microsaccade rate only prior to non-insight solutions. Pupil dilation and microsaccades were not reliably related, but both appear to be robust markers of how people solve problems (with or without insight). The pupil size change seen when people have an "Aha!" moment represents an indicator of the switch into awareness of unconscious processes humans depend upon for insight, and suggests important involvement of norepinephrine, via the locus coeruleus, in sudden insight.

Component processes in free-viewing visual search: Insights from fixation-aligned pupillary response averaging

Pupil size changes during a visual search may reflect cognitive processes, such as effort and memory accumulation, but methodological confounds and the general lack of literature in this area leave the reliability of findings open to question. We used a novel synthesis of experimental methods and averaging techniques to explore how cognitive processing unfolds during free-viewing visual search for multiple targets. Twenty-seven participants completed 152 searches across two separate 1-hour sessions. The number of targets present (Targets: 0, 1, 2, and 3) in each trial was the main manipulation and the task was to "find all of the targets" and report the total via mouse-click at the end of the trial. Search time lasted for 10 seconds or until the participant purported to have found all of the targets, in which case they could terminate the search via keypress. Whole-trial pupil analysis revealed a significant effect of button pressing as well as a significant main effect of targets for trials that were not self-terminated via button press. Fixation-aligned pupil responses revealed transient modulations in pupil size following initial fixations on targets but not distractors and refixations on both targets and distractors. Owing to rigorous control over experimental confounds and a detailed analysis and correction of eye-movement-related measurement error, we confidently discuss these findings in terms of task-related processing and underlying brain activity.

Rapid adaptation to fully intelligible nonnative-accented speech reduces listening effort

In noisy settings or when listening to an unfamiliar talker or accent, it can be difficult to understand spoken language. This difficulty typically results in reductions in speech intelligibility, but may also increase the effort necessary to process the speech even when intelligibility is unaffected. In this study, we used a dual-task paradigm and pupillometry to assess the cognitive costs associated with processing fully intelligible accented speech, predicting that rapid perceptual adaptation to an accent would result in decreased listening effort over time. The behavioural and physiological paradigms provided converging evidence that listeners expend greater effort when processing nonnative- relative to native-accented speech, and both experiments also revealed an overall reduction in listening effort over the course of the experiment. Only the pupillometry experiment, however, revealed greater adaptation to nonnative- relative to native-accented speech. An exploratory analysis of the dual-task data that attempted to minimise practice effects revealed weak evidence for greater adaptation to the nonnative accent. These results suggest that even when speech is fully intelligible, resolving deviations between the acoustic input and stored lexical representations incurs a processing cost, and adaptation may attenuate this cost.

Cognitive functioning, sleep quality, and work performance in non-clinical burnout: The role of working memory

We investigated the relationship between cognitive functioning, work performance, and sleep in non-clinical burnout. In a working population, an online survey was conducted with additional online neuropsychological tests of varying complexity, measuring attention and different components of working memory, of which the coordinating subcomponent the ‘Central Executive’ is thought to be the most vulnerable to stress. Results indicate that non-clinical burnout is associated with more—though not severe—sleep problems, more depressive complaints, impaired work performance, and with both subjective and objective cognitive impairments. Compared with healthy respondents (N = 107), people with non-clinical burnout (N = 17) had a significantly poorer performance on the tests of the visuospatial sketchpad and the Central Executive of the working memory. Our study also indicates that more complex tests may be more sensitive in detecting cognitive dysfunction in non-clinical burnout. Furthermore, a relationship was found between dual-task performance and work performance. Regarding to sleep quality, in our sample of people with non-clinical burnout, there were no severe sleep problems. In the entire sample, however, insomnia was significantly related to subjective, but not objective, cognitive functioning, and also not to work performance.

The uulmMAC Database-A Multimodal Affective Corpus for Affective Computing in Human-Computer Interaction

In this paper, we present a multimodal dataset for affective computing research acquired in a human-computer interaction (HCI) setting. An experimental mobile and interactive scenario was designed and implemented based on a gamified generic paradigm for the induction of dialog-based HCI relevant emotional and cognitive load states. It consists of six experimental sequences, inducing Interest, Overload, Normal, Easy, Underload, and Frustration. Each sequence is followed by subjective feedbacks to validate the induction, a respiration baseline to level off the physiological reactions, and a summary of results. Further, prior to the experiment, three questionnaires related to emotion regulation (ERQ), emotional control (TEIQue-SF), and personality traits (TIPI) were collected from each subject to evaluate the stability of the induction paradigm. Based on this HCI scenario, the University of Ulm Multimodal Affective Corpus (uulmMAC), consisting of two homogenous samples of 60 participants and 100 recording sessions was generated. We recorded 16 sensor modalities including 4 × video, 3 × audio, and 7 × biophysiological, depth, and pose streams. Further, additional labels and annotations were also collected. After recording, all data were post-processed and checked for technical and signal quality, resulting in the final uulmMAC dataset of 57 subjects and 95 recording sessions. The evaluation of the reported subjective feedbacks shows significant differences between the sequences, well consistent with the induced states, and the analysis of the questionnaires shows stable results. In summary, our uulmMAC database is a valuable contribution for the field of affective computing and multimodal data analysis: Acquired in a mobile interactive scenario close to real HCI, it consists of a large number of subjects and allows transtemporal investigations. Validated via subjective feedbacks and checked for quality issues, it can be used for affective computing and machine learning applications.

Dissociable mappings of tonic and phasic pupillary features onto cognitive processes involved in mental arithmetic

Pupil size modulations have been used for decades as a window into the mind, and several pupillary features have been implicated in a variety of cognitive processes. Thus, a general challenge facing the field of pupillometry has been understanding which pupil features should be most relevant for explaining behavior in a given task domain. In the present study, a longitudinal design was employed where participants completed 8 biweekly sessions of a classic mental arithmetic task for the purposes of teasing apart the relationships between tonic/phasic pupil features (baseline, peak amplitude, peak latency) and two task-related cognitive processes including mental processing load (indexed by math question difficulty) and decision making (indexed by response times). We used multi-level modeling to account for individual variation while identifying pupil-to-behavior relationships at the single-trial and between-session levels. We show a dissociation between phasic and tonic features with peak amplitude and latency (but not baseline) driven by ongoing task-related processing, whereas baseline was driven by state-level effects that changed over a longer time period (i.e. weeks). Finally, we report a dissociation between peak amplitude and latency whereby amplitude reflected surprise and processing load, and latency reflected decision making times.

Illuminating the Dark Ages: Pupil Dilation as a Measure of Expectancy Violation Across the Life Span

Mainly for methodological reasons, little is known about the course of development of early cognitive competencies diagnosed with the violation of expectation (VoE) method in infants. The goal of this research was to evaluate the use of pupillometry as a unified approach to assess expectancy violations during and beyond the "dark ages" between 1 and 3 years. We tested children aged 1-6 years and adults (N = 279) with pictures of animals combined with matching or mismatching animal sounds. All age groups exhibited significantly greater pupil dilation in mismatched than matched trials. We conclude that pupillometry is a viable alternative to the VoE method that, by contrast to the latter, can be used throughout the life span.

Preprocessing pupil size data: Guidelines and code

Pupillometry has been one of the most widely used response systems in psychophysiology. Changes in pupil size can reflect diverse cognitive and emotional states, ranging from arousal, interest and effort to social decisions, but they are also widely used in clinical practice to assess patients’ brain functioning. As a result, research involving pupil size measurements has been reported in practically all psychology, psychiatry, and psychophysiological research journals, and now it has found its way into the primatology literature as well as into more practical applications, such as using pupil size as a measure of fatigue or a safety index during driving. The different systems used for recording pupil size are almost as variable as its applications, and all yield, as with many measurement techniques, a substantial amount of noise in addition to the real pupillometry data. Before analyzing pupil size, it is therefore of crucial importance first to detect this noise and deal with it appropriately, even prior to (if need be) resampling and baseline-correcting the data. In this article we first provide a short review of the literature on pupil size measurements, then we highlight the most important sources of noise and show how these can be detected. Finally, we provide step-by-step guidelines that will help those interested in pupil size to preprocess their data correctly. These guidelines are accompanied by an open source MATLAB script (available at https://github.com/ElioS-S/pupil-size). Given that pupil diameter is easily measured by standard eyetracking technologies and can provide fundamental insights into cognitive and emotional processes, it is hoped that this article will further motivate scholars from different disciplines to study pupil size.

CHAP: Open-source software for processing and analyzing pupillometry data

Pupil dilation is an effective indicator of cognitive and affective processes. Although several eyetracker systems on the market can provide effective solutions for pupil dilation measurement, there is a lack of tools for processing and analyzing the data provided by these systems. For this reason, we developed CHAP: open-source software written in MATLAB. This software provides a user-friendly graphical user interface for processing and analyzing pupillometry data. Our software creates uniform conventions for the preprocessing and analysis of pupillometry data and provides a quick and easy-to-use tool for researchers interested in pupillometry. To download CHAP or join our mailing list, please visit CHAP's website: http://in.bgu.ac.il/en/Labs/CNL/chap .

In a heart beat: Using driver's physiological changes to determine the quality of a takeover in highly automated vehicles

Developing conditionally automated driving systems is on the rise. Vehicles with full longitudinal and latitudinal control will allow drivers to engage in secondary tasks without monitoring the roadway, but users may be required to resume vehicle control to handle critical hazards. The loss of driver's situational awareness increases the potential for accidents. Thus, the automated systems need to estimate the driver's ability to resume control of the driving task. The aim of this study was to assess the physiological behaviour (heart rate and pupil diameter) of drivers. The assessment was performed during two naturalistic secondary tasks. The tasks were the email and the twenty questions task in addition to a control group that did not perform any tasks. The study aimed at finding possible correlations between the driver's physiological data and their responses to a takeover request. A driving simulator study was used to collect data from a total of 33 participants in a repeated measures design to examine the physiological changes during driving and to measure their takeover quality and response time. Secondary tasks induced changes on physiological measures and a small influence on response time. However, there was a strong observed correlation between the physiological measures and response time. Takeover quality in this study was assessed using two new performance measures called PerSpeed and PerAngle. They are identified as the mean percentage change of vehicle's speed and heading angle starting from a take-over request time. Using linear mixed models, there was a strong interaction between task, heart rate and pupil diameter and PerSpeed, PerAngle and response time. This, in turn, provided a measurable understanding of a driver's future responses to the automated system based on the driver's physiological changes to allow better decision making. The present findings of this study emphasised the possibility of building a driver mental state model and prediction system to determine the quality of the driver's responses in a highly automated vehicle. Such results will reduce accidents and enhance the driver's experience in highly automated vehicles.

Effects of spatial frequency and attention on pupillary response

Research has shown that the pupil responds differently depending on the spatial frequency of the gazing stimulus. In this study, we examined the effects of spatial- and object-based attention on pupillary response as a function of spatial frequency using grating stimuli and filtered natural images by manipulating the participants' attentional state. Furthermore, we aimed to obtain the pupillary response to spatial frequency accurately by reducing the contamination of unintended spatial frequency components in the stimulus by using gratings with a Gabor envelope. We revealed that all stimuli could elicit large pupil constriction for an intermediate range (2-8 c/d) of spatial frequency and that both spatial- and object-based attention modulate the pupillary response function to spatial frequency. These facts may enhance Human Computer Interaction design to use people's attentional state.

The Pupillary Response to the Unknown: Novelty Versus Familiarity

Object recognition is a type of perception that enables observers to recognize familiar shapes and categorize them into real-world identities. In this preregistered study, we aimed to determine whether pupil size changes occur during the perception and recognition of identifiable objects. We compared pupil size changes for familiar objects, nonobjects, and random noise. Nonobjects and noise produced greater pupil dilation than familiar objects. Contrary to previous evidence showing greater pupil dilation to stimuli with more perceptual and affective content, these results indicate a greater pupil dilation to stimuli that are unidentifiable. This is consistent with the relative salience of novelty compared to familiarity at the physiological level driving the pupil response.