Using eye-tracking and EEG to study the mental processing demands during learning of text-picture combinations

Task-irrelevant decorative pictures increase cognitive load during text processing but have no effects on learning or working memory performance: an EEG and eye-tracking study

Decorative pictures (DP) are often used in multimedia task materials and are commonly considered so-called seductive details as they are commonly not task-relevant. Typically, DP result in mixed effects on behavioral performance measures. The current study focused on the effects of DP on the cognitive load during text reading and working memory task performance. The theta and alpha frequency band power of the electroencephalogram (EEG) and pupil dilation served as proxies of cognitive load. The number of fixations, mean fixation durations, and the number of transitions served as proxies of the attentional focus. For both, text reading and n-back working memory tasks, the presence and congruency of DP were manipulated in four task conditions. DP did neither affect behavioral performance nor subjective ratings of emotional-motivational factors. However, in both tasks, DP increased the cognitive load as revealed by the EEG alpha frequency band power and (at least to some extent) by subjective effort ratings. Notably, the EEG alpha frequency band power was a quite reliable and sensitive proxy of cognitive load. Analyzing the EEG data stimulus-locked and fixation-related, the EEG alpha frequency band power revealed a difference in global and local cognitive load. In sum, the current study underlines the feasibility and use of EEG for multimedia research, especially when combined with eye-tracking.

Seeing beyond words: Visualizing autism spectrum disorder biomarker insights

Objective

This study employs bibliometric and visual analysis to elucidate global research trends in Autism Spectrum Disorder (ASD) biomarkers, identify critical research focal points, and discuss the potential integration of diverse biomarker modalities for precise ASD assessment.

Methods

A comprehensive bibliometric analysis was conducted using data from the Web of Science Core Collection database until December 31, 2022. Visualization tools, including R, VOSviewer, CiteSpace, and gCLUTO, were utilized to examine collaborative networks, co-citation patterns, and keyword associations among countries, institutions, authors, journals, documents, and keywords.

Results

ASD biomarker research emerged in 2004, accumulating a corpus of 4348 documents by December 31, 2022. The United States, with 1574 publications and an H-index of 213, emerged as the most prolific and influential country. The University of California, Davis, contributed significantly with 346 publications and an H-index of 69, making it the leading institution. Concerning journals, the Journal of Autism and Developmental Disorders, Autism Research, and PLOS ONE were the top three publishers of ASD biomarker-related articles among a total of 1140 academic journals. Co-citation and keyword analyses revealed research hotspots in genetics, imaging, oxidative stress, neuroinflammation, gut microbiota, and eye tracking. Emerging topics included "DNA methylation," "eye tracking," "metabolomics," and "resting-state fMRI."

Conclusion

The field of ASD biomarker research is dynamically evolving. Future endeavors should prioritize individual stratification, methodological standardization, the harmonious integration of biomarker modalities, and longitudinal studies to advance the precision of ASD diagnosis and treatment.

Connecting the dots: Exploring brain connectivity during responsibility recognition in construction contract negotiations

Despite recent advancements in monitoring brain activity, causal relationships within the brain during responsibility identification in construction contracts remain unexplored. We aimed to understand the neural mechanisms involved in the cognitive components and their interactions related to contract text reading by delving into the brain mechanisms of contract responsibility identification. This study investigated students' brain connectivity using electroencephalography (EEG) data during a text-based contract responsibility-identification task. It employed an adaptive directed transfer function based on Granger causality to simulate directed and time-varying information flow in observed brain activity. We evaluated the EEG records of 18 participants under two reading conditions (involving or not involving contractor responsibility). During responsibility identification, the most substantial information exchange occurs in the somatosensory area of the brain. The results revealed a "top-down" cortical mechanism for responsibility identification, with the left parietal-occipital area (PO3) as the central hub promoting connectivity structures. These findings indicate that the perceptual processing of contract responsibility texts is associated with higher visual learning and memory quality. Contracts without contractor-responsibility clauses resulted in more substantial information flow output in the frontal cortex and consumed more cognitive resources. Our findings advance the understanding of cognitive processes involved in contract responsibility identification, providing a framework for investigating causal relationships within the brain and novel insights into cortical mechanisms. By identifying the neural basis of responsibility identification, stakeholders can develop effective training programs for negotiators and enhance their ability to interpret and implement construction contracts.

Effects and prediction of cognitive load on encoding model of brain response to auditory and linguistic stimuli in educational multimedia

Multimedia is extensively used for educational purposes. However, certain types of multimedia lack proper design, which could impose a cognitive load on the user. Therefore, it is essential to predict cognitive load and understand how it impairs brain functioning. Participants watched a version of educational multimedia that applied Mayer's principles, followed by a version that did not. Meanwhile, their electroencephalography (EEG) was recorded. Subsequently, they participated in a post-test and completed a self-reported cognitive load questionnaire. The audio envelope and word frequency were extracted from the multimedia, and the temporal response functions (TRFs) were obtained using a linear encoding model. We observed that the behavioral data are different between the two groups and the TRFs of the two multimedia versions were different. We saw changes in the amplitude and latencies of both early and late components. In addition, correlations were found between behavioral data and the amplitude and latencies of TRF components. Cognitive load decreased participants' attention to the multimedia, and semantic processing of words also occurred with a delay and smaller amplitude. Hence, encoding models provide insights into the temporal and spatial mapping of the cognitive load activity, which could help us detect and reduce cognitive load in potential environments such as educational multimedia or simulators for different purposes.

Neurophysiological measures and correlates of cognitive load in attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder (ASD) and dyslexia: A scoping review and research recommendations

Working memory is integral to a range of critical cognitive functions such as reasoning and decision-making. Although alterations in working memory have been observed in neurodivergent populations, there has been no review mapping how cognitive load is measured in common neurodevelopmental conditions such as attention-deficit/hyperactivity disorder (ADHD), autism spectrum disorder (ASD) and dyslexia. This scoping review explores the neurophysiological measures used to study cognitive load in these specific populations. Our findings highlight that electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) are the most frequently used methods, with a limited number of studies employing functional near-infrared spectroscopy (fNIRs), magnetoencephalography (MEG) or eye-tracking. Notably, eye-related measures are less commonly used, despite their prominence in cognitive load research among neurotypical individuals. The review also highlights potential correlates of cognitive load, such as neural oscillations in the theta and alpha ranges for EEG studies, blood oxygenation level-dependent (BOLD) responses in lateral and medial frontal brain regions for fMRI and fNIRS studies and eye-related measures such as pupil dilation and blink rate. Finally, critical issues for future studies are discussed, including the technical challenges associated with multimodal approaches, the possible impact of atypical features on cognitive load measures and balancing data richness with participant well-being. These insights contribute to a more nuanced understanding of cognitive load measurement in neurodivergent populations and point to important methodological considerations for future neuroscientific research in this area.

Gamification of an n-back working memory task - is it worth the effort? An EEG and eye-tracking study

Gamification of cognitive tasks might positively affect emotional-motivational factors (emotional design perspective) or negatively affect cognitive factors like working memory load (minimalistic design perspective). The current study examined the effects of gamification in a spatial n-back working memory task on task performance, task load (i.e., working memory load and effort), and subjective task experience. Task load was assessed by the physiological process measures pupil dilation and EEG theta (4 - 6Hz) and alpha (8 - 13Hz) frequency band power. Gamification was achieved by elements of emotional design (i.e., the visual screen design using, e.g., color, cartoon figures as n-back stimuli, and a narrative embedding of the task). While EEG and eye-tracking were recorded, participants conducted gamified and non-gamified 1-back and 2-back load levels. The gamification resulted in positive effects on subjective task experience and affect. Despite these effects, gamification did not affect task performance and task load. However, exploratory analyses revealed increased EEG theta power at right-parietal electrodes for gamified task versions compared to non-gamified ones. Potentially, this effect might indicate participants' increased effort or concentration in the gamified n-back task. In line with an emotional design perspective, gamification positively altered subjective task experience and affect without hampering task performance and therefore justify the extra effort of implementing game elements.

Clear Aligners and Smart Eye Tracking Technology as a New Communication Strategy between Ethical and Legal Issues

Smart eye-tracking technology (SEET) that determines visual attention using smartphones can be used to determine the aesthetic perception of different types of clear aligners. Its value as a communication and comprehension tool, in addition to the ethical and legal concerns which it entails, can be assessed. One hundred subjects (50 F, 50 M; age range 15-70) were equally distributed in non-orthodontic (A) and orthodontic (B) groups. A smartphone-based SEET app assessed their knowledge of and opinions on aligners. Subjects evaluated images of smiles not wearing aligners, with/without attachments and with straight/scalloped gingival margins, as a guided calibration step which formed the image control group. Subsequently, the subjects rated the same smiles, this time wearing aligners (experimental images group). Questionnaire data and average values for each group of patients, and images relating to fixation times and overall star scores, were analyzed using these tests: chi-square, t-test, Mann-Whitney U, Spearman's rho, and Wilcoxon (p < 0.05). One-way ANOVA and related post-hoc tests were also applied. Orthodontic patients were found to be better informed than non-orthodontic patients. Aesthetic perception could be swayed by several factors. Attachments scored lower in aesthetic evaluation. Lips distracted attention from attachments and improved evaluations. Attachment-free aligners were better rated overall. A more thorough understanding as to the opinions, expectations and aesthetic perception of aligners can improve communication with patients. Mobile SEET is remarkably promising, although it does require a careful medicolegal risk-benefit assessments for responsible and professional use.

Visual preference of plant features in different living environments using eye tracking and EEG

Plants play a very important role in landscape construction. In order to explore whether different living environment will affect people's preference for the structural features of plant organs, this study examined 26 villagers and 33 college students as the participants, and pictures of leaves, flowers and fruits of plants as the stimulus to conduct eye-tracking and EEG detection experiments. We found that eye movement indicators can explain people's visual preferences, but they are unable to find differences in preferences between groups. EEG indicators can make up for this deficiency, which further reveals the difference in psychological and physiological responses between the two groups when viewing stimuli. The final results show that the villagers and the students liked leaves best, preferring aciculiform and leathery leaves; solitary, purple and capitulum flowers; and medium-sized, spathulate, black and pear fruits. In addition, it was found that the overall attention of the villagers when watching stimuli was far lower than that of the students, but the degree of meditation was higher. With regard to eye movement and EEG, the total duration of fixations is highly positively correlated with the number of fixations, and the average pupil size has a weak negative correlation with attention. On the contrary, the average duration of fixations has a weak positive correlation with meditation. Generally speaking, we believe that Photinia×fraseri, Metasequoia glyptostroboides, Photinia serratifolia, Koelreuteria bipinnata and Cunninghamia lanceolata are superior landscape building plants in rural areas and on campuses; Pinus thunbergii, Myrica rubra, Camellia japonica and other plants with obvious features and bright colours are also the first choice in rural landscapes; and Yulania biondii, Cercis chinensis, Hibiscus mutabilis and other plants with simple structures are the first choice in campus landscapes. This study is of great significance for selecting plants for landscape construction and management according to different environments and local conditions.

The comparison of electroencephalography power and event related potential in success and failure during multitask game

The first game-based treatment for children with attention-deficit hyperactivity disorder (ADHD) was approved by the United States Food and Drug Administration (FDA) in 2020. This game was developed for use at home as part of everyday training and can be used along with one’s usual training plan. In this game, two tasks are performed in parallel: (1) a perceptual discrimination targeting task (response and not response and avoiding responding to sudden pop-up targets) and (2) a sensory-motor navigation task (players continuously adjust their location to interact with or avoid positional targets). However, the brain activity of people playing this game was not examined, and the immersive environment (3D virtual world) was not considered. Therefore, we aimed to develop a system to investigate brain activity using electroencephalography (EEG) during multitask gameplay in virtual reality (VR). In this experiment, we focused on the difference between the success and failure of the Go/No-Go task in a multitask game. We created a color discrimination task and a target tracking task in VR. The content of this game task was designed using previous multitask training. EEG and event data were recorded. Using event data, we can analyze the data in detail. We divided the trial types (Go and No-Go) and results (success and failure). We then compared the success and failure of each task. In the Go trial, the relative theta power in success at Fz was significantly higher than that of failure. However, no difference in power was observed in the No-Go trial. On the other hand, theta power was no different between success and failure in the other task. These results of the Go trial suggest that the participants were attentive to processing both tasks. Thus, it is possible that theta power in the frontal area 1 s before stimulation could predict the success or failure of the Go trial. On the other hand, the results of the No-Go trial may be due to the low number of No-Go failure trials and the fact that stimulus oversight is one of the factors for success.

Effects of decorative pictures on mental processing demands and learning: An EEG and eye-tracking study

Across two experiments, the effects of presenting decorative pictures (DP) alongside texts were assessed with respect to the mental processing demands during reading, learning outcomes, and subjective task experience. DP were thought to function as so-called seductive details. Eye-tracking and the electroencephalogram (EEG) were recorded during reading. Pupil dilation and the EEG theta (4-8 Hz) and alpha (8-13 Hz) frequency band power served as measures of the mental processing demands. Texts on features and habitats of animals served as learning materials. Working memory (WM) load was manipulated as an additional factor during reading. Neither the additional WM load nor DP had significant effects on learning outcomes, albeit in both experiments the increased WM load affected the physiological measures as expected. DP with a few different motifs generally were not very seductive (Experiment 1). DP with more diverse motifs resulted in increased mental processing demands as indicated by the EEG alpha frequency band power (Experiment 2). Subjective task experience was positively affected by the DP present in Experiment 2 as indicated by higher subjective ratings of interestingness, comprehensibility, emotionality, and aesthetical pleasantness when DP were present. In sum, the study indicates that DP with diverse motifs may function as seductive details, increasing mental processing demands without detrimental effects on learning outcomes.

Enhancing the feasibility of cognitive load recognition in remote learning using physiological measures and an adaptive feature recalibration convolutional neural network

The precise assessment of cognitive load during a learning phase is an important pathway to improving students’ learning efficiency and performance. Physiological measures make it possible to continuously monitor learners’ cognitive load in remote learning during the COVID-19 outbreak. However, maintaining a good balance between performance and computational cost is still a major challenge in advancing cognitive load recognition technology to real-world applications. This paper introduced an adaptive feature recalibration (AFR) convolutional neural network to overcome this challenge by capturing the most discriminative physiological features (EEG and eye-tracking). The results revealed that the optimal average classification accuracy of the feature combination obtained by the AFR method reached 95.56% with only 60 feature dimensions. Additionally, compared with the best result of the conventional correlation-based feature selection (CFS) method, the introduced AFR algorithm achieved higher accuracy and cheaper computational cost, as well as a 2.06% improvement in accuracy and a 51.21% reduction in feature dimension, which is more in line with the requirements of low delay and real-time performance in practical BCI applications.

Assessment of instantaneous cognitive load imposed by educational multimedia using electroencephalography signals

The use of multimedia learning is increasing in modern education. On the other hand, it is crucial to design multimedia contents that impose an optimal amount of cognitive load, which leads to efficient learning. Objective assessment of instantaneous cognitive load plays a critical role in educational design quality evaluation. Electroencephalography (EEG) has been considered a potential candidate for cognitive load assessment among neurophysiological methods. In this study, we experiment to collect EEG signals during a multimedia learning task and then build a model for instantaneous cognitive load measurement. In the experiment, we designed four educational multimedia in two categories to impose different levels of cognitive load by intentionally applying/violating Mayer's multimedia design principles. Thirty university students with homogenous English language proficiency participated in our experiment. We divided them randomly into two groups, and each watched a version of the multimedia followed by a recall test task and filling out a NASA-TLX questionnaire. EEG signals are collected during these tasks. To construct the load assessment model, at first, power spectral density (PSD) based features are extracted from EEG signals. Using the minimum redundancy - maximum relevance (MRMR) feature selection approach, the best features are selected. In this way, the selected features consist of only about 12% of the total number of features. In the next step, we propose a scoring model using a support vector machine (SVM) for instantaneous cognitive load assessment in 3s segments of multimedia. Our experiments indicate that the selected feature set can classify the instantaneous cognitive load with an accuracy of 84.5 ± 2.1%. The findings of this study indicate that EEG signals can be used as an appropriate tool for measuring the cognitive load introduced by educational videos. This can be help instructional designers to develop more effective content.

Backward and forward neck tilt affects perceptual bias when interpreting ambiguous figures

The relationships between posture and perception have already been investigated in several studies. However, it is still unclear how perceptual bias and experiential contexts of human perception affect observers’ perception when posture is changed. In this study, we hypothesized that a change in the perceptual probability caused by perceptual bias also depends on posture. In order to verify this hypothesis, we used the Necker cube with two types of appearance, from above and below, although the input is constant, and investigated the change of the probability of perceptual content. Specifically, we asked observers their perception of the appearance of the Necker cube placed at any of the five angles in the space of virtual reality. There were two patterns of neck movement, vertical and horizontal. During the experiment, pupil diameter, one of the cognitive indices, was also measured. Results showed that during the condition of looking down vertically, the probability of the viewing-from-above perception of the Necker cube was significantly greater than during the condition of looking up. Interestingly, the pupillary results were also consistent with the probability of the perception. These results indicate that perception was modulated by the posture of the neck and suggest that neck posture is incorporated into ecological constraints.

Study on selection of native greening plants based on eye-tracking technology

The selection of native greening plants to improve rural greening technology is crucial for enriching methods of building rural plant landscapes. However, there are few studies from the perspective of visual preference using quantitative methods. By using eye-tracking technology, this study studies students in the Central South University of Forestry and Technology and villagers in Changkou Village, Fujian Province, employing pictures of plant organs—leaves, flowers, and fruits—as stimulating materials to analyze five indicators: the total duration of fixations, the number of fixations, average duration of fixations, average pupil size and average amplitude of saccades. A number of findings came from this research First, people visually prefer leaves, followed by flowers and fruits. In terms of species, Photinia × fraseri, Metasequoia glyptostroboides, Photinia serratifolia, Cunninghamia lanceolata and Koelreuteria bipinnata have higher overall preference. Families such as Malvaceae, Fabaceae, Araliaceae, Myricaceae and Cupressaceae have stronger visual attraction than others. Second, there are distinct differences in the preference of shapes and textures of leaves: aciculiform, strip, cordiform, sector and jacket-shape are more attractive; leather-like leaves have a higher visual preference than paper-like leaves; different colors and whether leaves are cracked or not have little effect on leaf observation. Third, the preference for flowers with different inflorescence and colors is significant. Capitulum, cymes and panicles are more attractive; purple is the most preferred color, followed by white, yellow and red. Finally, there are significant differences in preferences for fruit characteristics, with medium-sized fruits and black fruits preferred, while kidney-shaped and spoon-shaped fruits are considered more attractive. Pomes, pods, samaras, and berries have received relatively more attention.

Impacts of Color Coding on Programming Learning in Multimedia Learning: Moving Toward a Multimodal Methodology

In the present study, we tested the effectiveness of color coding on the programming learning of students who were learning from video lectures. Effectiveness was measured using multimodal physiological measures, combining eye tracking and electroencephalography (EEG). Using a between-subjects design, 42 university students were randomly assigned to two video lecture conditions (color-coded vs. grayscale). The participants' eye tracking and EEG signals were recorded while watching the assigned video, and their learning performance was subsequently assessed. The results showed that the color-coded design was more beneficial than the grayscale design, as indicated by smaller pupil diameter, shorter fixation duration, higher EEG theta and alpha band power, lower EEG cognitive load, and better learning performance. The present findings have practical implications for designing slide-based programming learning video lectures; slides should highlight the format of the program code using color coding.

The Validation and Further Development of the Multidimensional Cognitive Load Scale for Physical and Online Lectures (MCLS-POL)

Cognitive load theory (CLT) has been widely used to help understand the process of learning and to design teaching interventions. The Cognitive Load Scale (CLS) developed by Leppink and colleagues has emerged as one of the most validated and widely used self-report measures of intrinsic load (IL), extraneous load (EL), and germane load (GL). In this paper we investigated an expansion of the CLS by using a multidimensional conceptualization of the EL construct that is relevant for physical and online teaching environments. The Multidimensional Cognitive Load Scale for Physical and Online Lectures (MCLS-POL) goes beyond the CLS's operationalization of EL by expanding the EL component which originally included factors related to instructions/explanations with sub-dimensions including EL stemming from noises, and EL stemming from both media and devices within the environment. Through three studies, we investigated the reliability, and internal and external validity of the MCLS-POL using the Partial Credit Model, Confirmatory Factor Analysis, and differences between students either attending a lecture physically or online (Study 2 and 3). The results of Study 1 (N = 250) provide initial evidence for the validity and reliability of the MCLS-POL within a higher education sample, but also highlighted several potential improvements which could be made to the measure. These changes were made before re-evaluating the validity and reliability of the measure in a new sample of higher education psychology students (N = 140, Study 2), and psychological testing students (N = 119, Study 3). Together the studies provide evidence for a multidimensional conceptualization cognitive load and provide evidence of the validity, reliability, and sensitivity of the MCLS-POL and provide suggestions for future research directions.