Using Machine Learning to Train a Wearable Device for Measuring Students' Cognitive Load during Problem-Solving Activities Based on Electrodermal Activity, Body Temperature, and Heart Rate: Development of a Cognitive Load Tracker for Both Personal and Classroom Use

Smart Classrooms: How Sensors and AI Are Shaping Educational Paradigms

The integration of advanced technologies is revolutionizing classrooms, significantly enhancing their intelligence, interactivity, and personalization. Central to this transformation are sensor technologies, which play pivotal roles. While numerous surveys summarize research progress in classrooms, few studies focus on the integration of sensor and AI technologies in developing smart classrooms. This systematic review classifies sensors used in smart classrooms and explores their current applications from both hardware and software perspectives. It delineates how different sensors enhance educational outcomes and the crucial role AI technologies play. The review highlights how sensor technology improves the physical classroom environment, monitors physiological and behavioral data, and is widely used to boost student engagements, manage attendance, and provide personalized learning experiences. Additionally, it shows that combining sensor software algorithms with AI technology not only enhances the data processing and analysis efficiency but also expands sensor capabilities, enriching their role in smart classrooms. The article also addresses challenges such as data privacy protection, cost, and algorithm optimization associated with emerging sensor technologies, proposing future research directions to advance educational sensor technologies.

PEMOCS: theory derivation of a concept for PErsonalized MOtor-Cognitive exergame training in chronic Stroke-a methodological paper with an application example

Background

Coping with residual cognitive and gait impairments is a prominent unmet need in community-dwelling chronic stroke survivors. Motor-cognitive exergames may be promising to address this unmet need. However, many studies have so far implemented motor-cognitive exergame interventions in an unstructured manner and suitable application protocols remain yet unclear. We, therefore, aimed to summarize existing literature on this topic, and developed a training concept for motor-cognitive exergame interventions in chronic stroke.

Methods

The development of the training concept for personalized motor-cognitive exergame training for stroke (PEMOCS) followed Theory Derivation procedures. This comprised (1.1) a thorough (narrative) literature search on long-term stroke rehabilitation; (1.2) a wider literature search beyond the topic of interest to identify analogies, and to induce creativity; (2) the identification of parent theories; (3) the adoption of suitable content or structure of the main parent theory; and (4) the induction of modifications to adapt it to the new field of interest. We also considered several aspects of the "Framework for Developing and Evaluating Complex Interventions" by the Medical Research Council. Specifically, a feasibility study was conducted, and refining actions based on the findings were performed.

Results

A training concept for improving cognitive functions and gait in community-dwelling chronic stroke survivors should consider the principles for neuroplasticity, (motor) skill learning, and training. We suggest using a step-based exergame training for at least 12 weeks, 2-3 times a week for approximately 45 min. Gentile's Taxonomy for Motor Learning was identified as suitable fundament for the personalized progression and variability rules, and extended by a third cognitive dimension. Concepts and models from related fields inspired further additions and modifications to the concept.

Conclusion

We propose the PEMOCS concept for improving cognitive functioning and gait in community-dwelling chronic stroke survivors, which serves as a guide for structuring and implementing motor-cognitive exergame interventions. Future research should focus on developing objective performance parameters that enable personalized progression independent of the chosen exergame type.

Blink-Related Oscillations Provide Naturalistic Assessments of Brain Function and Cognitive Workload within Complex Real-World Multitasking Environments

BACKGROUND

There is a significant need to monitor human cognitive performance in complex environments, with one example being pilot performance. However, existing assessments largely focus on subjective experiences (e.g., questionnaires) and the evaluation of behavior (e.g., aircraft handling) as surrogates for cognition or utilize brainwave measures which require artificial setups (e.g., simultaneous auditory stimuli) that intrude on the primary tasks. Blink-related oscillations (BROs) are a recently discovered neural phenomenon associated with spontaneous blinking that can be captured without artificial setups and are also modulated by cognitive loading and the external sensory environment-making them ideal for brain function assessment within complex operational settings.

METHODS

Electroencephalography (EEG) data were recorded from eight adult participants (five F, M = 21.1 years) while they completed the Multi-Attribute Task Battery under three different cognitive loading conditions. BRO responses in time and frequency domains were derived from the EEG data, and comparisons of BRO responses across cognitive loading conditions were undertaken. Simultaneously, assessments of blink behavior were also undertaken.

RESULTS

Blink behavior assessments revealed decreasing blink rate with increasing cognitive load (p < 0.001). Prototypical BRO responses were successfully captured in all participants (p < 0.001). BRO responses reflected differences in task-induced cognitive loading in both time and frequency domains (p < 0.05). Additionally, reduced pre-blink theta band desynchronization with increasing cognitive load was also observed (p < 0.05).

CONCLUSION

This study confirms the ability of BRO responses to capture cognitive loading effects as well as preparatory pre-blink cognitive processes in anticipation of the upcoming blink during a complex multitasking situation. These successful results suggest that blink-related neural processing could be a potential avenue for cognitive state evaluation in operational settings-both specialized environments such as cockpits, space exploration, military units, etc. and everyday situations such as driving, athletics, human-machine interactions, etc.-where human cognition needs to be seamlessly monitored and optimized.

The effects of increased mental workload of air traffic controllers on time perception: Behavioral and physiological evidence

Research has shown that timing is modulated by mental workload, making duration judgments a measure of cognitive demand, alongside subjective assessments, and physiological measurements. Yet, it is unclear whether such findings can be extended in less controlled setups. By employing air traffic controllers in a real aviation environment, we tested whether tasks with different levels of cognitive load can affect their timing behavior. Participants completed temporal production, verbal estimation, and passage of time judgments, while actively engaging in real flight control sessions. Subjective assessments of task demands, as well as physiological responses (cardiac and electrodermal activity) were also measured. Accuracy of the produced intervals was measured at two distinct phases of the flight (during low-load cruising vs. high-load landing) and under two different task load manipulations (controlling one vs. two helicopters and speaking in native vs. non-native language). Analysis of interval production accuracy showed that during the high-load landing phase significant overproductions were made, compared to the low-load cruising phase, and landing two helicopters led to greater overproductions compared to landing only one. The duration of the two-helicopter sessions was significantly overestimated compared to the single-helicopter ones, and the passage of time was felt significantly faster. Subjective assessments of workload were positively correlated with the temporal estimations and passage of time judgments, and skin responses were positively correlated with the produced intervals. Overall, our results are consistent with past research, suggesting that mental workload modulates time perception in complex, real-world environments, thus making timing behavior a reliable index of the workload changes.

Predicting Office Workers' Productivity: A Machine Learning Approach Integrating Physiological, Behavioral, and Psychological Indicators

This research pioneers the application of a machine learning framework to predict the perceived productivity of office workers using physiological, behavioral, and psychological features. Two approaches were compared: the baseline model, predicting productivity based on physiological and behavioral characteristics, and the extended model, incorporating predictions of psychological states such as stress, eustress, distress, and mood. Various machine learning models were utilized and compared to assess their predictive accuracy for psychological states and productivity, with XGBoost emerging as the top performer. The extended model outperformed the baseline model, achieving an R2 of 0.60 and a lower MAE of 10.52, compared to the baseline model's R2 of 0.48 and MAE of 16.62. The extended model's feature importance analysis revealed valuable insights into the key predictors of productivity, shedding light on the role of psychological states in the prediction process. Notably, mood and eustress emerged as significant predictors of productivity. Physiological and behavioral features, including skin temperature, electrodermal activity, facial movements, and wrist acceleration, were also identified. Lastly, a comparative analysis revealed that wearable devices (Empatica E4 and H10 Polar) outperformed workstation addons (Kinect camera and computer-usage monitoring application) in predicting productivity, emphasizing the potential utility of wearable devices as an independent tool for assessment of productivity. Implementing the model within smart workstations allows for adaptable environments that boost productivity and overall well-being among office workers.

Decoding Mental Effort in a Quasi-Realistic Scenario: A Feasibility Study on Multimodal Data Fusion and Classification

Humans' performance varies due to the mental resources that are available to successfully pursue a task. To monitor users' current cognitive resources in naturalistic scenarios, it is essential to not only measure demands induced by the task itself but also consider situational and environmental influences. We conducted a multimodal study with 18 participants (nine female, M = 25.9 with SD = 3.8 years). In this study, we recorded respiratory, ocular, cardiac, and brain activity using functional near-infrared spectroscopy (fNIRS) while participants performed an adapted version of the warship commander task with concurrent emotional speech distraction. We tested the feasibility of decoding the experienced mental effort with a multimodal machine learning architecture. The architecture comprised feature engineering, model optimisation, and model selection to combine multimodal measurements in a cross-subject classification. Our approach reduces possible overfitting and reliably distinguishes two different levels of mental effort. These findings contribute to the prediction of different states of mental effort and pave the way toward generalised state monitoring across individuals in realistic applications.

Toward Mental Effort Measurement Using Electrodermal Activity Features

The ability to monitor mental effort during a task using a wearable sensor may improve productivity for both work and study. The use of the electrodermal activity (EDA) signal for tracking mental effort is an emerging area of research. Through analysis of over 92 h of data collected with the Empatica E4 on a single participant across 91 different activities, we report on the efficacy of using EDA features getting at signal intensity, signal dispersion, and peak intensity for prediction of the participant's self-reported mental effort. We implemented the logistic regression algorithm as an interpretable machine learning approach and found that features related to signal intensity and peak intensity were most useful for the prediction of whether the participant was in a self-reported high mental effort state; increased signal and peak intensity were indicative of high mental effort. When cross-validated by activity moderate predictive efficacy was achieved (AUC = 0.63, F1 = 0.63, precision = 0.64, recall = 0.63) which was significantly stronger than using the model bias alone. Predicting mental effort using physiological data is a complex problem, and our findings add to research from other contexts showing that EDA may be a promising physiological indicator to use for sensor-based self-monitoring of mental effort throughout the day. Integration of other physiological features related to heart rate, respiration, and circulation may be necessary to obtain more accurate predictions.

Machine Learning Based Real-Time Diagnosis of Mental Stress Using Photoplethysmography

Mental stress is a natural response to life activities. However, acute and prolonged stress may cause psychological and heart diseases. Heart rate variability (HRV) is considered an indicator of mental stress and physical fitness. The standard way of obtaining HRV is using electrocardiography (ECG) as the time interval between two consecutive R-peaks. ECG signal is collected by attaching electrodes on different locations of the body, which need a proper clinical setup and is costly as well; therefore, it is not feasible to monitor stress with ECG. Photoplethysmography (PPG) is considered an alternative for mental stress detection using pulse rate variability (PRV), the time interval between two successive peaks of PPG. This study aims to diagnose daily life stress using low-cost portable PPG devices instead of lab trials and expensive devices. Data is collected from 27 subjects both in rest and in stressed conditions in daily life routine. Thirty-six time domain, frequency domain, and non-linear features are extracted from PRV. Multiple machine learning classifiers are used to classify these features. Recursive feature elimination, student t-test and genetic algorithm are used to select these features. An accuracy of 72% is achieved using stratified leave out cross-validation using K-Nearest Neighbor, and it increased up to 81% using a genetic algorithm. Once the model is trained with the best features selected with the genetic algorithm, we used the trained weights for the real-time prediction of mental stress. The results show that using a low-cost device; stress can be diagnosed in real life. The proposed method enable the regular monitoring of stress in short time that help to control the occurrence of psychological and cardiovascular diseases.

Current trends and opportunities in the methodology of electrodermal activity measurement

Electrodermal activity (EDA) has been measured in the laboratory since the late 1800s. Although the influence of sudomotor nerve activity and the sympathetic nervous system on EDA is well established, the mechanisms underlying EDA signal generation are not completely understood. Owing to simplicity of instrumentation and modern electronics, these measurements have recently seen a transfer from the laboratory to wearable devices, sparking numerous novel applications while bringing along both challenges and new opportunities. In addition to developments in electronics and miniaturization, current trends in material technology and manufacturing have sparked innovations in electrode technologies, and trends in data science such as machine learning and sensor fusion are expanding the ways that measurement data can be processed and utilized. Although challenges remain for the quality of wearable EDA measurement, ongoing research and developments may shorten the quality gap between wearable EDA and standardized recordings in the laboratory. In this topical review, we provide an overview of the basics of EDA measurement, discuss the challenges and opportunities of wearable EDA, and review recent developments in instrumentation, material technology, signal processing, modeling and data science tools that may advance the field of EDA research and applications over the coming years.

Sensor-Based Prediction of Mental Effort during Learning from Physiological Data: A Longitudinal Case Study

Trackers for activity and physical fitness have become ubiquitous. Although recent work has demonstrated significant relationships between mental effort and physiological data such as skin temperature, heart rate, and electrodermal activity, we have yet to demonstrate their efficacy for the forecasting of mental effort such that a useful mental effort tracker can be developed. Given prior difficulty in extracting relationships between mental effort and physiological responses that are repeatable across individuals, we make the case that fusing self-report measures with physiological data within an internet or smartphone application may provide an effective method for training a useful mental effort tracking system. In this case study, we utilized over 90 h of data from a single participant over the course of a college semester. By fusing the participant’s self-reported mental effort in different activities over the course of the semester with concurrent physiological data collected with the Empatica E4 wearable sensor, we explored questions around how much data were needed to train such a device, and which types of machine-learning algorithms worked best. We concluded that although baseline models such as logistic regression and Markov models provided useful explanatory information on how the student’s physiology changed with mental effort, deep-learning algorithms were able to generate accurate predictions using the first 28 h of data for training. A system that combines long short-term memory and convolutional neural networks is recommended in order to generate smooth predictions while also being able to capture transitions in mental effort when they occur in the individual using the device.

Wearable inertial sensors for human movement analysis: a five-year update

INTRODUCTION

The aim of the present review is to track the evolution of wearable IMUs from their use in supervised laboratory- and ambulatory-based settings to their application for long-term monitoring of human movement in unsupervised naturalistic settings.

AREAS COVERED

Four main emerging areas of application were identified and synthesized, namely, mobile health solutions (specifically, for the assessment of frailty, risk of falls, chronic neurological diseases, and for the monitoring and promotion of active living), occupational ergonomics, rehabilitation and telerehabilitation, and cognitive assessment. Findings from recent scientific literature in each of these areas was synthesized from an applied and/or clinical perspective with the purpose of providing clinical researchers and practitioners with practical guidance on contemporary uses of inertial sensors in applied clinical settings.

EXPERT OPINION

IMU-based wearable devices have undergone a rapid transition from use in laboratory-based clinical practice to unsupervised, applied settings. Successful use of wearable inertial sensing for assessing mobility, motor performance and movement disorders in applied settings will rely also on machine learning algorithms for managing the vast amounts of data generated by these sensors for extracting information that is both clinically relevant and interpretable by practitioners.