1
|
Li X, Kang Y, Chen W, Liu F, Jiao Y, Luo Y. Recognizing the situation awareness of forklift operators based on EEG techniques in a field experiment. Front Neurosci 2024; 18:1323190. [PMID: 38445257 PMCID: PMC10912158 DOI: 10.3389/fnins.2024.1323190] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Accepted: 01/30/2024] [Indexed: 03/07/2024] Open
Abstract
Lack of situation awareness (SA) is the primary cause of human errors when operating forklifts, so determining the SA level of the forklift operator is crucial to the safety of forklift operations. An EEG recognition approach of forklift operator SA in actual settings was presented in order to address the issues with invasiveness, subjectivity, and intermittency of existing measuring methods. In this paper, we conducted a field experiment that mimicked a typical forklift operation scenario to verify the differences in EEG states of forklift operators with different SA levels and investigate the correlation of multi-band combination features of each brain region of forklift operators with SA. Based on the sensitive EEG combination indexes, Support Vector Mechanism was used to construct a forklift operator SA recognition model. The results revealed that there were differences between forklift operators with high and low SA in the θ, α, and β frequency bands in zones F, C, P, and O; combined EEG indicators θ/β, (α + θ)/(α + β), and θ/(α + β) in zones F, P, and C were significantly correlated with SA; the recognition accuracy of the model reached 88.64% in the case of combined EEG indicators of zones C & F & P as input. It could provide a reference for SA measurement, contributing to the improvement of SA.
Collapse
Affiliation(s)
- Xin Li
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
- COSCO SHIPPING Heavy Industry Co., Ltd., Shanghai, China
| | - Yutao Kang
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Weijiong Chen
- Merchant Marine College, Shanghai Maritime University, Shanghai, China
| | - Feng Liu
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Yu Jiao
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Yabin Luo
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| |
Collapse
|
2
|
Alessa FM, Alhaag MH, Al-harkan IM, Ramadan MZ, Alqahtani FM. A Neurophysiological Evaluation of Cognitive Load during Augmented Reality Interactions in Various Industrial Maintenance and Assembly Tasks. SENSORS (BASEL, SWITZERLAND) 2023; 23:7698. [PMID: 37765755 PMCID: PMC10536580 DOI: 10.3390/s23187698] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/25/2023] [Revised: 09/03/2023] [Accepted: 09/04/2023] [Indexed: 09/29/2023]
Abstract
Augmented reality (AR) has been shown to improve productivity in industry, but its adverse effects (e.g., headaches, eye strain, nausea, and mental workload) on users warrant further investigation. The objective of this study is to investigate the effects of different instruction methods (i.e., HoloLens AR-based and paper-based instructions) and task complexity (low and high-demanding tasks) on cognitive workloads and performance. Twenty-eight healthy males with a mean age of 32.12 (SD 2.45) years were recruited in this study and were randomly divided into two groups. The first group performed the experiment using AR-based instruction, and the second group used paper-based instruction. Performance was measured using total task time (TTT). The cognitive workload was measured using the power of electroencephalograph (EEG) features and the NASA task load index (NASA TLX). The results showed that using AR instructions resulted in a reduction in maintenance times and an increase in mental workload compared to paper instructions, particularly for the more demanding tasks. With AR instruction, 0.45% and 14.94% less time was spent on low- and high-demand tasks, respectively, as compared to paper instructions. According to the EEG features, employing AR to guide employees during highly demanding maintenance tasks increased information processing, which could be linked with an increased germane cognitive load. Increased germane cognitive load means participants can better facilitate long-term knowledge and skill acquisition. These results suggested that AR is superior and recommended for highly demanding maintenance tasks since it speeds up maintenance times and increases the possibility that information is stored in long-term memory and encrypted for recalls.
Collapse
Affiliation(s)
- Faisal M. Alessa
- Department of Industrial Engineering, College of Engineering, King Saud University, P.O. Box 800, Riyadh 11421, Saudi Arabia
| | | | | | | | | |
Collapse
|
3
|
Achanccaray D, Sumioka H. Analysis of Physiological Response of Attention and Stress States in Teleoperation Performance of Social Tasks. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38083262 DOI: 10.1109/embc40787.2023.10340007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Some studies addressed monitoring mental states by physiological responses analysis in robots' teleoperation in traditional applications such as inspection and exploration; however, no study analyzed the physiological response during teleoperated social tasks to the best of our knowledge. We analyzed the physiological response of attention and stress mental states by computing the correlation between multimodal biomarkers and performance, pleasure-arousal scale, and workload. Physiological data were recorded during simulated teleoperated social tasks to induce mental states, such as normal, attention, and stress. The results showed that task performance and workload subscales achieved moderate correlations with some multimodal biomarkers. The correlations depended on the induced state. The cognitive workload was related to brain biomarkers of attention in the frontal and frontal-central regions. These regions were close to the frontopolar region, which is commonly reported in attentional studies. Thus, some multimodal biomarkers of attention and stress mental states could monitor or predict metrics related to the performance in teleoperation of social tasks.
Collapse
|
4
|
Petrovic M, Vukicevic AM, Djapan M, Peulic A, Jovicic M, Mijailovic N, Milovanovic P, Grajic M, Savkovic M, Caiazzo C, Isailovic V, Macuzic I, Jovanovic K. Experimental Analysis of Handcart Pushing and Pulling Safety in an Industrial Environment by Using IoT Force and EMG Sensors: Relationship with Operators' Psychological Status and Pain Syndromes. SENSORS (BASEL, SWITZERLAND) 2022; 22:7467. [PMID: 36236564 PMCID: PMC9572849 DOI: 10.3390/s22197467] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/27/2022] [Revised: 09/27/2022] [Accepted: 09/28/2022] [Indexed: 06/16/2023]
Abstract
Non-ergonomic execution of repetitive physical tasks represents a major cause of work-related musculoskeletal disorders (WMSD). This study was focused on the pushing and pulling (P&P) of an industrial handcart (which is a generic physical task present across many industries), with the aim to investigate the dependence of P&P execution on the operators' psychological status and the presence of pain syndromes of the upper limbs and spine. The developed acquisition system integrated two three-axis force sensors (placed on the left and right arm) and six electromyography (EMG) electrodes (placed on the chest, back, and hand flexor muscles). The conducted experiment involved two groups of participants (with and without increased psychological scores and pain syndromes). Ten force parameters (for both left and right side), one EMG parameter (for three different muscles, both left and right side), and two time-domain parameters were extracted from the acquired signals. Data analysis showed intergroup differences in the examined parameters, especially in force integral values and EMG mean absolute values. To the best of our knowledge, this is the first study that evaluated the composite effects of pain syndromes, spine mobility, and psychological status of the participants on the execution of P&P tasks-concluding that they have a significant impact on the P&P task execution and potentially on the risk of WMSD. The future work will be directed towards the development of a personalized risk assessment system by considering more muscle groups, supplementary data derived from operators' poses (extracted with computer vision algorithms), and cognitive parameters (extracted with EEG sensors).
Collapse
Affiliation(s)
- Milos Petrovic
- School of Electrical Engineering, University of Belgrade, Bulevar kralja Aleksandra 73, 11000 Belgrade, Serbia
| | - Arso M. Vukicevic
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Marko Djapan
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Aleksandar Peulic
- Faculty of Geography, University of Belgrade, Studentski trg 3, 11000 Belgrade, Serbia
| | - Milos Jovicic
- The AI Institute of Serbia, Fruškogorska 1, 21000 Novi Sad, Serbia
| | - Nikola Mijailovic
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Petar Milovanovic
- Center of Bone Biology, Faculty of Medicine, University of Belgrade, Dr Subotica 4/2, 11000 Belgrade, Serbia
- Laboratory of Bone Biology and Bioanthropology, Faculty of Medicine, Institute of Anatomy, University of Belgrade, Dr Subotica 4/2, 11000 Belgrade, Serbia
| | - Mirko Grajic
- Center for Physical Medicine and Rehabilitation, Faculty of Medicine, University Clinical Center of Serbia, University of Belgrade, Pasterova 2, 11000 Belgrade, Serbia
| | - Marija Savkovic
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Carlo Caiazzo
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Velibor Isailovic
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Ivan Macuzic
- Faculty of Engineering, University of Kragujevac, Sestre Janjic 6, 34000 Kragujevac, Serbia
| | - Kosta Jovanovic
- School of Electrical Engineering, University of Belgrade, Bulevar kralja Aleksandra 73, 11000 Belgrade, Serbia
| |
Collapse
|
5
|
Wang G, Yin Z, Zhao M, Tian Y, Sun Z. Identification of human mental workload levels in a language comprehension task with imbalance neurophysiological data. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 224:107011. [PMID: 35863122 DOI: 10.1016/j.cmpb.2022.107011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/09/2021] [Revised: 05/23/2022] [Accepted: 07/06/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Operator's capability for accurately comprehending verbal commands is critically important to maintain the performance of human-machine interaction. It can be evaluated by human mental workload measured with electroencephalography (EEG). However, the time duration of different workload conditions within a task session is unequal due to varied psychophysiological processes across individuals. It leads to data imbalance of the EEG for training workload classifiers. METHODS In this study, we propose an EEG feature oversampling technique, Gaussian-SMOTE based feature ensemble (GSMOTE-FE), for workload recognition with imbalanced classes. First, artificial EEG instances are drawn from a Gaussian distribution in the margin between the minority and majority workload classes. Tomek links are detected as clues to remove redundant feature vectors. Then, we embed a feature selection module based on the GINI importance while an ensemble classifier committee with bootstrap aggregating is used to further enhance classification performance. RESULTS We validate the GSMOTE-FE framework based on an experiment that simulates operators to understand the correct meaning of the instructions in the Chinese language. Participants' EEG signals and reaction time data were both recorded to validate the proposed workload classifier. Workload classification accuracy and Macro-F1 values are 0.6553 and 0.5862, respectively. Corresponding G-mean and AUC achieve at 0.5757 and 0.5958, respectively. CONCLUSIONS The performance of the GSMOTE-FE is demonstrated to be comparable with the advanced oversampling techniques. The workload classifier has the capability to indicate low and high levels of the task demand of the Chinese language understanding task.
Collapse
Affiliation(s)
- Guangying Wang
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, PR China
| | - Zhong Yin
- Engineering Research Center of Optical Instrument and System, Ministry of Education, Shanghai Key Lab of Modern Optical System, University of Shanghai for Science and Technology, Shanghai, 200093, PR China; School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, PR China.
| | - Mengyuan Zhao
- College of Foreign Languages, University of Shanghai for Science and Technology, Shanghai, 200093, PR China
| | - Ying Tian
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, PR China
| | - Zhanquan Sun
- School of Optical-Electrical and Computer Engineering, University of Shanghai for Science and Technology, Shanghai, 200093, PR China
| |
Collapse
|
6
|
Bhavsar P. "Context-Dependent Cognitive Workload Monitoring using Pupillometry for Control Room Operators to Prevent Overload". IISE Trans Occup Ergon Hum Factors 2022; 10:91-103. [PMID: 35575073 DOI: 10.1080/24725838.2022.2076177] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
Occupational ApplicationModernization of process plants has replaced traditional machine control with sophisticated computerized control, with the work required of control room operators changing from predominantly physical to cognitive. As a result, excessive cognitive workload during an abnormal condition is not an uncommon situation. Almost 70% of accidents in process industries are the result of human error, which is largely modulated by high cognitive workload. A context-dependent methodology using pupillometry was adopted in this study for workload assessment. We found that workload has a significant effect on task performance and subsequent success in handling an abnormal condition. Incorporating contextual information along with pupillometry-based measures was useful to explore the dynamics of variations in workload in depth. In addition, the proposed methodology is a more reliable method for a dynamic environment in which the situation evolves with operator interactions with the process.Technical Abstract:Background: Operator workload monitoring has become an important aspect of system engineering due to ever increasing cognitive demands in the control room. Higher cognitive workload, beyond capability, is directly associated with human error in plant operations. Even a small increase in workload during peak demand may result in human error, which can endanger the safety of a process plant.Purpose: Subjective methods are of limited use due to the requirement for near real-time evaluation of cognitive workload. Applications such as process plant control room require the operator to reiterate and re-represent the situation based on the contextual cues from environment. Hence, our objective was to develop and test a methodology that incorporates contextual information in workload measurement using pupillometry.Method: Participants (n = 44) performed simulated process control tasks. Pupil size was recorded using an eye-tracking device. Recorded pupil size was analyzed based on a methodology that incorporates contextual information from the human-machine interface. Events that present new information about the current state of the process were categorized as critical events. Pupil size variation was adopted to provide an estimate of cognitive workload between consecutive critical events.Results: Cognitive workload was associated with process behavior and operator actions. An increase in cognitive workload for specific events was observed when participant actions deteriorated process conditions. We also observed that overall task performance was associated with the frequency of certain events and corresponding variations in pupillary behavior.Conclusion: Incorporating contextual information provides more detailed insights into the dynamics of variations in cognitive workload. The mental representation of a current situation may vary depending on expertise level. The proposed methodology assesses such dynamic mental representations in the form of frequency of critical events and corresponding changes in the cognitive workload that ultimately determines the likelihood of an operator successfully completing a task.
Collapse
Affiliation(s)
- Punitkumar Bhavsar
- Electronics and Communication Department, Visvesvaraya National Institute of Technology, Nagpur
| |
Collapse
|
7
|
Srinivasan B, Iqbal MU, Shahab MA, Srinivasan R. Review of Virtual Reality (VR) Applications To Enhance Chemical Safety: From Students to Plant Operators. ACS CHEMICAL HEALTH & SAFETY 2022. [DOI: 10.1021/acs.chas.2c00006] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
Affiliation(s)
- Babji Srinivasan
- Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India
- American Express Lab for Data Analytics, Risk & Technology, Indian Institute of Technology Madras, Chennai 600036, India
| | - Mohd Umair Iqbal
- Department of Chemical Engineering, Indian Institute of Technology Gandhinagar, Gandhinagar 382355, India
| | - Mohammed Aatif Shahab
- Department of Applied Mechanics, Indian Institute of Technology Madras, Chennai 600036, India
| | - Rajagopalan Srinivasan
- Department of Chemical Engineering, Indian Institute of Technology Madras, Chennai 600036, India
- American Express Lab for Data Analytics, Risk & Technology, Indian Institute of Technology Madras, Chennai 600036, India
| |
Collapse
|
8
|
HMM-based models of control room operator's cognition during process abnormalities. 2. Application to operator training. J Loss Prev Process Ind 2022. [DOI: 10.1016/j.jlp.2022.104749] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
9
|
Shahab MA, Iqbal MU, Srinivasan B, Srinivasan R. HMM-based models of control room operator's cognition during process abnormalities. 1. Formalism and model identification. J Loss Prev Process Ind 2022. [DOI: 10.1016/j.jlp.2022.104748] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
|
10
|
Gupta SS, Taori TJ, Ladekar MY, Manthalkar RR, Gajre SS, Joshi YV. Classification of cross task cognitive workload using deep recurrent network with modelling of temporal dynamics. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.103070] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023]
|