1
|
Wu Y, Jiang X, Guo Y, Zhu H, Dai C, Chen W. Physiological measurements for driving drowsiness: A comparative study of multi-modality feature fusion and selection. Comput Biol Med 2023; 167:107590. [PMID: 37897962 DOI: 10.1016/j.compbiomed.2023.107590] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2023] [Revised: 09/18/2023] [Accepted: 10/17/2023] [Indexed: 10/30/2023]
Abstract
A large number of traffic accidents were caused by drowsiness while driving. In-vehicle alert system based on physiological signals was one of the most promising solutions to monitor driving fatigue. However, different physiological modalities can be used, and many relative studies compared different modalities without considering the implementation feasibility of portable or wearable devices. Moreover, evaluations of each modality in previous studies were based on inconsistent choices of fatigue label and signal features, making it hard to compare the results of different studies. Therefore, the modality comparison and fusion for continuous drowsiness estimation while driving was still unclear. This work sought to comprehensively compare widely-used physiological modalities, including forehead electroencephalogram (EEG), electrooculogram (EOG), R-R intervals (RRI) and breath, in a hardware setting feasible for portable or wearable devices to monitor driving fatigue. Moreover, a more general conclusion on modality comparison and fusion was reached based on the regression of features or their combinations and the awake-to-drowsy transition. Finally, the feature subset of fused modalities was produced by feature selection method, to select the optimal feature combination and reduce computation consumption. Considering practical feasibility, the most effective combination with the highest correlation coefficient was using forehead EEG or EOG, along with RRI and RRI-derived breath. If more comfort and convenience was required, the combination of RRI and RRI-derived breath was also promising.
Collapse
Affiliation(s)
- Yonglin Wu
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China; School of Information Science and Technology, Fudan University, Shanghai, China
| | - Xinyu Jiang
- School of Information Science and Technology, Fudan University, Shanghai, China
| | - Yao Guo
- School of Information Science and Technology, Fudan University, Shanghai, China
| | - Hangyu Zhu
- School of Information Science and Technology, Fudan University, Shanghai, China
| | - Chenyun Dai
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, China.
| | - Wei Chen
- School of Information Science and Technology, Fudan University, Shanghai, China.
| |
Collapse
|
2
|
Zeng Z, Tao L, Zhu H, Zhu Y, Meng L, Fan J, Chen C, Chen W. A Robust Gaze Estimation Approach via Exploring Relevant Electrooculogram Features and Optimal Electrodes Placements. IEEE JOURNAL OF TRANSLATIONAL ENGINEERING IN HEALTH AND MEDICINE 2023; 12:56-65. [PMID: 38088999 PMCID: PMC10712680 DOI: 10.1109/jtehm.2023.3320713] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 07/16/2023] [Accepted: 09/21/2023] [Indexed: 12/18/2023]
Abstract
Gaze estimation, as a technique that reflects individual attention, can be used for disability assistance and assisting physicians in diagnosing diseases such as autism spectrum disorder (ASD), Parkinson's disease, and attention deficit hyperactivity disorder (ADHD). Various techniques have been proposed for gaze estimation and achieved high resolution. Among these approaches, electrooculography (EOG)-based gaze estimation, as an economical and effective method, offers a promising solution for practical applications. OBJECTIVE In this paper, we systematically investigated the possible EOG electrode locations which are spatially distributed around the orbital cavity. Afterward, quantities of informative features to characterize physiological information of eye movement from the temporal-spectral domain are extracted from the seven differential channels. METHODS AND PROCEDURES To select the optimum channels and relevant features, and eliminate irrelevant information, a heuristical search algorithm (i.e., forward stepwise strategy) is applied. Subsequently, a comparative analysis of the impacts of electrode placement and feature contributions on gaze estimation is evaluated via 6 classic models with 18 subjects. RESULTS Experimental results showed that the promising performance was achieved both in the Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) within a wide gaze that ranges from -50° to +50°. The MAE and RMSE can be improved to 2.80° and 3.74° ultimately, while only using 10 features extracted from 2 channels. Compared with the prevailing EOG-based techniques, the performance improvement of MAE and RMSE range from 0.70° to 5.48° and 0.66° to 5.42°, respectively. CONCLUSION We proposed a robust EOG-based gaze estimation approach by systematically investigating the optimal channel/feature combination. The experimental results indicated not only the superiority of the proposed approach but also its potential for clinical application. Clinical and translational impact statement: Accurate gaze estimation is a key step for assisting disabilities and accurate diagnosis of various diseases including ASD, Parkinson's disease, and ADHD. The proposed approach can accurately estimate the points of gaze via EOG signals, and thus has the potential for various related medical applications.
Collapse
Affiliation(s)
- Zheng Zeng
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
| | - Linkai Tao
- Department of Industrial DesignEindhoven University of Technology5600 MBEindhovenThe Netherlands
| | - Hangyu Zhu
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
| | - Yunfeng Zhu
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
| | - Long Meng
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
| | - Jiahao Fan
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
| | - Chen Chen
- Human Phenome Institute, Fudan UniversityShanghai201203China
| | - Wei Chen
- Center for Intelligent Medical Electronics, School of Information Science and TechnologyFudan UniversityShanghai200433China
- Human Phenome Institute, Fudan UniversityShanghai201203China
| |
Collapse
|
3
|
Amri Bin Suhaimi MS, Matsushita K, Kitamura T, Laksono PW, Sasaki M. Object Grasp Control of a 3D Robot Arm by Combining EOG Gaze Estimation and Camera-Based Object Recognition. Biomimetics (Basel) 2023; 8:biomimetics8020208. [PMID: 37218794 DOI: 10.3390/biomimetics8020208] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2023] [Revised: 05/11/2023] [Accepted: 05/12/2023] [Indexed: 05/24/2023] Open
Abstract
The purpose of this paper is to quickly and stably achieve grasping objects with a 3D robot arm controlled by electrooculography (EOG) signals. A EOG signal is a biological signal generated when the eyeballs move, leading to gaze estimation. In conventional research, gaze estimation has been used to control a 3D robot arm for welfare purposes. However, it is known that the EOG signal loses some of the eye movement information when it travels through the skin, resulting in errors in EOG gaze estimation. Thus, EOG gaze estimation is difficult to point out the object accurately, and the object may not be appropriately grasped. Therefore, developing a methodology to compensate, for the lost information and increase spatial accuracy is important. This paper aims to realize highly accurate object grasping with a robot arm by combining EMG gaze estimation and the object recognition of camera image processing. The system consists of a robot arm, top and side cameras, a display showing the camera images, and an EOG measurement analyzer. The user manipulates the robot arm through the camera images, which can be switched, and the EOG gaze estimation can specify the object. In the beginning, the user gazes at the screen's center position and then moves their eyes to gaze at the object to be grasped. After that, the proposed system recognizes the object in the camera image via image processing and grasps it using the object centroid. The object selection is based on the object centroid closest to the estimated gaze position within a certain distance (threshold), thus enabling highly accurate object grasping. The observed size of the object on the screen can differ depending on the camera installation and the screen display state. Therefore, it is crucial to set the distance threshold from the object centroid for object selection. The first experiment is conducted to clarify the distance error of the EOG gaze estimation in the proposed system configuration. As a result, it is confirmed that the range of the distance error is 1.8-3.0 cm. The second experiment is conducted to evaluate the performance of the object grasping by setting two thresholds from the first experimental results: the medium distance error value of 2 cm and the maximum distance error value of 3 cm. As a result, it is found that the grasping speed of the 3 cm threshold is 27% faster than that of the 2 cm threshold due to more stable object selection.
Collapse
Affiliation(s)
- Muhammad Syaiful Amri Bin Suhaimi
- Faculty of Information and Communication Technology, Universiti Tunku Abdul Rahman, Jalan Universiti, Bandar Barat, Kampar 31900, Malaysia
- Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan
| | - Kojiro Matsushita
- Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan
- Intelligent Production Technology Research & Development Center for Aerospace (IPTeCA), Tokai National Higher Education and Research System, Gifu 501-1193, Japan
| | - Takahide Kitamura
- Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan
- Intelligent Production Technology Research & Development Center for Aerospace (IPTeCA), Tokai National Higher Education and Research System, Gifu 501-1193, Japan
| | - Pringgo Widyo Laksono
- Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan
- Industrial Engineering, Faculty of Engineering, Universitas Sebelas Maret, Surakarta 57126, Indonesia
| | - Minoru Sasaki
- Graduate School of Engineering, Gifu University, 1-1 Yanagido, Gifu 501-1193, Japan
- Intelligent Production Technology Research & Development Center for Aerospace (IPTeCA), Tokai National Higher Education and Research System, Gifu 501-1193, Japan
| |
Collapse
|
4
|
Hernández Pérez SN, Pérez Reynoso FD, Gutiérrez CAG, Cosío León MDLÁ, Ortega Palacios R. EOG Signal Classification with Wavelet and Supervised Learning Algorithms KNN, SVM and DT. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23094553. [PMID: 37177757 PMCID: PMC10181598 DOI: 10.3390/s23094553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 04/21/2023] [Accepted: 05/02/2023] [Indexed: 05/15/2023]
Abstract
The work carried out in this paper consists of the classification of the physiological signal generated by eye movement called Electrooculography (EOG). The human eye performs simultaneous movements, when focusing on an object, generating a potential change in origin between the retinal epithelium and the cornea and modeling the eyeball as a dipole with a positive and negative hemisphere. Supervised learning algorithms were implemented to classify five eye movements; left, right, down, up and blink. Wavelet Transform was used to obtain information in the frequency domain characterizing the EOG signal with a bandwidth of 0.5 to 50 Hz; training results were obtained with the implementation of K-Nearest Neighbor (KNN) 69.4%, a Support Vector Machine (SVM) of 76.9% and Decision Tree (DT) 60.5%, checking the accuracy through the Jaccard index and other metrics such as the confusion matrix and ROC (Receiver Operating Characteristic) curve. As a result, the best classifier for this application was the SVM with Jaccard Index.
Collapse
Affiliation(s)
- Sandy Nohemy Hernández Pérez
- Master's Degree in Information and Communications Technologies, Universidad Politécnica de Pachuca (UPP), Zempoala 43830, Mexico
| | - Francisco David Pérez Reynoso
- Center for Research, Innovation and Technological Development UVM (CIIDETEC-UVM), Universidad del Valle de México, Querétaro 76230, Mexico
| | - Carlos Alberto González Gutiérrez
- Center for Research, Innovation and Technological Development UVM (CIIDETEC-UVM), Universidad del Valle de México, Querétaro 76230, Mexico
| | | | - Rocío Ortega Palacios
- Master's Degree in Information and Communications Technologies, Universidad Politécnica de Pachuca (UPP), Zempoala 43830, Mexico
| |
Collapse
|
5
|
EOG-Based Human–Computer Interface: 2000–2020 Review. SENSORS 2022; 22:s22134914. [PMID: 35808414 PMCID: PMC9269776 DOI: 10.3390/s22134914] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 06/23/2022] [Accepted: 06/25/2022] [Indexed: 11/28/2022]
Abstract
Electro-oculography (EOG)-based brain–computer interface (BCI) is a relevant technology influencing physical medicine, daily life, gaming and even the aeronautics field. EOG-based BCI systems record activity related to users’ intention, perception and motor decisions. It converts the bio-physiological signals into commands for external hardware, and it executes the operation expected by the user through the output device. EOG signal is used for identifying and classifying eye movements through active or passive interaction. Both types of interaction have the potential for controlling the output device by performing the user’s communication with the environment. In the aeronautical field, investigations of EOG-BCI systems are being explored as a relevant tool to replace the manual command and as a communicative tool dedicated to accelerating the user’s intention. This paper reviews the last two decades of EOG-based BCI studies and provides a structured design space with a large set of representative papers. Our purpose is to introduce the existing BCI systems based on EOG signals and to inspire the design of new ones. First, we highlight the basic components of EOG-based BCI studies, including EOG signal acquisition, EOG device particularity, extracted features, translation algorithms, and interaction commands. Second, we provide an overview of EOG-based BCI applications in the real and virtual environment along with the aeronautical application. We conclude with a discussion of the actual limits of EOG devices regarding existing systems. Finally, we provide suggestions to gain insight for future design inquiries.
Collapse
|
6
|
Wang X, Xiao Y, Deng F, Chen Y, Zhang H. Eye-Movement-Controlled Wheelchair Based on Flexible Hydrogel Biosensor and WT-SVM. BIOSENSORS 2021; 11:198. [PMID: 34208524 PMCID: PMC8234407 DOI: 10.3390/bios11060198] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/12/2021] [Revised: 05/31/2021] [Accepted: 06/07/2021] [Indexed: 11/17/2022]
Abstract
To assist patients with restricted mobility to control wheelchair freely, this paper presents an eye-movement-controlled wheelchair prototype based on a flexible hydrogel biosensor and Wavelet Transform-Support Vector Machine (WT-SVM) algorithm. Considering the poor deformability and biocompatibility of rigid metal electrodes, we propose a flexible hydrogel biosensor made of conductive HPC/PVA (Hydroxypropyl cellulose/Polyvinyl alcohol) hydrogel and flexible PDMS (Polydimethylsiloxane) substrate. The proposed biosensor is affixed to the wheelchair user's forehead to collect electrooculogram (EOG) and strain signals, which are the basis to recognize eye movements. The low Young's modulus (286 KPa) and exceptional breathability (18 g m-2 h-1 of water vapor transmission rate) of the biosensor ensures a conformal and unobtrusive adhesion between it and the epidermis. To improve the recognition accuracy of eye movements (straight, upward, downward, left, and right), the WT-SVM algorithm is introduced to classify EOG and strain signals according to different features (amplitude, duration, interval). The average recognition accuracy reaches 96.3%, thus the wheelchair can be manipulated precisely.
Collapse
Affiliation(s)
| | | | - Fangming Deng
- School of Electrical and Automation Engineering, East China Jiaotong University, Nanchang 330013, China; (X.W.); (Y.X.); (Y.C.); (H.Z.)
| | | | | |
Collapse
|
7
|
Kaur A. Wheelchair control for disabled patients using EMG/EOG based human machine interface: a review. J Med Eng Technol 2020; 45:61-74. [PMID: 33302770 DOI: 10.1080/03091902.2020.1853838] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/11/2023]
Abstract
The human-machine interface (HMI) and bio-signals have been used to control rehabilitation equipment and improve the lives of people with severe disabilities. This research depicts a review of electromyogram (EMG) or electrooculogram (EOG) signal-based control system for driving the wheelchair for disabled. For a paralysed person, EOG is one of the most useful signals that help to successfully communicate with the environment by using eye movements. In the case of amputation, the selection of muscles according to the distribution of power and frequency highly contributes to the specific motion of a wheelchair. Taking into account the day-to-day activities of persons with disabilities, both technologies are being used to design EMG or EOG based wheelchairs. This review paper examines a total of 70 EMG studies and 25 EOG studies published from 2000 to 2019. In addition, this paper covers current technologies used in wheelchair systems for signal capture, filtering, characterisation, and classification, including control commands such as left and right turns, forward and reverse motion, acceleration, deceleration, and wheelchair stop.
Collapse
Affiliation(s)
- Amanpreet Kaur
- Department of Electronics and Communication Engineering, Thapar Institute of Engineering and Technology, Patiala, India
| |
Collapse
|
8
|
Martínez-Cerveró J, Ardali MK, Jaramillo-Gonzalez A, Wu S, Tonin A, Birbaumer N, Chaudhary U. Open Software/Hardware Platform for Human-Computer Interface Based on Electrooculography (EOG) Signal Classification. SENSORS 2020; 20:s20092443. [PMID: 32344820 PMCID: PMC7248971 DOI: 10.3390/s20092443] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/21/2020] [Revised: 04/21/2020] [Accepted: 04/23/2020] [Indexed: 12/28/2022]
Abstract
Electrooculography (EOG) signals have been widely used in Human-Computer Interfaces (HCI). The HCI systems proposed in the literature make use of self-designed or closed environments, which restrict the number of potential users and applications. Here, we present a system for classifying four directions of eye movements employing EOG signals. The system is based on open source ecosystems, the Raspberry Pi single-board computer, the OpenBCI biosignal acquisition device, and an open-source python library. The designed system provides a cheap, compact, and easy to carry system that can be replicated or modified. We used Maximum, Minimum, and Median trial values as features to create a Support Vector Machine (SVM) classifier. A mean of 90% accuracy was obtained from 7 out of 10 subjects for online classification of Up, Down, Left, and Right movements. This classification system can be used as an input for an HCI, i.e., for assisted communication in paralyzed people.
Collapse
Affiliation(s)
- Jayro Martínez-Cerveró
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
| | - Majid Khalili Ardali
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
| | - Andres Jaramillo-Gonzalez
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
| | - Shizhe Wu
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
| | - Alessandro Tonin
- Wyss-Center for Bio- and Neuro-Engineering, Chemin des Mines 9, Ch 1202 Geneva, Switzerland
| | - Niels Birbaumer
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
| | - Ujwal Chaudhary
- Institute of Medical Psychology and Behavioural Neurobiology, University of Tübingen, Silcherstraße 5, 72076 Tübingen, Germany
- Wyss-Center for Bio- and Neuro-Engineering, Chemin des Mines 9, Ch 1202 Geneva, Switzerland
- Correspondence:
| |
Collapse
|
9
|
Choudhari AM, Porwal P, Jonnalagedda V, Mériaudeau F. An Electrooculography based Human Machine Interface for wheelchair control. Biocybern Biomed Eng 2019. [DOI: 10.1016/j.bbe.2019.04.002] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
10
|
Bio-potentials for smart control applications. HEALTH AND TECHNOLOGY 2019. [DOI: 10.1007/s12553-019-00314-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|
11
|
Santana R, Mendiburu A, Lozano JA. Multi-view classification of psychiatric conditions based on saccades. Appl Soft Comput 2015. [DOI: 10.1016/j.asoc.2015.02.038] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|