1
|
Gilmore J, Nasseri M. Human Activity Recognition Algorithm with Physiological and Inertial Signals Fusion: Photoplethysmography, Electrodermal Activity, and Accelerometry. SENSORS (BASEL, SWITZERLAND) 2024; 24:3005. [PMID: 38793858 PMCID: PMC11124986 DOI: 10.3390/s24103005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/11/2024] [Revised: 04/23/2024] [Accepted: 05/07/2024] [Indexed: 05/26/2024]
Abstract
Inertial signals are the most widely used signals in human activity recognition (HAR) applications, and extensive research has been performed on developing HAR classifiers using accelerometer and gyroscope data. This study aimed to investigate the potential enhancement of HAR models through the fusion of biological signals with inertial signals. The classification of eight common low-, medium-, and high-intensity activities was assessed using machine learning (ML) algorithms, trained on accelerometer (ACC), blood volume pulse (BVP), and electrodermal activity (EDA) data obtained from a wrist-worn sensor. Two types of ML algorithms were employed: a random forest (RF) trained on features; and a pre-trained deep learning (DL) network (ResNet-18) trained on spectrogram images. Evaluation was conducted on both individual activities and more generalized activity groups, based on similar intensity. Results indicated that RF classifiers outperformed corresponding DL classifiers at both individual and grouped levels. However, the fusion of EDA and BVP signals with ACC data improved DL classifier performance compared to a baseline DL model with ACC-only data. The best performance was achieved by a classifier trained on a combination of ACC, EDA, and BVP images, yielding F1-scores of 69 and 87 for individual and grouped activity classifications, respectively. For DL models trained with additional biological signals, almost all individual activity classifications showed improvement (p-value < 0.05). In grouped activity classifications, DL model performance was enhanced for low- and medium-intensity activities. Exploring the classification of two specific activities, ascending/descending stairs and cycling, revealed significantly improved results using a DL model trained on combined ACC, BVP, and EDA spectrogram images (p-value < 0.05).
Collapse
Affiliation(s)
- Justin Gilmore
- Department of Electrical and Computer Engineering, University of Central Florida, Orlando, FL 32816, USA
| | - Mona Nasseri
- School of Engineering, University of North Florida, Jacksonville, FL 32224, USA
| |
Collapse
|
2
|
Hu W, Bian G, Huang L, Pi Y, Zhang X, Zhang X, de Albuquerque VHC, Wu W. Constructing Bodily Emotion Maps Based on High-Density Body Surface Potentials for Psychophysiological Computing. IEEE J Biomed Health Inform 2024; 28:2500-2511. [PMID: 38051611 DOI: 10.1109/jbhi.2023.3339382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/07/2023]
Abstract
Emotion is a complex physiological and psychological activity, accompanied by subjective physiological sensations and objective physiological changes. The body sensation map describes the changes in body sensation associated with emotion in a topographic manner, but it relies on subjective evaluations from participants. Physiological signals are a more reliable measure of emotion, but most research focuses on the central nervous system, neglecting the importance of the peripheral nervous system. In this study, a body surface potential mapping (BSPM) system was constructed, and an experiment was designed to induce emotions and obtain high-density body surface potential information under negative and non-negative emotions. Then, by constructing and analyzing the functional connectivity network of BSPs, the high-density electrophysiological characteristics are obtained and visualized as bodily emotion maps. The results showed that the functional connectivity network of BSPs under negative emotions had denser connections, and emotion maps based on local clustering coefficient (LCC) are consistent with BSMs under negative emotions. in addition, our features can classify negative and non-negative emotions with the highest classification accuracy of 80.77%. In conclusion, this study constructs an emotion map based on high-density BSPs, which offers a novel approach to psychophysiological computing.
Collapse
|
3
|
P SK, Agastinose Ronickom JF. Optimal Electrodermal Activity Segment for Enhanced Emotion Recognition Using Spectrogram-Based Feature Extraction and Machine Learning. Int J Neural Syst 2024; 34:2450027. [PMID: 38511233 DOI: 10.1142/s0129065724500278] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/22/2024]
Abstract
In clinical and scientific research on emotion recognition using physiological signals, selecting the appropriate segment is of utmost importance for enhanced results. In our study, we optimized the electrodermal activity (EDA) segment for an emotion recognition system. Initially, we obtained EDA signals from two publicly available datasets: the Continuously annotated signals of emotion (CASE) and Wearable stress and affect detection (WESAD) for 4-class dimensional and three-class categorical emotional classification, respectively. These signals were pre-processed, and decomposed into phasic signals using the 'convex optimization to EDA' method. Further, the phasic signals were segmented into two equal parts, each subsequently segmented into five nonoverlapping windows. Spectrograms were then generated using short-time Fourier transform and Mel-frequency cepstrum for each window, from which we extracted 85 features. We built four machine learning models for the first part, second part, and whole phasic signals to investigate their performance in emotion recognition. In the CASE dataset, we achieved the highest multi-class accuracy of 62.54% using the whole phasic and 61.75% with the second part phasic signals. Conversely, the WESAD dataset demonstrated superior performance in three-class emotions classification, attaining an accuracy of 96.44% for both whole phasic and second part phasic segments. As a result, the second part of EDA is strongly recommended for optimal outcomes.
Collapse
Affiliation(s)
- Sriram Kumar P
- School of Biomedical Engineering, Indian Institute of Technology (BHU) Varanasi, Uttar Pradesh 221005, India
| | | |
Collapse
|
4
|
Ranaut A, Khandnor P, Chand T. Identifying autism using EEG: unleashing the power of feature selection and machine learning. Biomed Phys Eng Express 2024; 10:035013. [PMID: 38457850 DOI: 10.1088/2057-1976/ad31fb] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Accepted: 03/08/2024] [Indexed: 03/10/2024]
Abstract
Autism Spectrum Disorder (ASD) is a neurodevelopmental condition that is characterized by communication barriers, societal disengagement, and monotonous actions. Currently, the diagnosis of ASD is made by experts through a subjective and time-consuming qualitative behavioural examination using internationally recognized descriptive standards. In this paper, we present an EEG-based three-phase novel approach comprising 29 autistic subjects and 30 neurotypical people. In the first phase, preprocessing of data is performed from which we derived one continuous dataset and four condition-based datasets to determine the role of each dataset in the identification of autism from neurotypical people. In the second phase, time-domain and morphological features were extracted and four different feature selection techniques were applied. In the last phase, five-fold cross-validation is used to evaluate six different machine learning models based on the performance metrics and computational efficiency. The neural network outperformed when trained with maximum relevance and minimum redundancy (MRMR) algorithm on the continuous dataset with 98.10% validation accuracy and 0.9994 area under the curve (AUC) value for model validation, and 98.43% testing accuracy and AUC test value of 0.9998. The decision tree overall performed the second best in terms of computational efficiency and performance accuracy. The results indicate that EEG-based machine learning models have the potential for ASD identification from neurotypical people with a more objective and reliable method.
Collapse
Affiliation(s)
- Anamika Ranaut
- Department of Computer Science and Engineering, Punjab Engineering College, Chandigarh, India
| | - Padmavati Khandnor
- Department of Computer Science and Engineering, Punjab Engineering College, Chandigarh, India
| | - Trilok Chand
- Department of Computer Science and Engineering, Punjab Engineering College, Chandigarh, India
| |
Collapse
|
5
|
Pan L, Tang Z, Wang S, Song A. Cross-subject emotion recognition using hierarchical feature optimization and support vector machine with multi-kernel collaboration. Physiol Meas 2023; 44:125006. [PMID: 38029444 DOI: 10.1088/1361-6579/ad10c6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 11/29/2023] [Indexed: 12/01/2023]
Abstract
Objective. Due to individual differences, it is greatly challenging to realize the multiple types of emotion identification across subjects.Approach. In this research, a hierarchical feature optimization method is proposed in order to represent emotional states effectively based on peripheral physiological signals. Firstly, sparse learning combined with binary search is employed to achieve feature selection of single signals. Then an improved fast correlation-based filter is proposed to implement fusion optimization of multi-channel signal features. Aiming at overcoming the limitations of the support vector machine (SVM), which uses a single kernel function to make decisions, the multi-kernel function collaboration strategy is proposed to improve the classification performance of SVM.Main results. The effectiveness of the proposed method is verified on the DEAP dataset. Experimental results show that the proposed method presents a competitive performance for four cross-subject types of emotion identification with an accuracy of 84% (group 1) and 85.07% (group 2). Significance. The proposed model with hierarchical feature optimization and SVM with multi-kernel function collaboration demonstrates superior emotion recognition accuracy compared to state-of-the-art techniques. In addition, the analysis based on DEAP dataset composition characteristics presents a novel perspective to explore the emotion recognition issue more objectively and comprehensively.
Collapse
Affiliation(s)
- Lizheng Pan
- School of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, People's Republic of China
| | - Ziqin Tang
- School of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, People's Republic of China
| | - Shunchao Wang
- School of Mechanical Engineering and Rail Transit, Changzhou University, Changzhou 213164, People's Republic of China
| | - Aiguo Song
- School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| |
Collapse
|
6
|
Sánchez-Reolid R, López de la Rosa F, Sánchez-Reolid D, López MT, Fernández-Caballero A. Machine Learning Techniques for Arousal Classification from Electrodermal Activity: A Systematic Review. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22228886. [PMID: 36433482 PMCID: PMC9695360 DOI: 10.3390/s22228886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 11/14/2022] [Accepted: 11/14/2022] [Indexed: 05/14/2023]
Abstract
This article introduces a systematic review on arousal classification based on electrodermal activity (EDA) and machine learning (ML). From a first set of 284 articles searched for in six scientific databases, fifty-nine were finally selected according to various criteria established. The systematic review has made it possible to analyse all the steps to which the EDA signals are subjected: acquisition, pre-processing, processing and feature extraction. Finally, all ML techniques applied to the features of these signals for arousal classification have been studied. It has been found that support vector machines and artificial neural networks stand out within the supervised learning methods given their high-performance values. In contrast, it has been shown that unsupervised learning is not present in the detection of arousal through EDA. This systematic review concludes that the use of EDA for the detection of arousal is widely spread, with particularly good results in classification with the ML methods found.
Collapse
Affiliation(s)
- Roberto Sánchez-Reolid
- Departamento de Sistemas Informáticos, Universidad de Castilla-La Mancha, 02071 Albacete, Spain
- Neurocognition and Emotion Unit, Instituto de Investigación en Informática, 02071 Albacete, Spain
| | | | - Daniel Sánchez-Reolid
- Neurocognition and Emotion Unit, Instituto de Investigación en Informática, 02071 Albacete, Spain
| | - María T. López
- Departamento de Sistemas Informáticos, Universidad de Castilla-La Mancha, 02071 Albacete, Spain
- Neurocognition and Emotion Unit, Instituto de Investigación en Informática, 02071 Albacete, Spain
| | - Antonio Fernández-Caballero
- Departamento de Sistemas Informáticos, Universidad de Castilla-La Mancha, 02071 Albacete, Spain
- Neurocognition and Emotion Unit, Instituto de Investigación en Informática, 02071 Albacete, Spain
- CIBERSAM-ISCIII (Biomedical Research Networking Center in Mental Health, Instituto de Salud Carlos III), 28016 Madrid, Spain
- Correspondence:
| |
Collapse
|
7
|
Zhu M, Men Q, Ho ESL, Leung H, Shum HPH. A Two-stream Convolutional Network for Musculoskeletal and Neurological Disorders Prediction. J Med Syst 2022; 46:76. [PMID: 36201114 PMCID: PMC9537228 DOI: 10.1007/s10916-022-01857-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 08/17/2022] [Indexed: 11/24/2022]
Abstract
Musculoskeletal and neurological disorders are the most common causes of walking problems among older people, and they often lead to diminished quality of life. Analyzing walking motion data manually requires trained professionals and the evaluations may not always be objective. To facilitate early diagnosis, recent deep learning-based methods have shown promising results for automated analysis, which can discover patterns that have not been found in traditional machine learning methods. We observe that existing work mostly applies deep learning on individual joint features such as the time series of joint positions. Due to the challenge of discovering inter-joint features such as the distance between feet (i.e. the stride width) from generally smaller-scale medical datasets, these methods usually perform sub-optimally. As a result, we propose a solution that explicitly takes both individual joint features and inter-joint features as input, relieving the system from the need of discovering more complicated features from small data. Due to the distinctive nature of the two types of features, we introduce a two-stream framework, with one stream learning from the time series of joint position and the other from the time series of relative joint displacement. We further develop a mid-layer fusion module to combine the discovered patterns in these two streams for diagnosis, which results in a complementary representation of the data for better prediction performance. We validate our system with a benchmark dataset of 3D skeleton motion that involves 45 patients with musculoskeletal and neurological disorders, and achieve a prediction accuracy of 95.56%, outperforming state-of-the-art methods.
Collapse
Affiliation(s)
- Manli Zhu
- Department of Computer and Information Sciences, Northumbria University, Newcastle upon Tyne, UK
| | - Qianhui Men
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Edmond S L Ho
- School of Computing Science, University of Glasgow, Glasgow, UK
| | - Howard Leung
- Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong
| | - Hubert P H Shum
- Department of Computer Science, Durham University, Durham, UK.
| |
Collapse
|
8
|
Saffaryazdi N, Wasim ST, Dileep K, Nia AF, Nanayakkara S, Broadbent E, Billinghurst M. Using Facial Micro-Expressions in Combination With EEG and Physiological Signals for Emotion Recognition. Front Psychol 2022; 13:864047. [PMID: 35837650 PMCID: PMC9275379 DOI: 10.3389/fpsyg.2022.864047] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2022] [Accepted: 05/30/2022] [Indexed: 11/13/2022] Open
Abstract
Emotions are multimodal processes that play a crucial role in our everyday lives. Recognizing emotions is becoming more critical in a wide range of application domains such as healthcare, education, human-computer interaction, Virtual Reality, intelligent agents, entertainment, and more. Facial macro-expressions or intense facial expressions are the most common modalities in recognizing emotional states. However, since facial expressions can be voluntarily controlled, they may not accurately represent emotional states. Earlier studies have shown that facial micro-expressions are more reliable than facial macro-expressions for revealing emotions. They are subtle, involuntary movements responding to external stimuli that cannot be controlled. This paper proposes using facial micro-expressions combined with brain and physiological signals to more reliably detect underlying emotions. We describe our models for measuring arousal and valence levels from a combination of facial micro-expressions, Electroencephalography (EEG) signals, galvanic skin responses (GSR), and Photoplethysmography (PPG) signals. We then evaluate our model using the DEAP dataset and our own dataset based on a subject-independent approach. Lastly, we discuss our results, the limitations of our work, and how these limitations could be overcome. We also discuss future directions for using facial micro-expressions and physiological signals in emotion recognition.
Collapse
Affiliation(s)
- Nastaran Saffaryazdi
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Syed Talal Wasim
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Kuldeep Dileep
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Alireza Farrokhi Nia
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Suranga Nanayakkara
- Augmented Human Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| | - Elizabeth Broadbent
- Department of Psychological Medicine, The University of Auckland, Auckland, New Zealand
| | - Mark Billinghurst
- Empathic Computing Laboratory, Auckland Bioengineering Institute, The University of Auckland, Auckland, New Zealand
| |
Collapse
|
9
|
A universal emotion recognition method based on feature priority evaluation and classifier reinforcement. INT J MACH LEARN CYB 2022. [DOI: 10.1007/s13042-022-01590-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
10
|
One-dimensional convolutional neural networks for low/high arousal classification from electrodermal activity. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103203] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
11
|
Deserno TM, Dugas M, Löbe M, Stausberg J. A Topical Collection on ICT for Health Science Research - EFMI Special Topic Conference. J Med Syst 2021; 45:70. [PMID: 34002277 PMCID: PMC8128354 DOI: 10.1007/s10916-021-01739-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Grants] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Affiliation(s)
- Thomas M. Deserno
- Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School, Braunschweig, Germany
| | - Martin Dugas
- Institute of Medical Informatics, University Hospital Heidelberg, Heidelberg, Germany
| | - Matthias Löbe
- Institute for Medical Informatics, Statistics and Epidemiology (IMISE), University of Leipzig, Leipzig, Germany
| | - Jürgen Stausberg
- Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), Faculty of Medicine, University Duisburg-Essen, Essen, Germany
| |
Collapse
|