1
|
Kizhevska E, Šparemblek K, Luštrek M. Protocol of the study for predicting empathy during VR sessions using sensor data and machine learning. PLoS One 2024; 19:e0307385. [PMID: 39024217 PMCID: PMC11257359 DOI: 10.1371/journal.pone.0307385] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Accepted: 07/02/2024] [Indexed: 07/20/2024] Open
Abstract
Virtual reality (VR) technology is often referred to as the 'ultimate empathy machine' due to its capability to immerse users in alternate perspectives and environments beyond their immediate physical reality. In this study, participants will be immersed in 3-dimensional 360° VR videos where actors express different emotions (sadness, happiness, anger, and anxiousness). The primary objective is to investigate the potential relationship between participants' empathy levels and the changes in their physiological attributes. The empathy levels will be self-reported with questionnaires, and physiological attributes will be measured using different sensors. The main outcome of the study will be a machine learning model to predict a person's empathy level based on their physiological responses while watching VR videos. Despite the existence of established methodologies and metrics in research and clinical domains, our aim is to contribute to addressing the gap of a universally accepted "gold standard" for assessing empathy. Additionally, we expect to deepen our understanding of the relationship between different emotions and psychological attributes, gender differences in empathy, and the impact of narrative context on empathic responses.
Collapse
Affiliation(s)
- Emilija Kizhevska
- Department of Intelligent Systems, Jožef Stefan Institute, Ljubljana, Slovenia
- Jožef Stefan International Postgraduate School (IPS), Ljubljana, Slovenia
| | - Kristina Šparemblek
- Department of Intelligent Systems, Jožef Stefan Institute, Ljubljana, Slovenia
| | - Mitja Luštrek
- Department of Intelligent Systems, Jožef Stefan Institute, Ljubljana, Slovenia
- Jožef Stefan International Postgraduate School (IPS), Ljubljana, Slovenia
| |
Collapse
|
2
|
Chin WJ, Kwan BH, Lim WY, Tee YK, Darmaraju S, Liu H, Goh CH. A Novel Respiratory Rate Estimation Algorithm from Photoplethysmogram Using Deep Learning Model. Diagnostics (Basel) 2024; 14:284. [PMID: 38337800 PMCID: PMC10855057 DOI: 10.3390/diagnostics14030284] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Revised: 12/29/2023] [Accepted: 01/26/2024] [Indexed: 02/12/2024] Open
Abstract
Respiratory rate (RR) is a critical vital sign that can provide valuable insights into various medical conditions, including pneumonia. Unfortunately, manual RR counting is often unreliable and discontinuous. Current RR estimation algorithms either lack the necessary accuracy or demand extensive window sizes. In response to these challenges, this study introduces a novel method for continuously estimating RR from photoplethysmogram (PPG) with a reduced window size and lower processing requirements. To evaluate and compare classical and deep learning algorithms, this study leverages the BIDMC and CapnoBase datasets, employing the Respiratory Rate Estimation (RRest) toolbox. The optimal classical techniques combination on the BIDMC datasets achieves a mean absolute error (MAE) of 1.9 breaths/min. Additionally, the developed neural network model utilises convolutional and long short-term memory layers to estimate RR effectively. The best-performing model, with a 50% train-test split and a window size of 7 s, achieves an MAE of 2 breaths/min. Furthermore, compared to other deep learning algorithms with window sizes of 16, 32, and 64 s, this study's model demonstrates superior performance with a smaller window size. The study suggests that further research into more precise signal processing techniques may enhance RR estimation from PPG signals.
Collapse
Affiliation(s)
- Wee Jian Chin
- Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia; (W.J.C.); (B.-H.K.); (Y.K.T.); (S.D.)
- Centre for Healthcare Science and Technology, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia
| | - Ban-Hoe Kwan
- Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia; (W.J.C.); (B.-H.K.); (Y.K.T.); (S.D.)
- Centre for Healthcare Science and Technology, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia
| | - Wei Yin Lim
- Electrical and Computer Systems Engineering, School of Engineering and Advanced Engineering Platform, Monash University Malaysia, Bandar Sunway 47500, Selangor, Malaysia;
| | - Yee Kai Tee
- Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia; (W.J.C.); (B.-H.K.); (Y.K.T.); (S.D.)
- Centre for Healthcare Science and Technology, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia
| | - Shalini Darmaraju
- Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia; (W.J.C.); (B.-H.K.); (Y.K.T.); (S.D.)
| | - Haipeng Liu
- Centre for Intelligent Healthcare, Coventry University, Coventry CV1 5RW, UK;
| | - Choon-Hian Goh
- Department of Mechatronics and Biomedical Engineering, Lee Kong Chian Faculty of Engineering and Science, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia; (W.J.C.); (B.-H.K.); (Y.K.T.); (S.D.)
- Centre for Healthcare Science and Technology, Universiti Tunku Abdul Rahman, Kajang 43000, Selangor, Malaysia
| |
Collapse
|
3
|
Gnacek M, Quintero L, Mavridou I, Balaguer-Ballester E, Kostoulas T, Nduka C, Seiss E. AVDOS-VR: Affective Video Database with Physiological Signals and Continuous Ratings Collected Remotely in VR. Sci Data 2024; 11:132. [PMID: 38272936 PMCID: PMC10810824 DOI: 10.1038/s41597-024-02953-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/14/2023] [Accepted: 01/11/2024] [Indexed: 01/27/2024] Open
Abstract
Investigating emotions relies on pre-validated stimuli to evaluate induced responses through subjective self-ratings and physiological changes. The creation of precise affect models necessitates extensive datasets. While datasets related to pictures, words, and sounds are abundant, those associated with videos are comparatively scarce. To overcome this challenge, we present the first virtual reality (VR) database with continuous self-ratings and physiological measures, including facial EMG. Videos were rated online using a head-mounted VR device (HMD) with attached emteqPRO mask and a cinema VR environment in remote home and laboratory settings with minimal setup requirements. This led to an affective video database with continuous valence and arousal self-rating measures and physiological responses (PPG, facial-EMG (7x), IMU). The AVDOS-VR database includes data from 37 participants who watched 30 randomly ordered videos (10 positive, neutral, and negative). Each 30-second video was assessed with two-minute relaxation between categories. Validation results suggest that remote data collection is ecologically valid, providing an effective strategy for future affective study designs. All data can be accessed via: www.gnacek.com/affective-video-database-online-study .
Collapse
Affiliation(s)
- Michal Gnacek
- Centre for Digital Entertainment, Faculty of Media and Communication, Bournemouth University, Poole, BH12 5BB, UK.
- Emteq Labs, Brighton, BN1 9RS, UK.
| | - Luis Quintero
- Department of Computer and Systems Sciences, Stockholm University, 164 55, Stockholm, Sweden
| | | | - Emili Balaguer-Ballester
- Department of Computing and Informatics, Faculty of Science and Technology, Interdisciplinary Neuroscience Research Centre, Bournemouth University, Poole, BH12 5BB, UK
| | - Theodoros Kostoulas
- Department of Information and Communication Systems Engineering, University of the Aegean, Karlovasi, 832 00, Greece
| | | | - Ellen Seiss
- Department of Psychology, Faculty of Science and Technology, Interdisciplinary Neuroscience Research Centre, Bournemouth University, Poole, BH12 5BB, UK
| |
Collapse
|
4
|
Aldughayfiq B, Ashfaq F, Jhanjhi NZ, Humayun M. A Deep Learning Approach for Atrial Fibrillation Classification Using Multi-Feature Time Series Data from ECG and PPG. Diagnostics (Basel) 2023; 13:2442. [PMID: 37510187 PMCID: PMC10377944 DOI: 10.3390/diagnostics13142442] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2023] [Revised: 07/08/2023] [Accepted: 07/13/2023] [Indexed: 07/30/2023] Open
Abstract
Atrial fibrillation is a prevalent cardiac arrhythmia that poses significant health risks to patients. The use of non-invasive methods for AF detection, such as Electrocardiogram and Photoplethysmogram, has gained attention due to their accessibility and ease of use. However, there are challenges associated with ECG-based AF detection, and the significance of PPG signals in this context has been increasingly recognized. The limitations of ECG and the untapped potential of PPG are taken into account as this work attempts to classify AF and non-AF using PPG time series data and deep learning. In this work, we emploted a hybrid deep neural network comprising of 1D CNN and BiLSTM for the task of AF classification. We addressed the under-researched area of applying deep learning methods to transmissive PPG signals by proposing a novel approach. Our approach involved integrating ECG and PPG signals as multi-featured time series data and training deep learning models for AF classification. Our hybrid 1D CNN and BiLSTM model achieved an accuracy of 95% on test data in identifying atrial fibrillation, showcasing its strong performance and reliable predictive capabilities. Furthermore, we evaluated the performance of our model using additional metrics. The precision of our classification model was measured at 0.88, indicating its ability to accurately identify true positive cases of AF. The recall, or sensitivity, was measured at 0.85, illustrating the model's capacity to detect a high proportion of actual AF cases. Additionally, the F1 score, which combines both precision and recall, was calculated at 0.84, highlighting the overall effectiveness of our model in classifying AF and non-AF cases.
Collapse
Affiliation(s)
- Bader Aldughayfiq
- Department of Information Systems, College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia
| | - Farzeen Ashfaq
- School of Computer Science, SCS, Taylor's University, Subang Jaya 47500, Malaysia
| | - N Z Jhanjhi
- School of Computer Science, SCS, Taylor's University, Subang Jaya 47500, Malaysia
| | - Mamoona Humayun
- Department of Information Systems, College of Computer and Information Sciences, Jouf University, Sakaka 72388, Saudi Arabia
| |
Collapse
|
5
|
Kim J, Kim J. Classification of Breathing Signals According to Human Motions by Combining 1D Convolutional Neural Network and Embroidered Textile Sensor. SENSORS (BASEL, SWITZERLAND) 2023; 23:5736. [PMID: 37420902 DOI: 10.3390/s23125736] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/03/2023] [Revised: 06/13/2023] [Accepted: 06/16/2023] [Indexed: 07/09/2023]
Abstract
Research on healthcare and body monitoring has increased in recent years, with respiratory data being one of the most important factors. Respiratory measurements can help prevent diseases and recognize movements. Therefore, in this study, we measured respiratory data using a capacitance-based sensor garment with conductive electrodes. To determine the most stable measurement frequency, we conducted experiments using a porous Eco-flex and selected 45 kHz as the most stable frequency. Next, we trained a 1D convolutional neural network (CNN) model, which is a type of deep learning model, to classify the respiratory data according to four movements (standing, walking, fast walking, and running) using one input. The final test accuracy for classification was >95%. Therefore, the sensor garment developed in this study can measure respiratory data for four movements and classify them using deep learning, making it a versatile wearable in the form of a textile. We expect that this method will advance in various healthcare fields.
Collapse
Affiliation(s)
- Jiseon Kim
- Department of Smart Wearables Engineering, Soongsil University, Seoul 06978, Republic of Korea
| | - Jooyong Kim
- Department of Material Science and Engineering, Soongsil University, Seoul 06978, Republic of Korea
| |
Collapse
|
6
|
Nanocellulose-based sensors in medical/clinical applications: The state-of-the-art review. Carbohydr Polym 2023; 304:120509. [PMID: 36641173 DOI: 10.1016/j.carbpol.2022.120509] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 12/23/2022] [Accepted: 12/24/2022] [Indexed: 12/30/2022]
Abstract
In recent years, the considerable importance of healthcare and the indispensable appeal of curative issues, particularly the diagnosis of diseases, have propelled the invention of sensing platforms. With the development of nanotechnology, the integration of nanomaterials in such platforms has been much focused on, boosting their functionality in many fields. In this direction, there has been rapid growth in the utilisation of nanocellulose in sensors with medical applications. Indeed, this natural nanomaterial benefits from striking features, such as biocompatibility, cytocompatibility and low toxicity, as well as unprecedented physical and chemical properties. In this review, different classifications of nanocellulose-based sensors (biosensors, chemical and physical sensors), alongside some subcategories manufactured for health monitoring, stand out. Moreover, the types of nanocellulose and their roles in such sensors are discussed.
Collapse
|
7
|
Rathore KS, Vijayarangan S, Sp P, Sivaprakasam M. A Multifunctional Network with Uncertainty Estimation and Attention-Based Knowledge Distillation to Address Practical Challenges in Respiration Rate Estimation. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23031599. [PMID: 36772640 PMCID: PMC9920118 DOI: 10.3390/s23031599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 01/20/2023] [Accepted: 01/23/2023] [Indexed: 05/03/2023]
Abstract
Respiration rate is a vital parameter to indicate good health, wellbeing, and performance. As the estimation through classical measurement modes are limited only to rest or during slow movements, respiration rate is commonly estimated through physiological signals such as electrocardiogram and photoplethysmography due to the unobtrusive nature of wearable devices. Deep learning methodologies have gained much traction in the recent past to enhance accuracy during activities involving a lot of movement. However, these methods pose challenges, including model interpretability, uncertainty estimation in the context of respiration rate estimation, and model compactness in terms of deployment in wearable platforms. In this direction, we propose a multifunctional framework, which includes the combination of an attention mechanism, an uncertainty estimation functionality, and a knowledge distillation framework. We evaluated the performance of our framework on two datasets containing ambulatory movement. The attention mechanism visually and quantitatively improved instantaneous respiration rate estimation. Using Monte Carlo dropouts to embed the network with inferential uncertainty estimation resulted in the rejection of 3.7% of windows with high uncertainty, which consequently resulted in an overall reduction of 7.99% in the mean absolute error. The attention-aware knowledge distillation mechanism reduced the model's parameter count and inference time by 49.5% and 38.09%, respectively, without any increase in error rates. Through experimentation, ablation, and visualization, we demonstrated the efficacy of the proposed framework in addressing practical challenges, thus taking a step towards deployment in wearable edge devices.
Collapse
Affiliation(s)
- Kapil Singh Rathore
- Indian Institute of Technology Madras, Chennai 6000001, India
- Healthcare Technology Innovation Center, Chennai 6000001, India
| | - Sricharan Vijayarangan
- Indian Institute of Technology Madras, Chennai 6000001, India
- Healthcare Technology Innovation Center, Chennai 6000001, India
| | - Preejith Sp
- Healthcare Technology Innovation Center, Chennai 6000001, India
| | - Mohanasankar Sivaprakasam
- Indian Institute of Technology Madras, Chennai 6000001, India
- Healthcare Technology Innovation Center, Chennai 6000001, India
| |
Collapse
|