1
|
Heinisch JS, Kirchhoff J, Busch P, Wendt J, von Stryk O, David K. Physiological data for affective computing in HRI with anthropomorphic service robots: the AFFECT-HRI data set. Sci Data 2024; 11:333. [PMID: 38575624 PMCID: PMC10995145 DOI: 10.1038/s41597-024-03128-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Accepted: 03/05/2024] [Indexed: 04/06/2024] Open
Abstract
In human-human and human-robot interaction, the counterpart influences the human's affective state. Contrary to humans, robots inherently cannot respond empathically, meaning non-beneficial affective reactions cannot be mitigated. Thus, to create a responsible and empathetic human-robot interaction (HRI), involving anthropomorphic service robots, the effect of robot behavior on human affect in HRI must be understood. To contribute to this understanding, we provide the new comprehensive data set AFFECT-HRI, including, for the first time, physiological data labeled with human affect (i.e., emotions and mood) gathered from a conducted HRI study. Within the study, 146 participants interacted with an anthropomorphic service robot in a realistic and complex retail scenario. The participants' questionnaire ratings regarding affect, demographics, and socio-technical ratings are provided in the data set. Five different conditions (i.e., neutral, transparency, liability, moral, and immoral) were considered during the study, eliciting different affective reactions and allowing interdisciplinary investigations (e.g., computer science, law, and psychology). Each condition includes three scenes: a consultation regarding products, a request for sensitive personal information, and a handover.
Collapse
Affiliation(s)
- Judith S Heinisch
- University of Kassel, Chair for Communication Technology, Department of Electrical Engineering and Computer Science, WilhelmsöherAllee 73, 34121, Kassel, Germany.
| | - Jérôme Kirchhoff
- Technical University of Darmstadt, Chair for Simulation, Systems Opimization and Robotics, Department of Computer Science, Hochschulstrasse 10, 64289, Darmstadt, Germany
| | - Philip Busch
- Technical University of Darmstadt, Chair for Civil and Company Law, Department of Law and Economics, Hochschulstrasse 1, 64289, Darmstadt, Germany
| | - Janine Wendt
- Technical University of Darmstadt, Chair for Civil and Company Law, Department of Law and Economics, Hochschulstrasse 1, 64289, Darmstadt, Germany
| | - Oskar von Stryk
- Technical University of Darmstadt, Chair for Simulation, Systems Opimization and Robotics, Department of Computer Science, Hochschulstrasse 10, 64289, Darmstadt, Germany
| | - Klaus David
- University of Kassel, Chair for Communication Technology, Department of Electrical Engineering and Computer Science, WilhelmsöherAllee 73, 34121, Kassel, Germany
| |
Collapse
|
2
|
Bilucaglia M, Zito M, Fici A, Casiraghi C, Rivetti F, Bellati M, Russo V. I DARE: IULM Dataset of Affective Responses. Front Hum Neurosci 2024; 18:1347327. [PMID: 38571521 PMCID: PMC10987697 DOI: 10.3389/fnhum.2024.1347327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Accepted: 02/28/2024] [Indexed: 04/05/2024] Open
Affiliation(s)
- Marco Bilucaglia
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| | - Margherita Zito
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| | - Alessandro Fici
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| | - Chiara Casiraghi
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| | - Fiamma Rivetti
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
| | - Mara Bellati
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
| | - Vincenzo Russo
- Behaviour and Brain Lab, Neuromarketing Research Center, Università IULM, Milan, Italy
- Department of Business, Law, Economics and Consumer Behaviour “Carlo A. Ricciardi”, Università IULM, Milan, Italy
| |
Collapse
|
3
|
Bota P, Brito J, Fred A, Cesar P, Silva H. A real-world dataset of group emotion experiences based on physiological data. Sci Data 2024; 11:116. [PMID: 38263280 PMCID: PMC10805784 DOI: 10.1038/s41597-023-02905-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 12/29/2023] [Indexed: 01/25/2024] Open
Abstract
Affective computing has experienced substantial advancements in recognizing emotions through image and facial expression analysis. However, the incorporation of physiological data remains constrained. Emotion recognition with physiological data shows promising results in controlled experiments but lacks generalization to real-world settings. To address this, we present G-REx, a dataset for real-world affective computing. We collected physiological data (photoplethysmography and electrodermal activity) using a wrist-worn device during long-duration movie sessions. Emotion annotations were retrospectively performed on segments with elevated physiological responses. The dataset includes over 31 movie sessions, totaling 380 h+ of data from 190+ subjects. The data were collected in a group setting, which can give further context to emotion recognition systems. Our setup aims to be easily replicable in any real-life scenario, facilitating the collection of large datasets for novel affective computing systems.
Collapse
Affiliation(s)
- Patrícia Bota
- Instituto de Telecomunicações, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal.
- Instituto Superior Técnico, Dep. of Bioengineering, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal.
| | - Joana Brito
- Instituto de Telecomunicações, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal
| | - Ana Fred
- Instituto de Telecomunicações, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal
- Instituto Superior Técnico, Dep. of Bioengineering, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal
| | - Pablo Cesar
- Centrum Wiskunde & Informatica Amsterdam, The Netherlands & Multimedia Computing Group, 2600AA, Delft, The Netherlands
- Delft University of Technology, 2600AA, Delft, The Netherlands
| | - Hugo Silva
- Instituto de Telecomunicações, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal
- Instituto Superior Técnico, Dep. of Bioengineering, Avenida Rovisco Pais 1, Inst. Sup. Técnico, Torre Norte, Piso 10, 1049-001, Lisbon, Portugal
| |
Collapse
|
4
|
Ernst H, Scherpf M, Pannasch S, Helmert JR, Malberg H, Schmidt M. Assessment of the human response to acute mental stress-An overview and a multimodal study. PLoS One 2023; 18:e0294069. [PMID: 37943894 PMCID: PMC10635557 DOI: 10.1371/journal.pone.0294069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Accepted: 10/24/2023] [Indexed: 11/12/2023] Open
Abstract
Numerous vital signs are reported in association with stress response assessment, but their application varies widely. This work provides an overview over methods for stress induction and strain assessment, and presents a multimodal experimental study to identify the most important vital signs for effective assessment of the response to acute mental stress. We induced acute mental stress in 65 healthy participants with the Mannheim Multicomponent Stress Test and acquired self-assessment measures (Likert scale, Self-Assessment Manikin), salivary α-amylase and cortisol concentrations as well as 60 vital signs from biosignals, such as heart rate variability parameters, QT variability parameters, skin conductance level, and breath rate. By means of statistical testing and a self-optimizing logistic regression, we identified the most important biosignal vital signs. Fifteen biosignal vital signs related to ventricular repolarization variability, blood pressure, skin conductance, and respiration showed significant results. The logistic regression converged with QT variability index, left ventricular work index, earlobe pulse arrival time, skin conductance level, rise time and number of skin conductance responses, breath rate, and breath rate variability (F1 = 0.82). Self-assessment measures indicated successful stress induction. α-amylase and cortisol showed effect sizes of -0.78 and 0.55, respectively. In summary, the hypothalamic-pituitary-adrenocortical axis and sympathetic nervous system were successfully activated. Our findings facilitate a coherent and integrative understanding of the assessment of the stress response and help to align applications and future research concerning acute mental stress.
Collapse
Affiliation(s)
- Hannes Ernst
- Institute of Biomedical Engineering, TU Dresden, Dresden, Germany
| | - Matthieu Scherpf
- Institute of Biomedical Engineering, TU Dresden, Dresden, Germany
| | - Sebastian Pannasch
- Chair of Engineering Psychology and Applied Cognitive Research, TU Dresden, Dresden, Germany
| | - Jens R. Helmert
- Chair of Engineering Psychology and Applied Cognitive Research, TU Dresden, Dresden, Germany
| | - Hagen Malberg
- Institute of Biomedical Engineering, TU Dresden, Dresden, Germany
| | - Martin Schmidt
- Institute of Biomedical Engineering, TU Dresden, Dresden, Germany
| |
Collapse
|
5
|
Francisti J, Balogh Z, Reichel J, Benko Ľ, Fodor K, Turčáni M. Identification of heart rate change during the teaching process. Sci Rep 2023; 13:16674. [PMID: 37794176 PMCID: PMC10550993 DOI: 10.1038/s41598-023-43763-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2023] [Accepted: 09/28/2023] [Indexed: 10/06/2023] Open
Abstract
Internet of Things (IoT) technology can be used in many areas of everyday life. The objective of this paper is to obtain physiological functions in a non-invasive manner using commonly available IoT devices. The aim of the research is to point out the possibility of using physiological functions as an identifier of changes in students' level of arousal during the teaching process. The motivation of the work is to find a correlation between the change in heart rate, the student's level of arousal and the student's partial and final learning results. The research was focused on the collection of physiological data, namely heart rate and the evaluation of these data in the context of identification of arousal during individual teaching activities of the teaching process. The experiment was carried out during the COVID-19 pandemic via distance learning. During the teaching process, individual activities were recorded in time and HR was assigned to them. The benefit of the research is the proposed methodology of the system, which can identify changes in students' arousal in order to increase the efficiency of the teaching process. Based on the results of the designed system, they could also alert teachers who should be able to modify their teaching style in specific situations so that it is suitable for students and provides a basis for better teaching and understanding of educational materials. The presented methodology will be able to guarantee an increase in the success of the teaching process itself in terms of students' understanding of the teaching materials.
Collapse
Affiliation(s)
- Jan Francisti
- Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra, Nitra, Slovakia
| | - Zoltán Balogh
- Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra, Nitra, Slovakia
- Kandó Kálmán Faculty of Electrical Engineering, Óbuda University, Budapest, Hungary
| | - Jaroslav Reichel
- Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra, Nitra, Slovakia
| | - Ľubomír Benko
- Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra, Nitra, Slovakia
| | - Kristián Fodor
- Kandó Kálmán Faculty of Electrical Engineering, Óbuda University, Budapest, Hungary.
| | - Milan Turčáni
- Department of Informatics, Faculty of Natural Sciences and Informatics, Constantine the Philosopher University in Nitra, Nitra, Slovakia
| |
Collapse
|
6
|
Minusa S, Yoshimura C, Mizuno H. Emodiversity evaluation of remote workers through health monitoring based on intra-day emotion sampling. Front Public Health 2023; 11:1196539. [PMID: 37670827 PMCID: PMC10475727 DOI: 10.3389/fpubh.2023.1196539] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 07/31/2023] [Indexed: 09/07/2023] Open
Abstract
Introduction In recent years, the widespread shift from on-site to remote work has led to a decline in employees' mental health. Consequently, this transition to remote work poses several challenges for both employees and employers. To address these challenges, there is an urgent need for techniques to detect declining mental health in employees' daily lives. Emotion-based health assessment, which examines emotional diversity (emodiversity) experienced in daily life, is a possible solution. However, the feasibility of emodiversity remains unclear, especially from the perspectives of its applicability to remote workers and countries other than Europe and the United States. This study investigated the association between subjective mental health decline and emotional factors, such as emodiversity, as well as physical conditions, in remote workers in Japan. Method To explore this association, we conducted a consecutive 14-day prospective observational experiment on 18 Japanese remote workers. This experiment comprised pre-and post-questionnaire surveys, physiological sensing, daytime emotion self-reports, and subjective health reports at end-of-day. In daytime emotion self-reports, we introduced smartphone-based experience sampling (also known as ecological momentary assessment), which is suitable for collecting context-dependent self-reports precisely in a recall bias-less manner. For 17 eligible participants (mean ± SD, 39.1 ± 9.1 years), we evaluated whether and how the psycho-physical characteristics, including emodiversity, changed on subjective mental health-declined experimental days after analyzing descriptive statistics. Results Approximately half of the experimental days (46.3 ± 18.9%) were conducted under remote work conditions. Our analysis showed that physical and emotional indices significantly decreased on mental health-declined days. Especially on high anxiety and depressive days, we found that emodiversity indicators significantly decreased (global emodiversity on anxiety conditions, 0.409 ± 0.173 vs. 0.366 ± 0.143, p = 0.041), and positive emotional experiences were significantly suppressed (61.5 ± 7.7 vs. 55.5 ± 6.4, p < 0.001). Discussion Our results indicated that the concept of emodiversity can be applicable even to Japanese remote workers, whose cultural background differs from that of individuals in Europe and the United States. Emodiversity showed significant associations with emotion dysregulation-related mental health deterioration, suggesting the potential of emodiversity as useful indicators in managing such mental health deterioration among remote workers.
Collapse
Affiliation(s)
- Shunsuke Minusa
- Center for Exploratory Research, Research & Development Group, Hitachi, Ltd., Tokyo, Japan
| | | | | |
Collapse
|
7
|
Gong B, Li N, Li Q, Yan X, Chen J, Li L, Wu X, Wu C. The Mandarin Chinese auditory emotions stimulus database: A validated set of Chinese pseudo-sentences. Behav Res Methods 2023; 55:1441-1459. [PMID: 35641682 DOI: 10.3758/s13428-022-01868-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 04/29/2022] [Indexed: 11/08/2022]
Abstract
Emotional prosody is fully embedded in language and can be influenced by the linguistic properties of a specific language. Considering the limitations of existing Chinese auditory stimulus database studies, we developed and validated an emotional auditory stimuli database composed of Chinese pseudo-sentences, recorded by six professional actors in Mandarin Chinese. Emotional expressions included happiness, sadness, anger, fear, disgust, pleasant surprise, and neutrality. All emotional categories were vocalized into two types of sentence patterns, declarative and interrogative. In addition, all emotional pseudo-sentences, except for neutral, were vocalized at two levels of emotional intensity: normal and strong. Each recording was validated with 40 native Chinese listeners in terms of the recognition accuracy of the intended emotion portrayal; finally, 4361 pseudo-sentence stimuli were included in the database. Validation of the database using a forced-choice recognition paradigm revealed high rates of emotional recognition accuracy. The detailed acoustic attributes of vocalization were provided and connected to the emotion recognition rates. This corpus could be a valuable resource for researchers and clinicians to explore the behavioral and neural mechanisms underlying emotion processing of the general population and emotional disturbances in neurological, psychiatric, and developmental disorders. The Mandarin Chinese auditory emotion stimulus database is available at the Open Science Framework ( https://osf.io/sfbm6/?view_only=e22a521e2a7d44c6b3343e11b88f39e3 ).
Collapse
Affiliation(s)
- Bingyan Gong
- School of Nursing, Peking University Health Science Center, Room 510, 38 Xueyuan Road, Haidian District, Beijing, 100191, China
| | - Na Li
- Theatre Pedagogy Department, Central Academy of Drama, Beijing, 100710, China
| | - Qiuhong Li
- School of Nursing, Peking University Health Science Center, Room 510, 38 Xueyuan Road, Haidian District, Beijing, 100191, China
| | - Xinyuan Yan
- School of Computing, University of Utah, Salt Lake City, UT, USA
| | - Jing Chen
- Department of Machine Intelligence, Peking University, 5 Yiheyuan Road, Haidian District, Beijing, 100871, China
- Speech and Hearing Research Center, Key Laboratory on Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China
| | - Liang Li
- School of Psychological and Cognitive Sciences, Peking University, Beijing, 100871, China
| | - Xihong Wu
- Department of Machine Intelligence, Peking University, 5 Yiheyuan Road, Haidian District, Beijing, 100871, China.
- Speech and Hearing Research Center, Key Laboratory on Machine Perception (Ministry of Education), Peking University, Beijing, 100871, China.
| | - Chao Wu
- School of Nursing, Peking University Health Science Center, Room 510, 38 Xueyuan Road, Haidian District, Beijing, 100191, China.
| |
Collapse
|
8
|
Bosses without a heart: socio-demographic and cross-cultural determinants of attitude toward Emotional AI in the workplace. AI & SOCIETY 2023; 38:97-119. [PMID: 34776651 PMCID: PMC8571983 DOI: 10.1007/s00146-021-01290-1] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2021] [Accepted: 09/17/2021] [Indexed: 02/06/2023]
Abstract
Biometric technologies are becoming more pervasive in the workplace, augmenting managerial processes such as hiring, monitoring and terminating employees. Until recently, these devices consisted mainly of GPS tools that track location, software that scrutinizes browser activity and keyboard strokes, and heat/motion sensors that monitor workstation presence. Today, however, a new generation of biometric devices has emerged that can sense, read, monitor and evaluate the affective state of a worker. More popularly known by its commercial moniker, Emotional AI, the technology stems from advancements in affective computing. But whereas previous generations of biometric monitoring targeted the exterior physical body of the worker, concurrent with the writings of Foucault and Hardt, we argue that emotion-recognition tools signal a far more invasive disciplinary gaze that exposes and makes vulnerable the inner regions of the worker-self. Our paper explores attitudes towards empathic surveillance by analyzing a survey of 1015 responses of future job-seekers from 48 countries with Bayesian statistics. Our findings reveal affect tools, left unregulated in the workplace, may lead to heightened stress and anxiety among disadvantaged ethnicities, gender and income class. We also discuss a stark cross-cultural discrepancy whereby East Asians, compared to Western subjects, are more likely to profess a trusting attitude toward EAI-enabled automated management. While this emerging technology is driven by neoliberal incentives to optimize the worksite and increase productivity, ultimately, empathic surveillance may create more problems in terms of algorithmic bias, opaque decisionism, and the erosion of employment relations. Thus, this paper nuances and extends emerging literature on emotion-sensing technologies in the workplace, particularly through its highly original cross-cultural study. Supplementary Information The online version contains supplementary material available at 10.1007/s00146-021-01290-1.
Collapse
|
9
|
Fear Detection in Multimodal Affective Computing: Physiological Signals versus Catecholamine Concentration. SENSORS 2022; 22:s22114023. [PMID: 35684644 PMCID: PMC9183081 DOI: 10.3390/s22114023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/30/2022] [Revised: 05/16/2022] [Accepted: 05/18/2022] [Indexed: 12/04/2022]
Abstract
Affective computing through physiological signals monitoring is currently a hot topic in the scientific literature, but also in the industry. Many wearable devices are being developed for health or wellness tracking during daily life or sports activity. Likewise, other applications are being proposed for the early detection of risk situations involving sexual or violent aggressions, with the identification of panic or fear emotions. The use of other sources of information, such as video or audio signals will make multimodal affective computing a more powerful tool for emotion classification, improving the detection capability. There are other biological elements that have not been explored yet and that could provide additional information to better disentangle negative emotions, such as fear or panic. Catecholamines are hormones produced by the adrenal glands, two small glands located above the kidneys. These hormones are released in the body in response to physical or emotional stress. The main catecholamines, namely adrenaline, noradrenaline and dopamine have been analysed, as well as four physiological variables: skin temperature, electrodermal activity, blood volume pulse (to calculate heart rate activity. i.e., beats per minute) and respiration rate. This work presents a comparison of the results provided by the analysis of physiological signals in reference to catecholamine, from an experimental task with 21 female volunteers receiving audiovisual stimuli through an immersive environment in virtual reality. Artificial intelligence algorithms for fear classification with physiological variables and plasma catecholamine concentration levels have been proposed and tested. The best results have been obtained with the features extracted from the physiological variables. Adding catecholamine’s maximum variation during the five minutes after the video clip visualization, as well as adding the five measurements (1-min interval) of these levels, are not providing better performance in the classifiers.
Collapse
|
10
|
EmotiphAI: a biocybernetic engine for real-time biosignals acquisition in a collective setting. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07191-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
11
|
Iqbal T, Elahi A, Wijns W, Shahzad A. Exploring Unsupervised Machine Learning Classification Methods for Physiological Stress Detection. FRONTIERS IN MEDICAL TECHNOLOGY 2022; 4:782756. [PMID: 35359827 PMCID: PMC8962952 DOI: 10.3389/fmedt.2022.782756] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/24/2021] [Accepted: 02/10/2022] [Indexed: 12/04/2022] Open
Abstract
Over the past decade, there has been a significant development in wearable health technologies for diagnosis and monitoring, including application to stress monitoring. Most of the wearable stress monitoring systems are built on a supervised learning classification algorithm. These systems rely on the collection of sensor and reference data during the development phase. One of the most challenging tasks in physiological or pathological stress monitoring is the labeling of the physiological signals collected during an experiment. Commonly, different types of self-reporting questionnaires are used to label the perceived stress instances. These questionnaires only capture stress levels at a specific point in time. Moreover, self-reporting is subjective and prone to inaccuracies. This paper explores the potential feasibility of unsupervised learning clustering classifiers such as Affinity Propagation, Balanced Iterative Reducing and Clustering using Hierarchies (BIRCH), K-mean, Mini-Batch K-mean, Mean Shift, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) and Ordering Points To Identify the Clustering Structure (OPTICS) for implementation in stress monitoring wearable devices. Traditional supervised machine learning (linear, ensembles, trees, and neighboring models) classifiers require hand-crafted features and labels while on the other hand, the unsupervised classifier does not require any labels of perceived stress levels and performs classification based on clustering algorithms. The classification results of unsupervised machine learning classifiers are found comparable to supervised machine learning classifiers on two publicly available datasets. The analysis and results of this comparative study demonstrate the potential of unsupervised learning for the development of non-invasive, continuous, and robust detection and monitoring of physiological and pathological stress.
Collapse
Affiliation(s)
- Talha Iqbal
- Smart Sensors Lab, Lambe Institute of Translational Research, National University of Ireland Galway, Galway, Ireland
- *Correspondence: Talha Iqbal
| | - Adnan Elahi
- Electrical and Electronics Engineering, National University of Ireland Galway, Galway, Ireland
| | - William Wijns
- Smart Sensors Lab, Lambe Institute of Translational Research, National University of Ireland Galway, Galway, Ireland
| | - Atif Shahzad
- Smart Sensors Lab, Lambe Institute of Translational Research, National University of Ireland Galway, Galway, Ireland
- Centre for Systems Modelling and Quantitative Biomedicine, University of Birmingham, Birmingham, United Kingdom
| |
Collapse
|
12
|
Haruvi A, Kopito R, Brande-Eilat N, Kalev S, Kay E, Furman D. Measuring and Modeling the Effect of Audio on Human Focus in Everyday Environments Using Brain-Computer Interface Technology. Front Comput Neurosci 2022; 15:760561. [PMID: 35153708 PMCID: PMC8829886 DOI: 10.3389/fncom.2021.760561] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2021] [Accepted: 12/17/2021] [Indexed: 11/23/2022] Open
Abstract
The goal of this study was to investigate the effect of audio listened to through headphones on subjectively reported human focus levels, and to identify through objective measures the properties that contribute most to increasing and decreasing focus in people within their regular, everyday environment. Participants (N = 62, 18–65 years) performed various tasks on a tablet computer while listening to either no audio (silence), popular audio playlists designed to increase focus (pre-recorded music arranged in a particular sequence of songs), or engineered soundscapes that were personalized to individual listeners (digital audio composed in real-time based on input parameters such as heart rate, time of day, location, etc.). Audio stimuli were delivered to participants through headphones while their brain signals were simultaneously recorded by a portable electroencephalography headband. Participants completed four 1-h long sessions at home during which different audio played continuously in the background. Using brain-computer interface technology for brain decoding and based on an individual’s self-report of their focus, we obtained individual focus levels over time and used this data to analyze the effects of various properties of the sounds contained in the audio content. We found that while participants were working, personalized soundscapes increased their focus significantly above silence (p = 0.008), while music playlists did not have a significant effect. For the young adult demographic (18–36 years), all audio tested was significantly better than silence at producing focus (p = 0.001–0.009). Personalized soundscapes increased focus the most relative to silence, but playlists of pre-recorded songs also increased focus significantly during specific time intervals. Ultimately we found it is possible to accurately predict human focus levels a priori based on physical properties of audio content. We then applied this finding to compare between music genres and revealed that classical music, engineered soundscapes, and natural sounds were the best genres for increasing focus, while pop and hip-hop were the worst. These insights can enable human and artificial intelligence composers to produce increases or decreases in listener focus with high temporal (millisecond) precision. Future research will include real-time adaptation of audio for other functional objectives beyond affecting focus, such as affecting listener enjoyment, drowsiness, stress and memory.
Collapse
|
13
|
Föll S, Maritsch M, Spinola F, Mishra V, Barata F, Kowatsch T, Fleisch E, Wortmann F. FLIRT: A feature generation toolkit for wearable data. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 212:106461. [PMID: 34736174 DOI: 10.1016/j.cmpb.2021.106461] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/04/2021] [Accepted: 10/06/2021] [Indexed: 06/13/2023]
Abstract
BACKGROUND AND OBJECTIVE Researchers use wearable sensing data and machine learning (ML) models to predict various health and behavioral outcomes. However, sensor data from commercial wearables are prone to noise, missing, or artifacts. Even with the recent interest in deploying commercial wearables for long-term studies, there does not exist a standardized way to process the raw sensor data and researchers often use highly specific functions to preprocess, clean, normalize, and compute features. This leads to a lack of uniformity and reproducibility across different studies, making it difficult to compare results. To overcome these issues, we present FLIRT: A Feature Generation Toolkit for Wearable Data; it is an open-source Python package that focuses on processing physiological data specifically from commercial wearables with all its challenges from data cleaning to feature extraction. METHODS FLIRT leverages a variety of state-of-the-art algorithms (e.g., particle filters, ML-based artifact detection) to ensure a robust preprocessing of physiological data from wearables. In a subsequent step, FLIRT utilizes a sliding-window approach and calculates a feature vector of more than 100 dimensions - a basis for a wide variety of ML algorithms. RESULTS We evaluated FLIRT on the publicly available WESAD dataset, which focuses on stress detection with an Empatica E4 wearable. Preprocessing the data with FLIRT ensures that unintended noise and artifacts are appropriately filtered. In the classification task, FLIRT outperforms the preprocessing baseline of the original WESAD paper. CONCLUSION FLIRT provides functionalities beyond existing packages that can address unmet needs in physiological data processing and feature generation: (a) integrated handling of common wearable file formats (e.g., Empatica E4 archives), (b) robust preprocessing, and (c) standardized feature generation that ensures reproducibility of results. Nevertheless, while FLIRT comes with a default configuration to accommodate most situations, it offers a highly configurable interface for all of its implemented algorithms to account for specific needs.
Collapse
Affiliation(s)
- Simon Föll
- Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland.
| | - Martin Maritsch
- Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland.
| | - Federica Spinola
- Department of Mechanical and Process Engineering, ETH Zurich, Zurich, Switzerland.
| | - Varun Mishra
- Department of Computer Science, Dartmouth College, Hanover, NH, USA.
| | - Filipe Barata
- Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland.
| | - Tobias Kowatsch
- Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland; Institute of Technology Management, University of St. Gallen, St. Gallen, Switzerland.
| | - Elgar Fleisch
- Department of Management, Technology, and Economics, ETH Zurich, Zurich, Switzerland; Institute of Technology Management, University of St. Gallen, St. Gallen, Switzerland.
| | - Felix Wortmann
- Institute of Technology Management, University of St. Gallen, St. Gallen, Switzerland.
| |
Collapse
|
14
|
An Intra-Subject Approach Based on the Application of HMM to Predict Concentration in Educational Contexts from Nonintrusive Physiological Signals in Real-World Situations. SENSORS 2021; 21:s21051777. [PMID: 33806438 PMCID: PMC7961751 DOI: 10.3390/s21051777] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/31/2020] [Revised: 02/19/2021] [Accepted: 02/25/2021] [Indexed: 11/28/2022]
Abstract
Previous research has proven the strong influence of emotions on student engagement and motivation. Therefore, emotion recognition is becoming very relevant in educational scenarios, but there is no standard method for predicting students’ affects. However, physiological signals have been widely used in educational contexts. Some physiological signals have shown a high accuracy in detecting emotions because they reflect spontaneous affect-related information, which is fresh and does not require additional control or interpretation. Most proposed works use measuring equipment for which applicability in real-world scenarios is limited because of its high cost and intrusiveness. To tackle this problem, in this work, we analyse the feasibility of developing low-cost and nonintrusive devices to obtain a high detection accuracy from easy-to-capture signals. By using both inter-subject and intra-subject models, we present an experimental study that aims to explore the potential application of Hidden Markov Models (HMM) to predict the concentration state from 4 commonly used physiological signals, namely heart rate, breath rate, skin conductance and skin temperature. We also study the effect of combining these four signals and analyse their potential use in an educational context in terms of intrusiveness, cost and accuracy. The results show that a high accuracy can be achieved with three of the signals when using HMM-based intra-subject models. However, inter-subject models, which are meant to obtain subject-independent approaches for affect detection, fail at the same task.
Collapse
|