1
|
Corponi F, Li BM, Anmella G, Valenzuela-Pascual C, Mas A, Pacchiarotti I, Valentí M, Grande I, Benabarre A, Garriga M, Vieta E, Young AH, Lawrie SM, Whalley HC, Hidalgo-Mazzei D, Vergari A. Wearable Data From Subjects Playing Super Mario, Taking University Exams, or Performing Physical Exercise Help Detect Acute Mood Disorder Episodes via Self-Supervised Learning: Prospective, Exploratory, Observational Study. JMIR Mhealth Uhealth 2024; 12:e55094. [PMID: 39018100 DOI: 10.2196/55094] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2023] [Revised: 04/14/2024] [Accepted: 05/24/2024] [Indexed: 07/18/2024] Open
Abstract
BACKGROUND Personal sensing, leveraging data passively and near-continuously collected with wearables from patients in their ecological environment, is a promising paradigm to monitor mood disorders (MDs), a major determinant of the worldwide disease burden. However, collecting and annotating wearable data is resource intensive. Studies of this kind can thus typically afford to recruit only a few dozen patients. This constitutes one of the major obstacles to applying modern supervised machine learning techniques to MD detection. OBJECTIVE In this paper, we overcame this data bottleneck and advanced the detection of acute MD episodes from wearables' data on the back of recent advances in self-supervised learning (SSL). This approach leverages unlabeled data to learn representations during pretraining, subsequently exploited for a supervised task. METHODS We collected open access data sets recording with the Empatica E4 wristband spanning different, unrelated to MD monitoring, personal sensing tasks-from emotion recognition in Super Mario players to stress detection in undergraduates-and devised a preprocessing pipeline performing on-/off-body detection, sleep/wake detection, segmentation, and (optionally) feature extraction. With 161 E4-recorded subjects, we introduced E4SelfLearning, the largest-to-date open access collection, and its preprocessing pipeline. We developed a novel E4-tailored transformer (E4mer) architecture, serving as the blueprint for both SSL and fully supervised learning; we assessed whether and under which conditions self-supervised pretraining led to an improvement over fully supervised baselines (ie, the fully supervised E4mer and pre-deep learning algorithms) in detecting acute MD episodes from recording segments taken in 64 (n=32, 50%, acute, n=32, 50%, stable) patients. RESULTS SSL significantly outperformed fully supervised pipelines using either our novel E4mer or extreme gradient boosting (XGBoost): n=3353 (81.23%) against n=3110 (75.35%; E4mer) and n=2973 (72.02%; XGBoost) correctly classified recording segments from a total of 4128 segments. SSL performance was strongly associated with the specific surrogate task used for pretraining, as well as with unlabeled data availability. CONCLUSIONS We showed that SSL, a paradigm where a model is pretrained on unlabeled data with no need for human annotations before deployment on the supervised target task of interest, helps overcome the annotation bottleneck; the choice of the pretraining surrogate task and the size of unlabeled data for pretraining are key determinants of SSL success. We introduced E4mer, which can be used for SSL, and shared the E4SelfLearning collection, along with its preprocessing pipeline, which can foster and expedite future research into SSL for personal sensing.
Collapse
Affiliation(s)
- Filippo Corponi
- School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
| | - Bryan M Li
- School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
- The Alan Turing Institute, London, United Kingdom
| | - Gerard Anmella
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Clàudia Valenzuela-Pascual
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Ariadna Mas
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Isabella Pacchiarotti
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Marc Valentí
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Iria Grande
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Antoni Benabarre
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Marina Garriga
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Eduard Vieta
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
| | - Allan H Young
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | - Stephen M Lawrie
- Division of Psychiatry, Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
| | - Heather C Whalley
- Division of Psychiatry, Centre for Clinical Brain Sciences, University of Edinburgh, Edinburgh, United Kingdom
- Generation Scotland, Institute for Genetics and Cancer, University of Edinburgh, Edinburgh, United Kingdom
| | - Diego Hidalgo-Mazzei
- Bipolar and Depressive Disorders Unit, Department of Psychiatry and Psychology, Hospital Clínic de Barcelona, Barcelona, Spain
- Institut d'Investigacions Biomèdiques August Pi i Sunyer, Barcelona, Spain
- Centro de Investigación Biomédica en Red de Salud Mental, Instituto de Salud Carlos III, Madrid, Spain
- Departament de Medicina, Facultat de Medicina i Ciències de la Salut, Universitat de Barcelona, Barcelona, Spain
- Institute of Psychiatry, Psychology and Neuroscience, King's College London, London, United Kingdom
| | - Antonio Vergari
- School of Informatics, University of Edinburgh, Edinburgh, United Kingdom
| |
Collapse
|
2
|
Borghini G, Ronca V, Giorgi A, Aricò P, Di Flumeri G, Capotorto R, Rooseleer F, Kirwan B, De Visscher I, Goman M, Pugh J, Abramov N, Granger G, Alarcon DPM, Humm E, Pozzi S, Babiloni F. Reducing flight upset risk and startle response: A study of the wake vortex alert with licensed commercial pilots. Brain Res Bull 2024; 215:111020. [PMID: 38909913 DOI: 10.1016/j.brainresbull.2024.111020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 05/24/2024] [Accepted: 06/20/2024] [Indexed: 06/25/2024]
Abstract
The study aimed at investigating the impact of an innovative Wake Vortex Alert (WVA) avionics on pilots' operation and mental states, intending to improve aviation safety by mitigating the risks associated with wake vortex encounters (WVEs). Wake vortices, generated by jet aircraft, pose a significant hazard to trailing or crossing aircrafts. Despite existing separation rules, incidents involving WVEs continue to occur, especially affecting smaller aircrafts like business jets, resulting in aircraft upsets and occasional cabin injuries. To address these challenges, the study focused on developing and validating an alert system that can be presented to air traffic controllers, enabling them to warn flight crews. This empowers the flight crews to either avoid the wake vortex or secure the cabin to prevent injuries. The research employed a multidimensional approach including an analysis of human performance and human factors (HF) issues to determine the potential impact of the alert on pilots' roles, tasks, and mental states. It also utilizes Human Assurance Levels (HALs) to evaluate the necessary human factors support based on the safety criticality of the new system. Realistic flight simulations were conducted to collect data of pilots' behavioural, subjective and neurophysiological responses during WVEs. The data allowed for an objective evaluation of the WVA impact on pilots' operation, behaviour and mental states (mental workload, stress levels and arousal). In particular, the results highlighted the effectiveness of the alert system in facilitating pilots' preparation, awareness and crew resource management (CRM). The results also highlighted the importance of avionics able to enhance aviation safety and reducing risks associated with wake vortex encounters. In particular, we demonstrated how providing timely information and improving situational awareness, the WVA will minimize the occurrence of WVEs and contribute to safer aviation operations.
Collapse
Affiliation(s)
- Gianluca Borghini
- Department of Molecular Medicine, Sapienza University of Rome, Italy; BrainSigns srl, Rome, Italy.
| | - Vincenzo Ronca
- BrainSigns srl, Rome, Italy; Department of Computer, Control, and Management Engineering "Antonio Ruberti", Sapienza University of Rome, Italy
| | - Andrea Giorgi
- BrainSigns srl, Rome, Italy; Department of Anatomical, Histological, Forensic and Orthopedic Sciences, Sapienza University of Rome, Italy
| | - Pietro Aricò
- BrainSigns srl, Rome, Italy; Department of Computer, Control, and Management Engineering "Antonio Ruberti", Sapienza University of Rome, Italy
| | - Gianluca Di Flumeri
- Department of Molecular Medicine, Sapienza University of Rome, Italy; BrainSigns srl, Rome, Italy
| | - Rossella Capotorto
- Department of Anatomical, Histological, Forensic and Orthopedic Sciences, Sapienza University of Rome, Italy
| | | | - Barry Kirwan
- EUROCONTROL, Centre du Bois des Bordes, Bretigny-sur-Orge, France
| | - Ivan De Visscher
- EUROCONTROL, Centre du Bois des Bordes, Bretigny-sur-Orge, France
| | - Mikhail Goman
- Faculty of Computing, Engineering and Media, De Montfort University, Leicester, United Kingdom
| | - Jonathan Pugh
- Faculty of Computing, Engineering and Media, De Montfort University, Leicester, United Kingdom
| | - Nikolay Abramov
- Faculty of Computing, Engineering and Media, De Montfort University, Leicester, United Kingdom
| | - Géraud Granger
- Safety Management Research Program, École Nationale de l'Aviation Civile (ENAC), France
| | | | | | | | - Fabio Babiloni
- Department of Physiology and Pharmacology "Vittorio Erspamer", Sapienza University of Rome, Italy; School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China
| |
Collapse
|
4
|
Giorgi A, Ronca V, Vozzi A, Aricò P, Borghini G, Capotorto R, Tamborra L, Simonetti I, Sportiello S, Petrelli M, Polidori C, Varga R, van Gasteren M, Barua A, Ahmed MU, Babiloni F, Di Flumeri G. Neurophysiological mental fatigue assessment for developing user-centered Artificial Intelligence as a solution for autonomous driving. Front Neurorobot 2023; 17:1240933. [PMID: 38107403 PMCID: PMC10721973 DOI: 10.3389/fnbot.2023.1240933] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 10/18/2023] [Indexed: 12/19/2023] Open
Abstract
The human factor plays a key role in the automotive field since most accidents are due to drivers' unsafe and risky behaviors. The industry is now pursuing two main solutions to deal with this concern: in the short term, there is the development of systems monitoring drivers' psychophysical states, such as inattention and fatigue, and in the medium-long term, there is the development of fully autonomous driving. This second solution is promoted by recent technological progress in terms of Artificial Intelligence and sensing systems aimed at making vehicles more and more accurately aware of their "surroundings." However, even with an autonomous vehicle, the driver should be able to take control of the vehicle when needed, especially during the current transition from the lower (SAE < 3) to the highest level (SAE = 5) of autonomous driving. In this scenario, the vehicle has to be aware not only of its "surroundings" but also of the driver's psychophysical state, i.e., a user-centered Artificial Intelligence. The neurophysiological approach is one the most effective in detecting improper mental states. This is particularly true if considering that the more automatic the driving will be, the less available the vehicular data related to the driver's driving style. The present study aimed at employing a holistic approach, considering simultaneously several neurophysiological parameters, in particular, electroencephalographic, electrooculographic, photopletismographic, and electrodermal activity data to assess the driver's mental fatigue in real time and to detect the onset of fatigue increasing. This would ideally work as an information/trigger channel for the vehicle AI. In all, 26 professional drivers were engaged in a 45-min-lasting realistic driving task in simulated conditions, during which the previously listed biosignals were recorded. Behavioral (reaction times) and subjective measures were also collected to validate the experimental design and to support the neurophysiological results discussion. Results showed that the most sensitive and timely parameters were those related to brain activity. To a lesser extent, those related to ocular parameters were also sensitive to the onset of mental fatigue, but with a delayed effect. The other investigated parameters did not significantly change during the experimental session.
Collapse
Affiliation(s)
- Andrea Giorgi
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, Rome, Italy
- BrainSigns SRL, Rome, Italy
| | - Vincenzo Ronca
- BrainSigns SRL, Rome, Italy
- Department of Computer, Control, and Management Engineering, Sapienza University of Rome, Rome, Italy
| | - Alessia Vozzi
- Department of Anatomical, Histological, Forensic and Orthopaedic Sciences, Sapienza University of Rome, Rome, Italy
- BrainSigns SRL, Rome, Italy
| | - Pietro Aricò
- BrainSigns SRL, Rome, Italy
- Department of Computer, Control, and Management Engineering, Sapienza University of Rome, Rome, Italy
| | - Gianluca Borghini
- BrainSigns SRL, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Rossella Capotorto
- Department of Computer, Control, and Management Engineering, Sapienza University of Rome, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Luca Tamborra
- BrainSigns SRL, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Ilaria Simonetti
- BrainSigns SRL, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Simone Sportiello
- Department of Civil Engineering, Computer Science and Aeronautical Technologies, Roma Tre University, Rome, Italy
- Department of Enterprise Engineering, University of Rome Tor Vergata, Rome, Italy
| | - Marco Petrelli
- Department of Civil Engineering, Computer Science and Aeronautical Technologies, Roma Tre University, Rome, Italy
| | - Carlo Polidori
- Italian Association of Road Safety Professionals (AIPSS), Rome, Italy
| | - Rodrigo Varga
- Instituto Tecnologico de Castilla y Leon, Burgos, Spain
| | | | - Arnab Barua
- Academy for Innovation, Design and Technology, Mälardalens University, Västerås, Sweden
| | - Mobyen Uddin Ahmed
- Academy for Innovation, Design and Technology, Mälardalens University, Västerås, Sweden
| | - Fabio Babiloni
- BrainSigns SRL, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
- College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China
| | - Gianluca Di Flumeri
- BrainSigns SRL, Rome, Italy
- Laboratory of Industrial Neuroscience, Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| |
Collapse
|