An Investigation of Various Machine and Deep Learning Techniques Applied in Automatic Fear Level Detection and Acrophobia Virtual Therapy

Enhancing Aviation Safety through AI-Driven Mental Health Management for Pilots and Air Traffic Controllers

This article provides an overview of the mental health challenges faced by pilots and air traffic controllers (ATCs), whose stressful professional lives may negatively impact global flight safety and security. The adverse effects of mental health disorders on their flight performance pose a particular safety risk, especially in sudden unexpected startle situations. Therefore, the early detection, prediction and prevention of mental health deterioration in pilots and ATCs, particularly among those at high risk, are crucial to minimize potential air crash incidents caused by human factors. Recent research in artificial intelligence (AI) demonstrates the potential of machine and deep learning, edge and cloud computing, virtual reality and wearable multimodal physiological sensors for monitoring and predicting mental health disorders. Longitudinal monitoring and analysis of pilots' and ATCs physiological, cognitive and behavioral states could help predict individuals at risk of undisclosed or emerging mental health disorders. Utilizing AI tools and methodologies to identify and select these individuals for preventive mental health training and interventions could be a promising and effective approach to preventing potential air crash accidents attributed to human factors and related mental health problems. Based on these insights, the article advocates for the design of a multidisciplinary mental healthcare ecosystem in modern aviation using AI tools and technologies, to foster more efficient and effective mental health management, thereby enhancing flight safety and security standards. This proposed ecosystem requires the collaboration of multidisciplinary experts, including psychologists, neuroscientists, physiologists, psychiatrists, etc. to address these challenges in modern aviation.

Analyzing EEG patterns in young adults exposed to different acrophobia levels: a VR study

Introduction

The primary objective of this research is to examine acrophobia, a widely prevalent and highly severe phobia characterized by an overwhelming dread of heights, which has a substantial impact on a significant proportion of individuals worldwide. The objective of our study was to develop a real-time and precise instrument for evaluating levels of acrophobia by utilizing electroencephalogram (EEG) signals.

Methods

EEG data was gathered from a sample of 18 individuals diagnosed with acrophobia. Subsequently, a range of classifiers, namely Support Vector Classifier (SVC), K-nearest Neighbors (KNN), Random Forest (RF), Decision Tree (DT), Adaboost, Linear Discriminant Analysis (LDA), Convolutional Neural Network (CNN), and Artificial Neural Network (ANN), were employed in the analysis. These methodologies encompass both machine learning (ML) and deep learning (DL) techniques.

Results

The Convolutional Neural Network (CNN) and Artificial Neural Network (ANN) models demonstrated notable efficacy. The Convolutional Neural Network (CNN) model demonstrated a training accuracy of 96% and a testing accuracy of 99%, whereas the Artificial Neural Network (ANN) model attained a training accuracy of 96% and a testing accuracy of 97%. The findings of this study highlight the effectiveness of the proposed methodology in accurately categorizing real-time degrees of acrophobia using EEG data. Further investigation using correlation matrices for each level of acrophobia showed substantial EEG frequency band connections. Beta and Gamma mean values correlated strongly, suggesting cognitive arousal and acrophobic involvement could synchronize activity. Beta and Gamma activity correlated strongly with acrophobia, especially at higher levels.

Discussion

The results underscore the promise of this innovative approach as a dependable and sophisticated method for evaluating acrophobia. This methodology has the potential to make a substantial contribution toward the comprehension and assessment of acrophobia, hence facilitating the development of more individualized and efficacious therapeutic interventions.

PLUME: Record, Replay, Analyze and Share User Behavior in 6DoF XR Experiences

From education to medicine to entertainment, a wide range of industrial and academic fields now utilize eXtended Reality (XR) technologies. This diversity and growing use are boosting research and leading to an increasing number of XR experiments involving human subjects. The main aim of these studies is to understand the user experience in the broadest sense, such as the user cognitive and emotional states. Behavioral data collected during XR experiments, such as user movements, gaze, actions, and physiological signals constitute precious assets for analyzing and understanding the user experience. While they contribute to overcome the intrinsic flaws of explicit data such as post-experiment questionnaires, the required acquisition and analysis tools are costly and challenging to develop, especially for 6DoF (Degrees of Freedom) XR experiments. Moreover, there is no common format for XR behavioral data, which restrains data-sharing, and thus hinders wide usages across the community, replicability of studies, and the constitution of large datasets or meta-analysis. In this context, we present PLUME, an open-source software toolbox (PLUME Recorder, PLUME Viewer, PLUME Python) that allows for the exhaustive record of XR behavioral data (including synchronous physiological signals), their offline interactive replay and analysis (with a standalone application), and their easy sharing due to our compact and interoperable data format. We believe that PLUME can greatly benefit the scientific community by making the use of behavioral and physiological data available for the greatest, contributing to the reproducibility and replicability of XR user studies, enabling the creation of large datasets, and contributing to a deeper understanding of user experience.

Toward metacognition: subject-aware contrastive deep fusion representation learning for EEG analysis

We propose a subject-aware contrastive learning deep fusion neural network framework for effectively classifying subjects' confidence levels in the perception of visual stimuli. The framework, called WaveFusion, is composed of lightweight convolutional neural networks for per-lead time-frequency analysis and an attention network for integrating the lightweight modalities for final prediction. To facilitate the training of WaveFusion, we incorporate a subject-aware contrastive learning approach by taking advantage of the heterogeneity within a multi-subject electroencephalogram dataset to boost representation learning and classification accuracy. The WaveFusion framework demonstrates high accuracy in classifying confidence levels by achieving a classification accuracy of 95.7% while also identifying influential brain regions.

Classification and Analysis of Human Body Movement Characteristics Associated with Acrophobia Induced by Virtual Reality Scenes of Heights

Acrophobia (fear of heights), a prevalent psychological disorder, elicits profound fear and evokes a range of adverse physiological responses in individuals when exposed to heights, which will lead to a very dangerous state for people in actual heights. In this paper, we explore the behavioral influences in terms of movements in people confronted with virtual reality scenes of extreme heights and develop an acrophobia classification model based on human movement characteristics. To this end, we used wireless miniaturized inertial navigation sensors (WMINS) network to obtain the information of limb movements in the virtual environment. Based on these data, we constructed a series of data feature processing processes, proposed a system model for the classification of acrophobia and non-acrophobia based on human motion feature analysis, and realized the classification recognition of acrophobia and non-acrophobia through the designed integrated learning model. The final accuracy of acrophobia dichotomous classification based on limb motion information reached 94.64%, which has higher accuracy and efficiency compared with other existing research models. Overall, our study demonstrates a strong correlation between people's mental state during fear of heights and their limb movements at that time.

Multi-Input CNN-LSTM deep learning model for fear level classification based on EEG and peripheral physiological signals

Objective and accurate classification of fear levels is a socially important task that contributes to developing treatments for Anxiety Disorder, Obsessive-compulsive Disorder, Post-Traumatic Stress Disorder (PTSD), and Phobia. This study examines a deep learning model to automatically estimate human fear levels with high accuracy using multichannel EEG signals and multimodal peripheral physiological signals in the DEAP dataset. The Multi-Input CNN-LSTM classification model combining Convolutional Neural Network (CNN) and Long Sort-Term Memory (LSTM) estimated four fear levels with an accuracy of 98.79% and an F1 score of 99.01% in a 10-fold cross-validation. This study contributes to the following; (1) to present the possibility of recognizing fear emotion with high accuracy using a deep learning model from physiological signals without arbitrary feature extraction or feature selection, (2) to investigate effective deep learning model structures for high-accuracy fear recognition and to propose Multi-Input CNN-LSTM, and (3) to examine the model's tolerance to individual differences in physiological signals and the possibility of improving accuracy through additional learning.

Review of Psychophysiological and Psychotherapeutic Studies of Stress Using Virtual Reality Technologies

This review addresses the use of virtual reality technologies in the psychophysiology and psychotherapy of stress. Studies using virtual reality both to introduce subjects into a state of stress and to help reduce stress reactions are reviewed. Methods developed for treating patients suffering from stress-related disorders (in particular, PTSD and phobias) are described. In many cases, reductions in stress reactions with the help of virtual reality systems are achieved not only at the self-report (experiential) level, but also at the level of central and peripheral nervous system measures. This allows virtual reality to be regarded as a modern, inexpensive, and effective method, firstly, for introducing subjects into a state of stress with the aim of testing various hypotheses in psychophysiology and, secondly, to reduce stress reactions.

Virtual reality in the diagnostic and therapy for mental disorders: A systematic review

BACKGROUND

Virtual reality (VR) technologies are playing an increasingly important role in the diagnostics and treatment of mental disorders.

OBJECTIVE

To systematically review the current evidence regarding the use of VR in the diagnostics and treatment of mental disorders.

DATA SOURCE

Systematic literature searches via PubMed (last literature update: 9^th of May 2022) were conducted for the following areas of psychopathology: Specific phobias, panic disorder and agoraphobia, social anxiety disorder, generalized anxiety disorder, posttraumatic stress disorder (PTSD), obsessive-compulsive disorder, eating disorders, dementia disorders, attention-deficit/hyperactivity disorder, depression, autism spectrum disorder, schizophrenia spectrum disorders, and addiction disorders.

ELIGIBILITY CRITERIA

To be eligible, studies had to be published in English, to be peer-reviewed, to report original research data, to be VR-related, and to deal with one of the above-mentioned areas of psychopathology.

STUDY EVALUATION

For each study included, various study characteristics (including interventions and conditions, comparators, major outcomes and study designs) were retrieved and a risk of bias score was calculated based on predefined study quality criteria.

RESULTS

Across all areas of psychopathology, k = 9315 studies were inspected, of which k = 721 studies met the eligibility criteria. From these studies, 43.97% were considered assessment-related, 55.48% therapy-related, and 0.55% were mixed. The highest research activity was found for VR exposure therapy in anxiety disorders, PTSD and addiction disorders, where the most convincing evidence was found, as well as for cognitive trainings in dementia and social skill trainings in autism spectrum disorder.

CONCLUSION

While VR exposure therapy will likely find its way successively into regular patient care, there are also many other promising approaches, but most are not yet mature enough for clinical application.

REVIEW REGISTRATION

PROSPERO register CRD42020188436.

FUNDING

The review was funded by budgets from the University of Bonn. No third party funding was involved.

Detecting Fear of Heights Response to a Virtual Reality Environment Using Functional Near-Infrared Spectroscopy

To enable virtual reality exposure therapy (VRET) that treats anxiety disorders by gradually exposing the patient to fear using virtual reality (VR), it is important to monitor the patient's fear levels during the exposure. Despite the evidence of a fear circuit in the brain as reflected by functional near-infrared spectroscopy (fNIRS), the measurement of fear response in highly immersive VR using fNIRS is limited, especially in combination with a head-mounted display (HMD). In particular, it is unclear to what extent fNIRS can differentiate users with and without anxiety disorders and detect fear response in a highly ecological setting using an HMD. In this study, we investigated fNIRS signals captured from participants with and without a fear of height response. To examine the extent to which fNIRS signals of both groups differ, we conducted an experiment during which participants with moderate fear of heights and participants without it were exposed to VR scenarios involving heights and no heights. The between-group statistical analysis shows that the fNIRS data of the control group and the experimental group are significantly different only in the channel located close to right frontotemporal lobe, where the grand average oxygenated hemoglobin Δ[HbO] contrast signal of the experimental group exceeds that of the control group. The within-group statistical analysis shows significant differences between the grand average Δ[HbO] contrast values during fear responses and those during no-fear responses, where the Δ[HbO] contrast values of the fear responses were significantly higher than those of the no-fear responses in the channels located towards the frontal part of the prefrontal cortex. Also, the channel located close to frontocentral lobe was found to show significant difference for the grand average deoxygenated hemoglobin contrast signals. Support vector machine-based classifier could detect fear responses at an accuracy up to 70% and 74% in subject-dependent and subject-independent classifications, respectively. The results demonstrate that cortical hemodynamic responses of a control group and an experimental group are different to a considerable extent, exhibiting the feasibility and ecological validity of the combination of VR-HMD and fNIRS to elicit and detect fear responses. This research thus paves a way toward the a brain-computer interface to effectively manipulate and control VRET.

AI-Based Prediction and Prevention of Psychological and Behavioral Changes in Ex-COVID-19 Patients

The COVID-19 pandemic has adverse consequences on human psychology and behavior long after initial recovery from the virus. These COVID-19 health sequelae, if undetected and left untreated, may lead to more enduring mental health problems, and put vulnerable individuals at risk of developing more serious psychopathologies. Therefore, an early distinction of such vulnerable individuals from those who are more resilient is important to undertake timely preventive interventions. The main aim of this article is to present a comprehensive multimodal conceptual approach for addressing these potential psychological and behavioral mental health changes using state-of-the-art tools and means of artificial intelligence (AI). Mental health COVID-19 recovery programs at post-COVID clinics based on AI prediction and prevention strategies may significantly improve the global mental health of ex-COVID-19 patients. Most COVID-19 recovery programs currently involve specialists such as pulmonologists, cardiologists, and neurologists, but there is a lack of psychiatrist care. The focus of this article is on new tools which can enhance the current limited psychiatrist resources and capabilities in coping with the upcoming challenges related to widespread mental health disorders. Patients affected by COVID-19 are more vulnerable to psychological and behavioral changes than non-COVID populations and therefore they deserve careful clinical psychological screening in post-COVID clinics. However, despite significant advances in research, the pace of progress in prevention of psychiatric disorders in these patients is still insufficient. Current approaches for the diagnosis of psychiatric disorders largely rely on clinical rating scales, as well as self-rating questionnaires that are inadequate for comprehensive assessment of ex-COVID-19 patients' susceptibility to mental health deterioration. These limitations can presumably be overcome by applying state-of-the-art AI-based tools in diagnosis, prevention, and treatment of psychiatric disorders in acute phase of disease to prevent more chronic psychiatric consequences.

Contributions of diagnostic, cognitive, and somatovisceral information to the prediction of fear ratings in spider phobic and non-spider-fearful individuals

BACKGROUND

Physiological responding is a key characteristic of fear responses. Yet, it is unknown whether the time-consuming measurement of somatovisceral responses ameliorates the prediction of individual fear responses beyond the accuracy reached by the consideration of diagnostic (e.g., phobic vs. non phobic) and cognitive (e.g., risk estimation) factors, which can be more easily assessed.

METHOD

We applied a machine learning approach to data of an experiment, in which spider phobic and non-spider fearful participants (diagnostic factor) faced pictures of spiders. For each experimental trial, participants specified their personal risk of encountering the spider (cognitive factor), as well as their subjective fear (outcome variable) on quasi-continuous scales, while diverse somatovisceral responses were registered (heart rate, electrodermal activity, respiration, facial muscle activity).

RESULTS

The machine-learning analyses revealed that fear ratings were predominantly predictable by the diagnostic factor. Yet, when allowing for learning of individual patterns in the data, somatovisceral responses contributed additional information on the fear ratings, yielding a prediction accuracy of 81% explained variance. Moreover, heart rate prior to picture onset, but not heart rate reactivity increased predictive power.

LIMITATIONS

Fear was solely assessed by verbal reports, only 27 females were considered, and no generalization to other anxiety disorders is possible.

CONCLUSIONS

After training the algorithm to learn about individual-specific responding, somatovisceral patterns can be successfully exploited. Our findings further point to the possibility that the expectancy-related autonomic state throughout the experiment predisposes an individual to experience specific levels of fear, with less influence of the actual visual stimulations.

Heart rate variability analysis for the assessment of immersive emotional arousal using virtual reality: Comparing real and virtual scenarios

Many affective computing studies have developed automatic emotion recognition models, mostly using emotional images, audio and videos. In recent years, virtual reality (VR) has been also used as a method to elicit emotions in laboratory environments. However, there is still a need to analyse the validity of VR in order to extrapolate the results it produces and to assess the similarities and differences in physiological responses provoked by real and virtual environments. We investigated the cardiovascular oscillations of 60 participants during a free exploration of a real museum and its virtualisation viewed through a head-mounted display. The differences between the heart rate variability features in the high and low arousal stimuli conditions were analysed through statistical hypothesis testing; and automatic arousal recognition models were developed across the real and the virtual conditions using a support vector machine algorithm with recursive feature selection. The subjects' self-assessments suggested that both museums elicited low and high arousal levels. In addition, the real museum showed differences in terms of cardiovascular responses, differences in vagal activity, while arousal recognition reached 72.92% accuracy. However, we did not find the same arousal-based autonomic nervous system change pattern during the virtual museum exploration. The results showed that, while the direct virtualisation of a real environment might be self-reported as evoking psychological arousal, it does not necessarily evoke the same cardiovascular changes as a real arousing elicitation. These contribute to the understanding of the use of VR in emotion recognition research; future research is needed to study arousal and emotion elicitation in immersive VR.

Deep learning applied to electroencephalogram data in mental disorders: A systematic review

In recent medical research, tremendous progress has been made in the application of deep learning (DL) techniques. This article systematically reviews how DL techniques have been applied to electroencephalogram (EEG) data for diagnostic and predictive purposes in conducting research on mental disorders. EEG-studies on psychiatric diseases based on the ICD-10 or DSM-V classification that used either convolutional neural networks (CNNs) or long -short-term-memory (LSTMs) networks for classification were searched and examined for the quality of the information they contained in three domains: clinical, EEG-data processing, and deep learning. Although we found that the description of EEG acquisition and pre-processing was sufficient in most of the studies, we found, that many of them lacked a systematic characterization of clinical features. Furthermore, many studies used misguided model selection procedures or flawed testing. It is recommended that the study of psychiatric disorders using DL in the future must improve the quality of clinical data and follow state of the art model selection and testing procedures so as to achieve a higher research standard and head toward a clinical significance.

Ethical Challenges of Using Virtual Environments in the Assessment and Treatment of Psychopathological Disorders

Clinicians are increasingly interested in the potential of virtual environments for research and praxes. Virtual environments include both immersive and non-immersive simulations of everyday activities. Moreover, algorithmic devices and adaptive virtual environments allow clinicians a medium for personalizing technologies to their patients. There is also increasing recognition of social virtual environments that connect virtual environments to social networks. Although there has been a great deal of deliberation on these novel technologies for assessment and treatment, less discourse has occurred around the ethical challenges that may ensue when these technologies are applied clinically. In this paper, some of the ethical issues involved in the clinical use of novel technologies are discussed.

Integrating Biosignals Measurement in Virtual Reality Environments for Anxiety Detection

This paper proposes a protocol for the acquisition and processing of biophysical signals in virtual reality applications, particularly in phobia therapy experiments. This protocol aims to ensure that the measurement and processing phases are performed effectively, to obtain clean data that can be used to estimate the users' anxiety levels. The protocol has been designed after analyzing the experimental data of seven subjects who have been exposed to heights in a virtual reality environment. The subjects' level of anxiety has been estimated based on the real-time evaluation of a nonlinear function that has as parameters various features extracted from the biophysical signals. The highest classification accuracy was obtained using a combination of seven heart rate and electrodermal activity features in the time domain and frequency domain.

Significant Applications of Machine Learning for COVID-19 Pandemic

Machine learning is an innovative approach that has extensive applications in prediction. This technique needs to be applied for the COVID-19 pandemic to identify patients at high risk, their death rate, and other abnormalities. It can be used to understand the nature of this virus and further predict the upcoming issues. This literature-based review is done by searching the relevant papers on machine learning for COVID-19 from the databases of SCOPUS, Academia, Google Scholar, PubMed, and ResearchGate. This research attempts to discuss the significance of machine learning in resolving the COVID-19 pandemic crisis. This paper studied how machine learning algorithms and methods can be employed to fight the COVID-19 virus and the pandemic. It further discusses the primary machine learning methods that are helpful during the COVID-19 pandemic. We further identified and discussed algorithms used in machine learning and their significant applications. Machine learning is a useful technique, and this can be witnessed in various areas to identify the existing drugs, which also seems advantageous for the treatment of COVID-19 patients. This learning algorithm creates interferences out of unlabeled input datasets, which can be applied to analyze the unlabeled data as an input resource for COVID-19. It provides accurate and useful features rather than a traditional explicitly calculation-based method. Further, this technique is beneficial to predict the risk in healthcare during this COVID-19 crisis. Machine learning also analyses the risk factors as per age, social habits, location, and climate.

Emotion Recognition in Immersive Virtual Reality: From Statistics to Affective Computing

Emotions play a critical role in our daily lives, so the understanding and recognition of emotional responses is crucial for human research. Affective computing research has mostly used non-immersive two-dimensional (2D) images or videos to elicit emotional states. However, immersive virtual reality, which allows researchers to simulate environments in controlled laboratory conditions with high levels of sense of presence and interactivity, is becoming more popular in emotion research. Moreover, its synergy with implicit measurements and machine-learning techniques has the potential to impact transversely in many research areas, opening new opportunities for the scientific community. This paper presents a systematic review of the emotion recognition research undertaken with physiological and behavioural measures using head-mounted displays as elicitation devices. The results highlight the evolution of the field, give a clear perspective using aggregated analysis, reveal the current open issues and provide guidelines for future research.

Development and Calibration of an Eye-Tracking Fixation Identification Algorithm for Immersive Virtual Reality

Fixation identification is an essential task in the extraction of relevant information from gaze patterns; various algorithms are used in the identification process. However, the thresholds used in the algorithms greatly affect their sensitivity. Moreover, the application of these algorithm to eye-tracking technologies integrated into head-mounted displays, where the subject's head position is unrestricted, is still an open issue. Therefore, the adaptation of eye-tracking algorithms and their thresholds to immersive virtual reality frameworks needs to be validated. This study presents the development of a dispersion-threshold identification algorithm applied to data obtained from an eye-tracking system integrated into a head-mounted display. Rules-based criteria are proposed to calibrate the thresholds of the algorithm through different features, such as number of fixations and the percentage of points which belong to a fixation. The results show that distance-dispersion thresholds between 1-1.6° and time windows between 0.25-0.4 s are the acceptable range parameters, with 1° and 0.25 s being the optimum. The work presents a calibrated algorithm to be applied in future experiments with eye-tracking integrated into head-mounted displays and guidelines for calibrating fixation identification algorithms.