The Current Research of Spatial Cognitive Evaluation and Training With Brain-Computer Interface and Virtual Reality

Virtual Reality and Spatial Augmented Reality for Social Inclusion: The “Includiamoci” Project

Extended Reality (XR) technology represents an innovative tool to address the challenges of the present, as it allows for experimentation with new solutions in terms of content creation and its fruition by different types of users. The potential to modulate the experience based on the target audience’s needs and the project’s objectives makes XR suitable for creating new accessibility solutions. The “Includiamoci” project was carried out with the aim of creating workshops on social inclusion through the combination of art and technology. Specifically, the experimentation involved ten young people between the ages of 28 and 50, with cognitive disabilities, who participated in Extended Reality workshops and Art Therapy workshops. In the course of these activities, the outputs obtained were two: a virtual museum, populated by the participants’ works, and a digital set design for a theatrical performance. Through two tests, one on user experience (UX) and one on the degree of well-being, the effectiveness of the entire project was evaluated. In conclusion, the project demonstrated how the adopted solutions were appropriate to the objectives, increasing our knowledge of UX for a target audience with specific user needs and using XR in the context of social inclusion.

Linking cognitive functioning and postural balance control through virtual reality environmental manipulations

Background

Dementia is becoming a relevant problem worldwide. A simple screening at an early stage will be important to detect the risk of developing dementia. Vestibular dysfunction is likely to be associated with cognitive impairment. Since head-mounted display (HMD) virtual reality (VR) technology has the potential to activate the vestibular function, assessing postural sway with visual stimulation using HMD VR technology could be potentially useful for dementia screening.

Objective

The purpose of this study is to evaluate the effect of HMD-based VR visual stimuli on posture in older adults and the relationship between the stimulated body sway behaviors and cognitive performance.

Method

Using a cross-sectional study design, we investigated the effect of an optokinetic design-based room with stripes (OKR) VR environment oscillating forwards and backwards at 23/60Hz. Center of pressure (COP) displacement was measured in older adults aged 65 years and over in the OKR VR environment. The frequency response of COP was compared to the cognitive performance of the Montreal Cognitive Assessment (MoCA).

Results

20 healthy older adults (70.4 ± 4.9 years; 27.2 ± 1.6 MoCA score) and 3 people with mild cognitive impairment (74.7 ± 4.0 years; 20.3 ± 2.1 MoCA score) were assessed. The results reveal that the oscillating OKR VR environment induced different postural sway in the anterior-posterior direction in the real world. Correlation analysis shows that the cognitive test score was associated with the frequency response of stimulated postural sway in the anterior-posterior direction (frequency Band 1 of 0−0.5Hz related to the visual and vestibular systems: r_s = 0.45, P = 0.03).

Conclusion

Outcomes would suggest that a potential link may emerge between cognition and posture when the HMD-based VR visual stimuli are applied. The simple screening of stimulated postural sway could explain cognitive functioning. Further studies are warranted to clarify the vestibular system and spatial cognitive function more specifically in the proposed assessment system.

Task-state EEG signal classification for spatial cognitive evaluation based on multi-scale high-density convolutional neural network

In this study, a multi-scale high-density convolutional neural network (MHCNN) classification method for spatial cognitive ability assessment was proposed, aiming at achieving the binary classification of task-state EEG signals before and after spatial cognitive training. Besides, the multi-dimensional conditional mutual information method was used to extract the frequency band features of the EEG data. And the coupling features under the combination of multi-frequency bands were transformed into multi-spectral images. At the same time, the idea of Densenet was introduced to improve the multi-scale convolutional neural network. Firstly, according to the discreteness of multispectral EEG image features, two-scale convolution kernels were used to calculate and learn useful channel and frequency band feature information in multispectral image data. Secondly, to enhance feature propagation and reduce the number of parameters, the dense network was connected after the multi-scale convolutional network, and the learning rate change function of the stochastic gradient descent algorithm was optimized to objectively evaluate the training effect. The experimental results showed that compared with the classical convolution neural network (CNN) and multi-scale convolution neural network, the proposed MHCNN had better classification performance in the six frequency band combinations with the highest accuracy of 98%: Theta-Alpha2-Gamma, Alpha2-Beta2-Gamma, Beta1-Beta2-Gamma, Theta-Beta2-Gamma, Theta-Alpha1-Gamma, and Alpha1-Alpha2-Gamma. By comparing the classification results of six frequency band combinations, it was found that the combination of the Theta-Beta2-Gamma band had the best classification effect. The MHCNN classification method proposed in this research could be used as an effective biological indicator of spatial cognitive training effect and could be extended to other brain function evaluations.

The Role of Virtual Reality in Screening, Diagnosing, and Rehabilitating Spatial Memory Deficits

Impairment of spatial memory, including an inability to recall previous locations and navigate the world, is often one of the first signs of functional disability on the road to cognitive impairment. While there are many screening and diagnostic tools which attempt to measure spatial memory ability, they are often not representative of real-life situations and can therefore lack applicability. One potential solution to this problem involves the use of virtual reality (VR), which immerses individuals in a virtually-simulated environment, allowing for scenarios more representative of real-life without any of the associated risks. Here, we review the evidence surrounding the use of VR for the screening and diagnosis of spatial memory impairments, including potential limitations and how it compares to standard neuropsychological tests. We will also discuss the evidence regarding the potential use of VR in the rehabilitation of spatial memory deficits, which has not been well studied, but which could be game-changing if proven successful.

The Current Research of Combining Multi-Modal Brain-Computer Interfaces With Virtual Reality

Combing brain-computer interfaces (BCI) and virtual reality (VR) is a novel technique in the field of medical rehabilitation and game entertainment. However, the limitations of BCI such as a limited number of action commands and low accuracy hinder the widespread use of BCI-VR. Recent studies have used hybrid BCIs that combine multiple BCI paradigms and/or the multi-modal biosensors to alleviate these issues, which may become the mainstream of BCIs in the future. The main purpose of this review is to discuss the current status of multi-modal BCI-VR. This study first reviewed the development of the BCI-VR, and explored the advantages and disadvantages of incorporating eye tracking, motor capture, and myoelectric sensing into the BCI-VR system. Then, this study discussed the development trend of the multi-modal BCI-VR, hoping to provide a pathway for further research in this field.

The EEG Signal Analysis for Spatial Cognitive Ability Evaluation Based on Multivariate Permutation Conditional Mutual Information-Multi-Spectral Image

This study aims to find an effective method to evaluate the efficacy of cognitive training of spatial memory under a virtual reality environment, by classifying the EEG signals of subjects in the early and late stages of spatial cognitive training. This study proposes a new EEG signal analysis method based on Multivariate Permutation Conditional Mutual Information-Multi-Spectral Image (MPCMIMSI). This method mainly considers the relationship between the coupled features of EEG signals in different channel pairs and transforms the multivariate permutation conditional mutual information features into multi-spectral images. Then, a convolutional neural networks (CNN) model classifies the resultant image data into different stages of cognitive training to objectively assess the efficacy of the training. Compared to the multi-spectral image transformation method based on Granger causality analysis (GCA) and permutation conditional mutual information (PCMI), the MPCMIMSI led to better classification performance, which can be as high as 95% accuracy. More specifically, the Theta-Beta2-Gamma-band combination has the best accuracy. The proposed MPCMIMSI method outperforms the multi-spectral image transformation methods based on GCA and PCMI in terms of classification performance. The MPCMIMSI feature in the Theta-Beta2-Gamma band is an effective biomarker for assessing the efficacy of spatial memory training. The proposed EEG feature-extraction method based on MPCMIMSI offers a new window to characterize spatial information of the noninvasive EEG recordings and might apply to assessing other brain functions.