Frequency Domain Channel-Wise Attack to CNN Classifiers in Motor Imagery Brain-Computer Interfaces

OBJECTIVE

Convolutional neural network (CNN), a classical structure in deep learning, has been commonly deployed in the motor imagery brain-computer interface (MIBCI). Many methods have been proposed to evaluate the vulnerability of such CNN models, primarily by attacking them using direct temporal perturbations. In this work, we propose a novel attacking approach based on perturbations in the frequency domain instead.

METHODS

For a given natural MI trial in the frequency domain, the proposed approach, called frequency domain channel-wise attack (FDCA), generates perturbations at each channel one after another to fool the CNN classifiers. The advances of this strategy are two-fold. First, instead of focusing on the temporal domain, perturbations are generated in the frequency domain where discriminative patterns can be extracted for motor imagery (MI) classification tasks. Second, the perturbing optimization is performed based on differential evolution algorithm in a black-box scenario where detailed model knowledge is not required.

RESULTS

Experimental results demonstrate the effectiveness of the proposed FDCA which achieves a significantly higher success rate than the baselines and existing methods in attacking three major CNN classifiers on four public MI benchmarks.

CONCLUSION

Perturbations generated in the frequency domain yield highly competitive results in attacking MIBCI deployed by CNN models even in a black-box setting, where the model information is well-protected.

SIGNIFICANCE

To our best knowledge, existing MIBCI attack approaches are all gradient-based methods and require details about the victim model, e.g., the parameters and objective function. We provide a more flexible strategy that does not require model details but still produces an effective attack outcome.

Altered grid-like coding in early blind people

Cognitive maps in the hippocampal-entorhinal system are central for the representation of both spatial and non-spatial relationships. Although this system, especially in humans, heavily relies on vision, the role of visual experience in shaping the development of cognitive maps remains largely unknown. Here, we test sighted and early blind individuals in both imagined navigation in fMRI and real-world navigation. During imagined navigation, the Human Navigation Network, constituted by frontal, medial temporal, and parietal cortices, is reliably activated in both groups, showing resilience to visual deprivation. However, neural geometry analyses highlight crucial differences between groups. A 60° rotational symmetry, characteristic of a hexagonal grid-like coding, emerges in the entorhinal cortex of sighted but not blind people, who instead show a 90° (4-fold) symmetry, indicative of a square grid. Moreover, higher parietal cortex activity during navigation in blind people correlates with the magnitude of 4-fold symmetry. In sum, early blindness can alter the geometry of entorhinal cognitive maps, possibly as a consequence of higher reliance on parietal egocentric coding during navigation.

Fatigue in children using motor imagery and P300 brain-computer interfaces

BACKGROUND

Brain-computer interface (BCI) technology offers children with quadriplegic cerebral palsy unique opportunities for communication, environmental exploration, learning, and game play. Research in adults demonstrates a negative impact of fatigue on BCI enjoyment, while effects on BCI performance are variable. To date, there have been no pediatric studies of BCI fatigue. The purpose of this study was to assess the effects of two different BCI paradigms, motor imagery and visual P300, on the development of self-reported fatigue and an electroencephalography (EEG) biomarker of fatigue in typically developing children.

METHODS

Thirty-seven typically-developing school-aged children were recruited to a prospective, crossover study. Participants attended three sessions: (A) motor imagery-BCI, (B) visual P300-BCI, and (C) video viewing (control). The motor imagery task involved an imagined left- or right-hand squeeze. The P300 task involved attending to one square on a 3 × 3 grid during a random single flash sequence. Each paradigm had respective calibration periods and a similar visual counting game. Primary outcomes were self-reported fatigue and the power of the EEG alpha band both collected during resting-state periods pre- and post-task. Self-reported fatigue was measured using a 10-point visual analog scale. EEG alpha band power was calculated as the integrated power spectral density from 8 to 12 Hz of the EEG spectrum.

RESULTS

Thirty-two children completed the protocol (age range 7-16, 63% female). Self-reported fatigue and EEG alpha band power increased across all sessions (F(1,155) = 33.9, p < 0.001; F = 5.0(1,149), p = 0.027 respectively). No differences in fatigue development were observed between session types. There was no correlation between self-reported fatigue and EEG alpha band power change. BCI performance varied between participants and paradigms as expected but was not associated with self-reported fatigue or EEG alpha band power.

CONCLUSION

Short periods (30-mintues) of BCI use can increase self-reported fatigue and EEG alpha band power to a similar degree in children performing motor imagery and P300 BCI paradigms. Performance was not associated with our measures of fatigue; the impact of fatigue on useability and enjoyment is unclear. Our results reflect the variability of fatigue and the BCI experience more broadly in children and warrant further investigation.

Effects of finger pinch motor imagery on short-latency afferent inhibition and corticospinal excitability

Motor imagery is a cognitive process involving the simulation of motor actions without actual movements. Despite the reported positive effects of motor imagery training on motor function, the underlying neurophysiological mechanisms have yet to be fully elucidated. Therefore, the purpose of the present study was to investigate how sustained tonic finger-pinching motor imagery modulates sensorimotor integration and corticospinal excitability using short-latency afferent inhibition (SAI) and single-pulse transcranial magnetic stimulation (TMS) assessments, respectively. Able-bodied individuals participated in the study and assessments were conducted under two experimental conditions in a randomized order between participants: (1) participants performed motor imagery of a pinch task while observing a visual image displayed on a monitor (Motor Imagery), and (2) participants remained at rest with their eyes fixed on the monitor displaying a cross mark (Control). For each condition, sensorimotor integration and corticospinal excitability were evaluated during sustained tonic motor imagery in separate sessions. Sensorimotor integration was assessed by SAI responses, representing inhibition of motor-evoked potentials (MEPs) in the first dorsal interosseous muscle elicited by TMS following median nerve stimulation. Corticospinal excitability was assessed by MEP responses elicited by single-pulse TMS. There was no significant difference in the magnitude of SAI responses between motor imagery and Control conditions, while MEP responses were significantly facilitated during the Motor Imagery condition compared to the Control condition. These findings suggest that motor imagery facilitates corticospinal excitability, without altering sensorimotor integration, possibly due to insufficient activation of the somatosensory circuits or lack of afferent feedback during sustained tonic motor imagery.

Unraveling motor imagery brain patterns using explainable artificial intelligence based on Shapley values

BACKGROUND AND OBJECTIVE

Motor imagery (MI) based brain-computer interfaces (BCIs) are widely used in rehabilitation due to the close relationship that exists between MI and motor execution (ME). However, the underlying brain mechanisms of MI remain not well understood. Most MI-BCIs use the sensorimotor rhythms elicited in the primary motor cortex (M1) and somatosensory cortex (S1), which consist of an event-related desynchronization followed by an event-related synchronization. Consequently, this has resulted in systems that only record signals around M1 and S1. However, MI could involve a more complex network including sensory, association, and motor areas. In this study, we hypothesize that the superior accuracies achieved by new deep learning (DL) models applied to MI decoding rely on focusing on a broader MI activation of the brain. Parallel to the success of DL, the field of explainable artificial intelligence (XAI) has seen continuous development to provide explanations for DL networks success. The goal of this study is to use XAI in combination with DL to extract information about MI brain activation patterns from non-invasive electroencephalography (EEG) signals.

METHODS

We applied an adaptation of Shapley additive explanations (SHAP) to EEGSym, a state-of-the-art DL network with exceptional transfer learning capabilities for inter-subject MI classification. We obtained the SHAP values from two public databases comprising 171 users generating left and right hand MI instances with and without real-time feedback.

RESULTS

We found that EEGSym based most of its prediction on the signal of the frontal electrodes, i.e. F7 and F8, and on the first 1500 ms of the analyzed imagination period. We also found that MI involves a broad network not only based on M1 and S1, but also on the prefrontal cortex (PFC) and the posterior parietal cortex (PPC). We further applied this knowledge to select a 8-electrode configuration that reached inter-subject accuracies of 86.5% ± 10.6% on the Physionet dataset and 88.7% ± 7.0% on the Carnegie Mellon University's dataset.

CONCLUSION

Our results demonstrate the potential of combining DL and SHAP-based XAI to unravel the brain network involved in producing MI. Furthermore, SHAP values can optimize the requirements for out-of-laboratory BCI applications involving real users.

No verbal overshadowing in aphantasia: The role of visual imagery for the verbal overshadowing effect

The verbal overshadowing effect refers to the phenomenon that the verbal description of a past complex stimulus impairs its subsequent recognition. Theoretical explanations range from interference between different mental representations to the activation of different processing orientations or a provoked shift in the recognition criterion. In our study, 61 participants with aphantasia (= lack of mental imagery) and 70 controls participated in a verbal overshadowing paradigm. The verbal overshadowing effect did not occur in people with aphantasia, although the effect was replicated in controls. We speculate that this is either due to the lack of visual representations in people with aphantasia that verbal descriptions could interfere with, or to the absence of a shift in processing orientation during verbalisation. To rule out criterion-based explanations, further research is needed to distinguish between discriminability and response bias in people with aphantasia. Finally, data indicated that the verbal overshadowing effect may even be reversed in individuals with aphantasia, partly due to a lower memory performance in the no verbalisation condition. Effects of further variables are discussed, such as mental strategies, memory confidence, and difficulty, quantity and quality of verbalisation.

Reality monitoring and metacognitive judgments in a false-memory paradigm

How well do we distinguish between different memory sources when the information from imagination and perception is similar? And how do metacognitive (confidence) judgments differ across different sources of experiences? To study these questions, we developed a reality monitoring task using semantically related words from the Deese-Roediger-McDermott (DRM) paradigm of false memories. In an orientation phase, participants either perceived word pairs or had to voluntarily imagine the second word of a word pair. In a test phase, participants viewed words and had to judge whether the paired word was previously perceived, imagined, or new. Results revealed an interaction between memory source and judgment type on both response rates and confidence judgments: reality monitoring was better for new and perceived (compared to imagined) sources, and participants often incorrectly reported imagined experiences to be perceived. Individuals exhibited similar confidence between correct imagined source judgments and incorrect imagined sources reported to be perceived. Modeling results indicated that the observed judgments were likely due to an externalizing bias (i.e., a bias to judge the memory source as perceived). Additionally, we found that overall metacognitive ability was best in the perceived source. Together, these results reveal a source-dependent effect on response rates and confidence ratings, and provide evidence that observers are surprisingly prone to externalizing biases when monitoring their own memories.

"The mere imagination scares me"-evidence for fear responses during mental imagery of pain-associated interoceptive sensations in adolescents with chronic pain

ABSTRACT

According to the bio-informational theory of emotion by Lang, mental imagery of fearful stimuli activates physiological and behavioural response systems, even in the absence of sensory input. We investigated whether instructed mental imagery of pain-associated (not painful) interoceptive sensations entails a threat value and elicits increased startle response, skin conductance level (SCL), and heart rate (HR) indicative of defensive mobilization in adolescents with chronic pain. Additionally, self-reported measures (fear, fear of pain, desire to avoid) were assessed. Adolescents (11-18 years) with chronic headache (CH, n = 46) or chronic abdominal pain (CAP, n = 29) and a control group (n = 28) were asked to imagine individualized pain-associated, neutral and standardized fear scripts. During pain-associated compared with neutral imagery, both pain groups showed higher mean HR, with CH also showing higher HR reactivity, while HR acceleration was not observed within control group. In contrast, during pain-associated compared with neutral imagery, startle response magnitude and SCL remained unchanged in all groups. Additionally, overall levels in self-reports were higher during pain-associated compared with neutral imagery, but significantly more pronounced in the pain groups compared with the control group. Results suggest that the mere imagination of pain-associated sensations elicits specific autonomic fear responses accompanied by increased self-reported fear in adolescents with chronic pain. The specific modulation of heart rate shed new light on our understanding of multimodal fear responses in adolescents with chronic pain and may help to refine paradigms to decrease fear of interoceptive sensations in chronic pain.

Can the prosocial benefits of episodic simulation transfer to different people and situational contexts?

Previous research has found that episodic simulation of events of helping others can effectively enhance intentions to help the same person involved and the identical situational context as the imagined scenarios. This 'prosocial simulation effect' is argued to reflect, at least in part, associative memory mechanisms whereby the simulation is reactivated when in the same situation as that imagined. However, to date, no study has examined systematically whether this 'prosocial simulation effect' can be transferred to response scenarios involving different people and/or situational contexts to the imagined scenarios, and if so, whether the degree of overlap with the imagined helping episode modulated the transfer effect. Across two experiments, we systematically varied the overlap of the simulated and response scenarios, both in terms of the persons in need and/or the situational contexts, and whether would influence the magnitude of prosocial simulation effect. Results from both experiments showed that the prosocial simulation effect can be transferred to response scenarios involving different people and situational contexts to the simulated scenarios. However, this finding was primarily driven by response scenarios that had a high degree of overlap to the simulated scenarios. The application of our findings to the practical implementation of simulation to promote prosociality in the real world is discussed.

Observing an expert's action swapped with an observer's face increases corticospinal excitability during combined action observation and motor imagery

This study aimed to examine whether observing an expert's action swapped with an observer's face increases corticospinal excitability during combined action observation and motor imagery (AOMI). Twelve young males performed motor imagery of motor tasks with different difficulties while observing the actions of an expert performer and an expert performer with a swapped face. Motor tasks included bilateral wrist dorsiflexion (EASY) and unilateral two-ball rotating motions (DIFF). During the AOMI of EASY and DIFF, single-pulse transcranial magnetic stimulation was delivered to the left primary motor cortex, and motor-evoked potentials (MEPs) were obtained from the extensor carpi ulnaris and first dorsal interosseous muscles of the right upper limb, respectively. Visual analogue scale (VAS) assessed the subjective similarity of the expert performer with the swapped face in the EASY and DIFF to the participants themselves. The MEP amplitude in DIFF was larger in the observation of the expert performer with the swapped face than that of the expert performer (P = 0.012); however, the corresponding difference was not observed in EASY (P = 1.000). The relative change in the MEP amplitude from observing the action of the expert performer to that of the expert performer with the swapped face was positively correlated with VAS only in DIFF (r = 0.644, P = 0.024). These results indicate that observing the action of an expert performer with the observer's face enhances corticospinal excitability during AOMI, depending on the task difficulty and subjective similarity between the expert performer being observed and the observer.

Cortical activation and BCI performance during brief tactile imagery: A comparative study with motor imagery

Brain-computer interfaces (BCIs) rely heavily on motor imagery (MI) for operation, yet tactile imagery (TI) presents a novel approach that may be advantageous in situations where visual feedback is impractical. The current study aimed to compare the cortical activity and digit classification performance induced by TI and MI to assess the viability of TI for use in BCIs. Twelve right-handed participants engaged in trials of TI and MI, focusing on their left and right index digits. Event-related desynchronization (ERD) in the mu and beta bands was analyzed, and classification accuracy was determined through an artificial neural network (ANN). Comparable ERD patterns were observed in both TI and MI, with significant decreases in ERD during imagery tasks. The ANN demonstrated high classification accuracy, with TI achieving a mean±SD of 79.30 ± 3.91 % and MI achieving 81.10 ± 2.96 %, with no significant difference between the two (p = 0.11). The study found that TI induces substantial ERD comparable to MI and maintains high classification accuracy, supporting its potential as an effective mental strategy for BCIs. This suggests that TI could be a valuable alternative in BCI applications, particularly for individuals unable to rely on visual cues.

Rapid-IAF: Rapid Identification of Individual Alpha Frequency in EEG Data Using Sequential Bayesian Estimation

Rapid and robust identification of the individual alpha frequency (IAF) in electroencephalogram (EEG) is an essential factor for successful brain-computer interface (BCI) use. Here we demonstrate an algorithm to determine the IAF from short-term resting-state scalp EEG data. First, we outlined the algorithm to determine IAF from short-term resting scalp EEG data and evaluated its reliability using a large-scale dataset of scalp EEG during motor imagery-based BCI use and independent dataset for generalizability confirmation (N = 147). Next, we characterized the relationship between IAF and responsive frequency band of sensorimotor rhythm, which exhibits prominent event-related desynchronization (SMR-ERD) while attempting unilateral and movement. The proposed sequential Bayesian estimation algorithm (Rapid-IAF) determined IAF from less than 26-second resting EEG data among 95% of participants, indicating a clear advance over the conventional methods, which uses 2-15 minutes of data in previous literatures. We confirmed that the determined IAF corresponded to the frequency of SMR, which exhibits the most prominent event-related desynchronization during BCI use (individual SMR-ERD frequency, ISF). Moreover, intraclass correlation revealed that the estimated IAF was more stable than ISF across sessions, suggesting its reliability and utility for robust BCI use without intermittent recalibration. In summary, our method rapidly and reliably determined IAF compared to the conventional method using the spectral power change based on task-related response. The method can be utilized to quick BCI initialization. The demonstration of rapid, task-free parametrization of individual variability of neural responses would be of importance for future BCI systems including neural communication via a cursor, an avatar or robots, and closed-loop neurofeedback training.

Health anxiety amplifies fearful responses to illness-related imagery

Severe health anxiety (HA) is characterized by excessive worry and anxiety about one's health, often accompanied by distressing intrusive imagery of signs of a serious illness or potentially receiving bad news about having a life-threatening disease. However, the emotional responses to these illness-related mental images in relation to HA have not been fully elucidated. Emotional responses to mental imagery of 142 participants were assessed in a well-controlled script-driven imagery task, systematically comparing emotional responses to illness-related imagery with neutral and standard fear imagery. The results revealed that participants reported higher anxiety, aversion, emotional arousal, and a stronger avoidance tendency during imagery of fear and illness-related scenes compared to neutral scenes. Importantly, the emotional modulation varied by the level of HA, indicating that individuals with higher HA experienced stronger emotional responses to illness-related imagery. This association between HA and fearful imagery could not be better accounted for by other psychological factors such as trait anxiety, anxiety sensitivity, somatic symptom severity, or symptoms of depression and anxiety. Fearful responding to standard threat material was not associated with HA. The present findings highlight the importance of considering fear responding to mental imagery in understanding and addressing HA.

Cross-Subject Motor Imagery Decoding by Transfer Learning of Tactile ERD

For Brain-Computer Interface (BCI) based on motor imagery (MI), the MI task is abstract and spontaneous, presenting challenges in measurement and control and resulting in a lower signal-to-noise ratio. The quality of the collected MI data significantly impacts the cross-subject calibration results. To address this challenge, we introduce a novel cross-subject calibration method based on passive tactile afferent stimulation, in which data induced by tactile stimulation is utilized to calibrate transfer learning models for cross-subject decoding. During the experiments, tactile stimulation was applied to either the left or right hand, with subjects only required to sense tactile stimulation. Data from these tactile tasks were used to train or fine-tune models and subsequently applied to decode pure MI data. We evaluated BCI performance using both the classical Common Spatial Pattern (CSP) combined with the Linear Discriminant Analysis (LDA) algorithm and a state-of-the-art deep transfer learning model. The results demonstrate that the proposed calibration method achieved decoding performance at an equivalent level to traditional MI calibration, with the added benefit of outperforming traditional MI calibration with fewer trials. The simplicity and effectiveness of the proposed cross-subject tactile calibration method make it valuable for practical applications of BCI, especially in clinical settings.

Boosting lower-limb motor imagery performance through an ensemble method for gait rehabilitation

Lower-limb exoskeletons have been used extensively in many rehabilitation applications to assist disabled people with their therapies. Brain-machine interfaces (BMIs) further provide effective and natural control schemes. However, the limited performance of brain signal decoding from lower-limb kinematics restricts the broad growth of both BMI and rehabilitation industry. To address these challenges, we propose an ensemble method for lower-limb motor imagery (MI) classification. The proposed model employs multiple techniques to boost performance, including deep and shallow parts. Traditional wavelet transformation followed by filter-bank common spatial pattern (CSP) employs neurophysiologically reasonable patterns, while multi-head self-attention (MSA) followed by temporal convolutional network (TCN) extracts deeper encoded generalized patterns. Experimental results in a customized lower-limb exoskeleton on 8 subjects in 3 consecutive sessions showed that the proposed method achieved 60.27% and 64.20% for three (MI of left leg, MI of right leg, and rest) and two classes (lower-limb MI vs. rest), respectively. Besides, the proposed model achieves improvements of up to 4% and 2% accuracy for the subject-specific and subject-independent modes compared to the current state-of-the-art (SOTA) techniques, respectively. Finally, feature analysis was conducted to show discriminative brain patterns in each MI task and sessions with different feedback modalities. The proposed models integrated in the brain-actuated lower-limb exoskeleton established a potential BMI for gait training and neuroprosthesis.

Fractal Dimension as a discriminative feature for high accuracy classification in motor imagery EEG-based brain-computer interface

BACKGROUND AND OBJECTIVE

The brain-computer interface (BCI) technology acquires human brain electrical signals, which can be effectively and successfully used to control external devices, potentially supporting subjects suffering from motor impairments in the interaction with the environment. To this aim, BCI systems must correctly decode and interpret neurophysiological signals reflecting the intention of the subjects to move. Therefore, the accurate classification of single events in motor tasks represents a fundamental challenge in ensuring efficient communication and control between users and BCIs. Movement-associated changes in electroencephalographic (EEG) sensorimotor rhythms, such as event-related desynchronization (ERD), are well-known features of discriminating motor tasks. Fractal dimension (FD) can be used to evaluate the complexity and self-similarity in brain signals, potentially providing complementary information to frequency-based signal features.

METHODS

In the present work, we introduce FD as a novel feature for subject-independent event classification, and we test several machine learning (ML) models in behavioural tasks of motor imagery (MI) and motor execution (ME).

RESULTS

Our results show that FD improves the classification accuracy of ML compared to ERD. Furthermore, unilateral hand movements have higher classification accuracy than bilateral movements in both MI and ME tasks.

CONCLUSIONS

These results provide further insights into subject-independent event classification in BCI systems and demonstrate the potential of FD as a discriminative feature for EEG signals.

Self-monitoring deficits in schizophrenia: A cross-sectional study of the underlying cognitive mechanisms

BACKGROUND

People diagnosed with schizophrenia share underlying cognitive deficits in self-monitoring (i.e., identifying the source of self-generated behaviours). This study aimed to investigate whether self-monitoring deficits in schizophrenia are due to a cognitive response bias towards external perceptions or a reduced discriminability of imagined and performed actions. We hypothesised that self-monitoring deficits in individuals with schizophrenia are primarily driven by bottom-up processes, leading to a compromised ability to discriminate between internally generated behaviours as opposed to a cognitive response bias towards performed actions.

METHODS

We recruited 333 participants, including 192 with schizophrenia and 141 healthy controls. As part of the Action-Memory Task, participants were instructed to either imagine or physically perform 36 different actions, half of which were presented as pictograms and half as text. In the test phase, participants indicated whether they had performed or imagined each action, whether it appeared in text or pictogram, or whether it was a new action. Using Signal Detection Theory, the study primarily analysed group differences in discriminability and response-bias.

RESULTS

Participants with schizophrenia made significantly more self-monitoring errors than healthy controls. This was primarily due to significantly lower sensitivity, but not a response bias. Whereas recognition memory errors were driven by both lower sensitivity and a response bias.

CONCLUSIONS

The findings suggest that self-monitoring in schizophrenia was specifically impaired by a compromised discriminability of imagined and performed events and an inability to appropriately compensate by adjusting decision-thresholds. Implications on the role of bottom-up and top-down cognitive mechanisms are discussed.

Learning beyond sensations: How dreams organize neuronal representations

Semantic representations in higher sensory cortices form the basis for robust, yet flexible behavior. These representations are acquired over the course of development in an unsupervised fashion and continuously maintained over an organism's lifespan. Predictive processing theories propose that these representations emerge from predicting or reconstructing sensory inputs. However, brains are known to generate virtual experiences, such as during imagination and dreaming, that go beyond previously experienced inputs. Here, we suggest that virtual experiences may be just as relevant as actual sensory inputs in shaping cortical representations. In particular, we discuss two complementary learning principles that organize representations through the generation of virtual experiences. First, "adversarial dreaming" proposes that creative dreams support a cortical implementation of adversarial learning in which feedback and feedforward pathways engage in a productive game of trying to fool each other. Second, "contrastive dreaming" proposes that the invariance of neuronal representations to irrelevant factors of variation is acquired by trying to map similar virtual experiences together via a contrastive learning process. These principles are compatible with known cortical structure and dynamics and the phenomenology of sleep thus providing promising directions to explain cortical learning beyond the classical predictive processing paradigm.

Motor cortex activation during visuomotor transformations: evoked potentials during overt and imagined movements

Despite the prevalence of visuomotor transformations in our motor skills, their mechanisms remain incompletely understood, especially when imagery actions are considered such as mentally picking up a cup or pressing a button. Here, we used a stimulus-response task to directly compare the visuomotor transformation underlying overt and imagined button presses. Electroencephalographic activity was recorded while participants responded to highlights of the target button while ignoring the second, non-target button. Movement-related potentials (MRPs) and event-related desynchronization occurred for both overt movements and motor imagery (MI), with responses present even for non-target stimuli. Consistent with the activity accumulation model where visual stimuli are evaluated and transformed into the eventual motor response, the timing of MRPs matched the response time on individual trials. Activity-accumulation patterns were observed for MI, as well. Yet, unlike overt movements, MI-related MRPs were not lateralized, which appears to be a neural marker for the distinction between generating a mental image and transforming it into an overt action. Top-down response strategies governing this hemispheric specificity should be accounted for in future research on MI, including basic studies and medical practice.

Colors in the mind's eye

The famous "Piazza del Duomo" paper, published in Cortex in 1978, inspired a considerable amount of research on visual mental imagery in brain-damaged patients. As a consequence, single-case reports featuring dissociations between perceptual and imagery abilities challenged the prevailing model of visual mental imagery. Here we focus on mental imagery for colors. A case study published in Cortex showed perfectly preserved color imagery in a patient with acquired achromatopsia after bilateral lesions at the borders between the occipital and temporal cortex. Subsequent neuroimaging findings in healthy participants extended and specified this result; color imagery elicited activation in both a domain-general region located in the left fusiform gyrus and the anterior color-biased patch within the ventral temporal cortex, but not in more posterior color-biased patches. Detailed studies of individual neurological patients, as those often published in Cortex, are still critical to inspire and constrain neurocognitive research and its theoretical models.

The relationship between attention networks and individual differences in visual mental imagery vividness - An EEG study

Previous research has established a strong link between attention and visual mental imagery, but it's remained uncertain whether attention networks influence individual differences in the vividness of visual mental imagery. In our study, we examined 140 participants, assessing the vividness of imagery using the Vividness of Visual Imagery Questionnaire in both eyes-open and eyes-closed conditions. We employed the Attention Network Test, coupled with EEG recording, to characterize three attention sub-networks: alerting, orienting, and executive control. To pinpoint the specific attentional networks associated with the vividness of visual mental imagery, we utilized latent profile analysis to categorize participants into distinct subgroups. Additionally, we constructed a regression mixture model to explore how attention networks predict different latent categories of visual imagery vividness. Our findings revealed that the efficiency of the alerting network, as indicated by the N1 component, demonstrated a positive correlation with the vividness of visual imagery. This electrophysiological evidence underscores the role of the alerting network in shaping individual differences in the vividness of visual mental imagery.

Where in the brain do internally generated and externally presented visual information interact?

Conscious experiences normally result from the flow of external input into our sensory systems. However, we can also create conscious percepts independently of sensory stimulation. These internally generated percepts are referred to as mental images, and they have many similarities with real visual percepts. Consequently, mental imagery is often referred to as "seeing in the mind's eye". While the neural basis of imagery has been widely studied, the interaction between internal and external sources of visual information has received little interest. Here we examined this question by using fMRI to record brain activity of healthy human volunteers while they were performing visual imagery that was distracted with a concurrent presentation of a visual stimulus. Multivariate pattern analysis (MVPA) was used to identify the brain basis of this interaction. Visual imagery was reflected in several brain areas in ventral temporal, lateral occipitotemporal, and posterior frontal cortices, with a left-hemisphere dominance. The key finding was that imagery content representations in the left lateral occipitotemporal cortex were disrupted when a visual distractor was presented during imagery. Our results thus demonstrate that the representations of internal and external visual information interact in brain areas associated with the encoding of visual objects and shapes.

Physiological and subjective arousal to prospective mental imagery: A mechanism for behavioral change?

Emotional prospective mental imagery, in which we simulate possible future events within our minds, have a pronounced impact on behavior. For example, repeated engagement in positive prospective imagery can lead to behavioral activation, while negative prospective imagery can lead to catastrophizing and avoidance. Physiological arousal boosts memory consolidation, creating emotional memories. Thus, if emotional prospective imagery produces an arousal response, the memory consolidation of these simulations of the future may be boosted, offering a possible underlying mechanism for the impact of emotional prospective imagery on behavior. In order to examine the feasibility of arousal as a possible mechanism behind the impact of emotional prospective imagery on behavior, sixty participants produced autobiographical prospective imagery of 30 scenes (10 positive, 10 neutral, and 10 negative), during which arousal responses (skin conductance) were measured, and ratings for subjective arousal, valence, and imagery vividness were collected. Moreover, because vividness of prospective imagery has been related to anxiety and depression, the study examined this relation also for event-related autobiographical prospective imagery. The results showed that emotional prospective imagery were associated with higher subjective arousal ratings as compared to neutral imagery. Physiological arousal responses showed a similar pattern, but further data is needed for a firm conclusion. Nevertheless, arousal-boosted consolidation remains a possible contributing mechanism for the impact of emotional prospective imagery on behavior. Moreover, results suggest both anxiety and depression may entail a reduced ability to invent prospective life situations. However, only anxiety was associated with less vivid imaginations, unless the imaginations were of negative content. Hence, anxious individuals may experience negative prospective imagery more vividly than imagery with neutral and positive content.

Sparse Bayesian Learning for End-to-End EEG Decoding

Decoding brain activity from non-invasive electroencephalography (EEG) is crucial for brain-computer interfaces (BCIs) and the study of brain disorders. Notably, end-to-end EEG decoding has gained widespread popularity in recent years owing to the remarkable advances in deep learning research. However, many EEG studies suffer from limited sample sizes, making it difficult for existing deep learning models to effectively generalize to highly noisy EEG data. To address this fundamental limitation, this paper proposes a novel end-to-end EEG decoding algorithm that utilizes a low-rank weight matrix to encode both spatio-temporal filters and the classifier, all optimized under a principled sparse Bayesian learning (SBL) framework. Importantly, this SBL framework also enables us to learn hyperparameters that optimally penalize the model in a Bayesian fashion. The proposed decoding algorithm is systematically benchmarked on five motor imagery BCI EEG datasets ( N=192) and an emotion recognition EEG dataset ( N=45), in comparison with several contemporary algorithms, including end-to-end deep-learning-based EEG decoding algorithms. The classification results demonstrate that our algorithm significantly outperforms the competing algorithms while yielding neurophysiologically meaningful spatio-temporal patterns. Our algorithm therefore advances the state-of-the-art by providing a novel EEG-tailored machine learning tool for decoding brain activity.

Temporo-Parietal cortex activation during motor imagery in older adults: A case study of Baduanjin

Age-associated cognitive and motor decline is related to central nervous system injury in older adults. Motor imagery training (MIT), as an emerging rehabilitative intervention, can activate neural basis similar to that in actual exercise, so as to promote motor function in older adults. The complex motor skills rely on the functional integration of the cerebral cortex. Understanding the neural mechanisms underlying motor imagery in older adults would support its application in motor rehabilitation and slowing cognitive decline. Based on this, the present study used functional near infrared spectroscopy (fNIRS) to record the changes in oxygen saturation in older adults (20 participants; mean age, 64.8 ± 4.5 years) during Baduanjin motor execution (ME) and motor imagery (MI). ME significantly activated the left postcentral gyrus, while the oxy-hemoglobin concentration in the right middle temporal gyrus increased significantly during motor imagery. These results indicate that advanced ME activates brain regions related to sensorimotor function, and MI increases the activation of the frontal-parietal cortex related to vision. In older adults, MI overactivated the temporo-parietal region associated with vision, and tend to be activated in the right brain.

No effects of different perturbations on the performance in a mental body-rotation task (MBRT) with egocentric perspective transformations and object-based transformations

The present study investigates participants' performance in two different mental body-rotation tasks (MBRTs) under conditions in which dynamic stability is challenged in two different balancing conditions: active balance control (Experiment 1), where participants actively maneuver, and re-active balance control (Experiment 2), where participants react to an external perturbation. The two MBRTs induced either an object-based spatial transformation (based on a same-different judgment) or an egocentric transformation (based on a left-right judgment). In Experiment 1, 48 participants were tested while standing on an even ground (low balancing requirements) or on a balance board (high balancing requirements). In Experiment 2, 32 participants performed while either standing still on a vibration plate or with the vibration plate moving in a low (20 Hz) or high (180 Hz) frequency. In both experiments, the results for response time and response error revealed effects of rotation angle and type of task. An effect of balancing condition was only observed for response error in Experiment 1. More precisely, response times and response errors increased for higher rotation angles. Also, performance was better for egocentric than for object-based spatial transformations. However, the different challenges to dynamic stability in Experiments 1 and 2 did not influence performance in the two MBRTs (except for response errors in Experiment 1) nor in a control condition (Experiment 1) without mental rotation.

Riemannian geometric and ensemble learning for decoding cross-session motor imagery electroencephalography signals

Objective.Brain-computer interfaces (BCIs) enable a direct communication pathway between the human brain and external devices, without relying on the traditional peripheral nervous and musculoskeletal systems. Motor imagery (MI)-based BCIs have attracted significant interest for their potential in motor rehabilitation. However, current algorithms fail to account for the cross-session variability of electroencephalography signals, limiting their practical application.Approach.We proposed a Riemannian geometry-based adaptive boosting and voting ensemble (RAVE) algorithm to address this issue. Our approach segmented the MI period into multiple sub-datasets using a sliding window approach and extracted features from each sub-dataset using Riemannian geometry. We then trained adaptive boosting (AdaBoost) ensemble learning classifiers for each sub-dataset, with the final BCI output determined by majority voting of all classifiers. We tested our proposed RAVE algorithm and eight other competing algorithms on four datasets (Pan2023, BNCI001-2014, BNCI001-2015, BNCI004-2015).Main results.Our results showed that, in the cross-session scenario, the RAVE algorithm outperformed the eight other competing algorithms significantly under different within-session training sample sizes. Compared to traditional algorithms that involved a large number of training samples, the RAVE algorithm achieved similar or even better classification performance on the datasets (Pan2023, BNCI001-2014, BNCI001-2015), even when it did not use or only used a small number of within-session training samples.Significance.These findings indicate that our cross-session decoding strategy could enable MI-BCI applications that require no or minimal training process.

Brief Visual Deprivation Effects on Brain Oscillations During Kinesthetic and Visual-motor Imagery

It is widely recognized that opening and closing the eyes can direct attention to external or internal stimuli processing. This has been confirmed by studies showing the effects of changes in visual stimulation changes on cerebral activity during different tasks, e.g., motor imagery and execution. However, an essential aspect of creating a mental representation of motion, such as imagery perspective, has not yet been investigated in the present context. Our study aimed to verify the effect of brief visual deprivation (under eyes open [EO] and eyes closed [EC] conditions) on brain wave oscillations and behavioral performance during kinesthetic imagery (KMI) and visual-motor imagery (VMI) tasks. We focused on the alpha and beta rhythms from visual- and motor-related EEG activity sources. Additionally, we used machine learning algorithms to establish whether the registered differences in brain oscillations might affect motor imagery brain-computer interface (MI-BCI) performance. The results showed that the occipital areas in the EC condition presented significantly stronger desynchronization during VMI tasks, which is typical for enhanced visual stimuli processing. Furthermore, the stronger desynchronization of alpha rhythms from motor areas in the EO, than EC condition confirmed previous effects obtained during real movements. It was also found that simulating movement under EC/EO conditions affected signal classification accuracy, which has practical implications for MI-BCI effectiveness. These findings suggest that shifting processing toward external or internal stimuli modulates brain rhythm oscillations associated with different perspectives on the mental representation of movement.

Volitional activation of remote place representations with a hippocampal brain-machine interface

The hippocampus is critical for recollecting and imagining experiences. This is believed to involve voluntarily drawing from hippocampal memory representations of people, events, and places, including maplike representations of familiar environments. However, whether representations in such "cognitive maps" can be volitionally accessed is unknown. We developed a brain-machine interface to test whether rats can do so by controlling their hippocampal activity in a flexible, goal-directed, and model-based manner. We found that rats can efficiently navigate or direct objects to arbitrary goal locations within a virtual reality arena solely by activating and sustaining appropriate hippocampal representations of remote places. This provides insight into the mechanisms underlying episodic memory recall, mental simulation and planning, and imagination and opens up possibilities for high-level neural prosthetics that use hippocampal representations.

Enhanced Motor Imagery Decoding by Calibration Model-Assisted With Tactile ERD

OBJECTIVE

In this study, we propose a tactile-assisted calibration method for a motor imagery (MI) based Brain-Computer Interface (BCI) system.

METHOD

In the proposed calibration, tactile stimulation was applied to the hand wrist to assist the subjects in the MI task, which is named SA-MI task. Then, classifier training in the SA-MI Calibration was performed using the SA-MI data, while the Conventional Calibration employed the MI data. After the classifiers were trained, the performance was evaluated on a common MI dataset.

RESULTS

Our study demonstrated that the SA-MI Calibration significantly improved the performance as compared with the Conventional Calibration, with a decoding accuracy of (78.3% vs. 71.3%). Moreover, the average calibration time could be reduced by 40%. This benefit of the SA-MI Calibration effect was further validated by an independent control group, which showed no improvement when tactile stimulation was not applied during the calibration phase. Further analysis showed that when compared with MI, greater motor-related cortical activation and higher R 2 value in the alpha-beta frequency band were induced in SA-MI.

CONCLUSION

Indeed, the SA-MI Calibration could significantly improve the performance and reduce the calibration time as compared with the Conventional Calibration.

SIGNIFICANCE

The proposed tactile stimulation-assisted MI Calibration method holds great potential for a faster and more accurate system setup at the beginning of BCI usage.

Motor Imagery and Pain Processing in Patients With Entrapment Neuropathies: A Cross-sectional Study

OBJECTIVES

(1) To assess the ability to generate both kinesthetic and visual motor imagery in participants with carpal tunnel syndrome (CTS), compared with asymptomatic participants. (2) To assess the influence of psychophysiological and functional variables in the motor imagery process.

METHODS

Twenty patients with unilateral CTS and 18 pain-free individuals were recruited. An observational case-control study with a nonprobability sample was conducted to assess visual and kinesthetic movement imagery ability and psychophysiological variables in patients with CTS compared with asymptomatic participants in a control group. The trial was conducted in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology statement.

RESULTS

CTS patients have more difficulties in generating visual motor images compared with asymptomatic individuals ( t =-2.099; P <0.05; d=0.70). They need more time to complete the mental tasks (visual t =-2.424; P <0.05 and kinesthetic t =-2.200; P <0.05). A negative correlation was found between the ability to imagine and functional deficits ( r =-0.569; P =0.021) for the kinesthetic subscale and temporal summation ( r =-0.515; P <0.5). A positive correlation was found between pain pressure threshold homolateral (homolateral) and time to generate the visual mental images ( r =0.537; P <0.05).

DISCUSSION

CTS patients have greater difficulty generating motor images than asymptomatic individuals. Patients also spend more time during mental tasks. CTS patients present a relationship between temporal summation and the capacity to generate kinesthetic images. In addition, the CST patients presented a correlation between chronometry mental tasking and mechanical hyperalgesia.

Effects of visual-electrotactile stimulation feedback on brain functional connectivity during motor imagery practice

The use of neurofeedback is an important aspect of effective motor rehabilitation as it offers real-time sensory information to promote neuroplasticity. However, there is still limited knowledge about how the brain's functional networks reorganize in response to such feedback. To address this gap, this study investigates the reorganization of the brain network during motor imagery tasks when subject to visual stimulation or visual-electrotactile stimulation feedback. This study can provide healthcare professionals with a deeper understanding of the changes in the brain network and help develop successful treatment approaches for brain-computer interface-based motor rehabilitation applications. We examine individual edges, nodes, and the entire network, and use the minimum spanning tree algorithm to construct a brain network representation using a functional connectivity matrix. Furthermore, graph analysis is used to detect significant features in the brain network that might arise in response to the feedback. Additionally, we investigate the power distribution of brain activation patterns using power spectral analysis and evaluate the motor imagery performance based on the classification accuracy. The results showed that the visual and visual-electrotactile stimulation feedback induced subject-specific changes in brain activation patterns and network reorganization in the [Formula: see text] band. Thus, the visual-electrotactile stimulation feedback significantly improved the integration of information flow between brain regions associated with motor-related commands and higher-level cognitive functions, while reducing cognitive workload in the sensory areas of the brain and promoting positive emotions. Despite these promising results, neither neurofeedback modality resulted in a significant improvement in classification accuracy, compared with the absence of feedback. These findings indicate that multimodal neurofeedback can modulate imagery-mediated rehabilitation by enhancing motor-cognitive communication and reducing cognitive effort. In future interventions, incorporating this technique to ease cognitive demands for participants could be crucial for maintaining their motivation to engage in rehabilitation.

Imagine to automatize: automatization of stimulus-response coupling after action imagery practice in implicit sequence learning

Action imagery practice (AIP) describes the repetitive imagination of an action to improve subsequent action execution. Because AIP and action execution practice (AEP) draw on partly similar motor mechanisms, it was assumed that AIP may lead to motor automatization, which is observable in a reduction of dual-task costs after AEP. To investigate automatization in AIP, we compared dual-task and single-task performance in practice and random sequences in pretests and posttests. All participants practiced serial reactions to visual stimuli in ten single-task practice sessions. An AIP group imagined the reactions. An AEP group and a control practice group executed the reactions. Practice followed a sequential sequence in AIP and AEP but was random in control practice. In dual-task test conditions, tones were counted that appeared in addition to the visual stimuli. RTs decreased from pretest to posttest in both practice and random sequences in all groups indicating general sequence-unspecific learning. Further, RTs decreased to a greater extent in the practice sequence than in the random sequence after AIP and AEP, indicating sequence-specific learning. Dual-task costs-the difference between RTs after tone and no tone events-were reduced independent from the performed sequence in all groups indicating sequence-unspecific automatization. It is concluded that the stimulus-response coupling can be automatized by both, AEP and AIP.

Neural retrieval processes occur more rapidly for visual mental images that were previously encoded with high-vividness

Visual mental imagery refers to our ability to experience visual images in the absence of sensory stimulation. Studies have shown that visual mental imagery can improve episodic memory. However, we have limited understanding of the neural mechanisms underlying this improvement. Using electroencephalography, we examined the neural processes associated with the retrieval of previously generated visual mental images, focusing on how the vividness at generation can modulate retrieval processes. Participants viewed word stimuli referring to common objects, forming a visual mental image of each word and rating the vividness of the mental image. This was followed by a surprise old/new recognition task. We compared retrieval performance for items rated as high- versus low-vividness at encoding. High-vividness items were retrieved with faster reaction times and higher confidence ratings in the memory judgment. While controlling for confidence, neural measures indicated that high-vividness items produced an earlier decrease in alpha-band activity at retrieval compared with low-vividness items, suggesting an earlier memory reinstatement. Even when low-vividness items were remembered with high confidence, they were not retrieved as quickly as high-vividness items. These results indicate that when highly vivid mental images are encoded, the speed of their retrieval occurs more rapidly, relative to low-vivid items.

Classification Algorithm for Electroencephalogram-based Motor Imagery Using Hybrid Neural Network with Spatio-temporal Convolution and Multi-head Attention Mechanism

Motor imagery (MI) is a brain-computer interface (BCI) technique in which specific brain regions are activated when people imagine their limbs (or muscles) moving, even without actual movement. The technology converts electroencephalogram (EEG) signals generated by the brain into computer-readable commands by measuring neural activity. Classification of motor imagery is one of the tasks in BCI. Researchers have done a lot of work on motor imagery classification, and the existing literature has relatively mature decoding methods for two-class motor tasks. However, as the categories of EEG-based motor imagery tasks increase, further exploration is needed for decoding research on four-class motor imagery tasks. In this study, we designed a hybrid neural network that combines spatiotemporal convolution and attention mechanisms. Specifically, the data is first processed by spatiotemporal convolution to extract features and then processed by a Multi-branch Convolution block. Finally, the processed data is input into the encoder layer of the Transformer for a self-attention calculation to obtain the classification results. Our approach was tested on the well-known MI datasets BCI Competition IV 2a and 2b, and the results show that the 2a dataset has a global average classification accuracy of 83.3% and a kappa value of 0.78. Experimental results show that the proposed method outperforms most of the existing methods.

Probing the unimaginable: The impact of aphantasia on distinct domains of visual mental imagery and visual perception

Different individuals experience varying degrees of vividness in their visual mental images. The distribution of these variations across different imagery domains, such as object shape, color, written words, faces, and spatial relationships, remains unknown. To address this issue, we conducted a study with 117 healthy participants who reported different levels of imagery vividness. Of these participants, 44 reported experiencing absent or nearly absent visual imagery, a condition known as "aphantasia". These individuals were compared to those with typical (N = 42) or unusually vivid (N = 31) imagery ability. We used an online version of the French-language Battérie Imagination-Perception (eBIP), which consists of tasks tapping each of the above-mentioned domains, both in visual imagery and in visual perception. We recorded the accuracy and response times (RTs) of participants' responses. Aphantasic participants reached similar levels of accuracy on all tasks compared to the other groups (Bayesian repeated measures ANOVA, BF = .02). However, their RTs were slower in both imagery and perceptual tasks (BF = 266), and they had lower confidence in their responses on perceptual tasks (BF = 7.78e5). A Bayesian regression analysis revealed that there was an inverse correlation between subjective vividness and RTs for the entire participant group: higher levels of vividness were associated with faster RTs. The pattern was similar in all the explored domains. The findings suggest that individuals with congenital aphantasia experience a slowing in processing visual information in both imagery and perception, but the precision of their processing remains unaffected. The observed performance pattern lends support to the hypotheses that congenital aphantasia is primarily a deficit of phenomenal consciousness, or that it employs alternative strategies other than visualization to access preserved visual information.

Vividness of imagery and affective response to episodic memories and episodic future thoughts: a systematic review and meta-analysis

Recalling personal past events and imagining personal future events are closely linked, yet also show differences. It has been claimed that episodic future thinking produces stronger intensity of in-the-moment affect than does recalling episodic memories [Schubert, T., Eloo, R., Scharfen, J., & Morina, N. (2020). How imagining personal future scenarios influences affect: Systematic review and meta-analysis. Clinical Psychology Review, 75, 101811. https://doi.org/10.1016/j.cpr.2019.101811]. In contrast, the literature indicates that memories are experienced more vividly than are episodic future thoughts, a quality that would be expected to produce a stronger rather than a weaker affective response. In this systematic review and meta-analysis, we examined (a) the intensity of affect, (b) the vividness and (c) the valence of emotion experienced in response to remembering personal past events compared to imagining personal future events. Sixteen studies with a combined sample of 1735 met criteria for inclusion. Remembered past events were experienced more vividly than imagined future events but there was no difference between the two types of representations on emotional intensity. Imagined future events were associated with more positive emotion than memories. Future research could examine factors responsible for the equivalent strength of emotional response in memories and future-thinking despite their differences in vividness.

Distinct roles of the orbitofrontal cortex, ventral striatum, and dopamine neurons in counterfactual thinking of decision outcomes

Individuals often assess past decisions by comparing what was gained with what would have been gained had they acted differently. Thoughts of past alternatives that counter what actually happened are called "counterfactuals." Recent theories emphasize the role of the prefrontal cortex in processing counterfactual outcomes in decision-making, although how subcortical regions contribute to this process remains to be elucidated. Here we report a clear distinction among the roles of the orbitofrontal cortex, ventral striatum and midbrain dopamine neurons in processing counterfactual outcomes in monkeys. Our findings suggest that actually gained and counterfactual outcome signals are both processed in the cortico-subcortical network constituted by these regions but in distinct manners and integrated only in the orbitofrontal cortex in a way to compare these outcomes. This study extends the prefrontal theory of counterfactual thinking and provides key insights regarding how the prefrontal cortex cooperates with subcortical regions to make decisions using counterfactual information.

LMDA-Net:A lightweight multi-dimensional attention network for general EEG-based brain-computer interfaces and interpretability

Electroencephalography (EEG)-based brain-computer interfaces (BCIs) pose a challenge for decoding due to their low spatial resolution and signal-to-noise ratio. Typically, EEG-based recognition of activities and states involves the use of prior neuroscience knowledge to generate quantitative EEG features, which may limit BCI performance. Although neural network-based methods can effectively extract features, they often encounter issues such as poor generalization across datasets, high predicting volatility, and low model interpretability. To address these limitations, we propose a novel lightweight multi-dimensional attention network, called LMDA-Net. By incorporating two novel attention modules designed specifically for EEG signals, the channel attention module and the depth attention module, LMDA-Net is able to effectively integrate features from multiple dimensions, resulting in improved classification performance across various BCI tasks. LMDA-Net was evaluated on four high-impact public datasets, including motor imagery (MI) and P300-Speller, and was compared with other representative models. The experimental results demonstrate that LMDA-Net outperforms other representative methods in terms of classification accuracy and predicting volatility, achieving the highest accuracy in all datasets within 300 training epochs. Ablation experiments further confirm the effectiveness of the channel attention module and the depth attention module. To facilitate an in-depth understanding of the features extracted by LMDA-Net, we propose class-specific neural network feature interpretability algorithms that are suitable for evoked responses and endogenous activities. By mapping the output of the specific layer of LMDA-Net to the time or spatial domain through class activation maps, the resulting feature visualizations can provide interpretable analysis and establish connections with EEG time-spatial analysis in neuroscience. In summary, LMDA-Net shows great potential as a general decoding model for various EEG tasks.

Assessing aphantasia prevalence and the relation of self-reported imagery abilities and memory task performance

Aphantasia is the experience of having little to no visual imagery. We assessed the prevalence rate of aphantasia in 5,010 people from the general population of adults in the United States through self-report and responses to two visual imagery scales. The self-reported prevalence rate of aphantasia was 8.9% in this sample. However, not all participants who reported themselves as aphantasic showed low-imagery profiles on the questionnaire scales, and scale prevalence was much lower (1.5%). Self-reported aphantasic individuals reported lower dream frequencies and self-talk and showed poorer memory performance compared to individuals who reported average and high mental imagery. Self-reported aphantasic individuals showed a greater preference for written instruction compared to video instruction for learning a hypothetical new task although there were differences for men and women in this regard. Categorizing aphantasia using a scale measure and relying on self-identification may provide a more consistent picture of who lacks visual imagery.

Does ASMR propensity reflect a mentally flexible mindset? Exploring the relationship between ASMR propensity, transliminality, emotional contagion, schizotypal traits, roleplaying ability, and creativity

Autonomous Sensory Meridian Response (ASMR) is an alternative state of consciousness characterised by changes in affect, feelings of relaxation, and tingling sensations on the body. Online videos designed to stimulate ASMR in viewers have become increasingly popular. Although there is evidence that ASMR may improve sleep, emotion regulation, and relaxation, the current understanding of ASMR propensity remains limited. This study examined whether a mentally flexible cognitive style may underlie the ability to experience ASMR. Undergraduate students (N = 376) completed an online survey involving a series of self-report questionnaires and two performance-based creative ability tasks. Findings did not provide support for an overall mentally flexible mindset, however, transliminality, emotional contagion susceptibility, positive schizotypal traits, and roleplaying ability all significantly positively predicted ASMR propensity. These findings suggest that ASMR propensity represents several possible underlying cognitive styles relating to enhanced imagination and perceptual ability, and cannot be simply characterised by mental flexibility.

Emotion in the mind's eye: Imagination for adaptive cognition

In a complex world, we are constantly faced with environmental stimuli that shape our moment-to-moment experiences. But just as rich and complex as the external world is the internal milieu-our imagination. Imagination offers a powerful vehicle for playing out hypothetical experiences in the mind's eye. It allows us to mentally time travel to behold what the future might bring, including our greatest desires or fears. Indeed, imagined experiences tend to be emotion-laden. How and why are humans capable of this remarkable feat? Based on psychological findings, we highlight the importance of imagination for emotional aspects of cognition and behavior, namely in the generation and regulation of emotions. Based on recent cognitive neuroscience work, we identify putative neural networks that are most critical for emotional imagination, with a major focus on the default mode network. Finally, we briefly highlight the possible functional implications of individual differences in imagination. Overall, we hope to address why humans have the capacity to simulate hypothetical emotional experiences and how this ability can be harnessed in adaptive (and sometimes maladaptive) ways. We end by discussing open questions.

Virtual reality and motor imagery for early post-stroke rehabilitation

BACKGROUND

Motor impairment is a common consequence of stroke causing difficulty in independent movement. The first month of post-stroke rehabilitation is the most effective period for recovery. Movement imagination, known as motor imagery, in combination with virtual reality may provide a way for stroke patients with severe motor disabilities to begin rehabilitation.

METHODS

The aim of this study is to verify whether motor imagery and virtual reality help to activate stroke patients' motor cortex. 16 acute/subacute (< 6 months) stroke patients participated in this study. All participants performed motor imagery of basketball shooting which involved the following tasks: listening to audio instruction only, watching a basketball shooting animation in 3D with audio, and also performing motor imagery afterwards. Electroencephalogram (EEG) was recorded for analysis of motor-related features of the brain such as power spectral analysis in the [Formula: see text] and [Formula: see text] frequency bands and spectral entropy. 18 EEG channels over the motor cortex were used for all stroke patients.

RESULTS

All results are normalised relative to all tasks for each participant. The power spectral densities peak near the [Formula: see text] band for all participants and also the [Formula: see text] band for some participants. Tasks with instructions during motor imagery generally show greater power spectral peaks. The p-values of the Wilcoxon signed-rank test for band power comparison from the 18 EEG channels between different pairs of tasks show a 0.01 significance of rejecting the band powers being the same for most tasks done by stroke subjects. The motor cortex of most stroke patients is more active when virtual reality is involved during motor imagery as indicated by their respective scalp maps of band power and spectral entropy.

CONCLUSION

The resulting activation of stroke patient's motor cortices in this study reveals evidence that it is induced by imagination of movement and virtual reality supports motor imagery. The framework of the current study also provides an efficient way to investigate motor imagery and virtual reality during post-stroke rehabilitation.

Vividness as the similarity between generated imagery and an internal model

Vividness in visual mental imagery has been relatively under-explored compared to imagery's representational format and neural mechanisms. In this paper, we take a deeper look at vividness and suggest that in re-framing it, we can potentially reconcile disparate findings regarding visual cortex activation during imagery. Unlike traditional views of vividness that define the concept in terms of perception, we frame vividness in terms of imagery's relation to an internal model; the closer the generated imagery is to this model, the more vivid it is. This view is considered alongside existing neuroscientific, psychological, and philosophical research, as well as directions for future research.

Improving Generalization of CNN-based Motor-Imagery EEG Decoders via Dynamic Convolutions

Deep Convolutional Neural Networks (CNNs) have recently demonstrated impressive results in electroencephalogram (EEG) decoding for several Brain-Computer Interface (BCI) paradigms, including Motor-Imagery (MI). However, neurophysiological processes underpinning EEG signals vary across subjects causing covariate shifts in data distributions and hence hindering the generalization of deep models across subjects. In this paper, we aim to address the challenge of inter-subject variability in MI. To this end, we employ causal reasoning to characterize all possible distribution shifts in the MI task and propose a dynamic convolution framework to account for shifts caused by the inter-subject variability. Using publicly available MI datasets, we demonstrate improved generalization performance (up to 5%) across subjects in various MI tasks for four well-established deep architectures.

Early sleep after action observation and motor imagery training boosts improvements in manual dexterity

The systematic observation and imagination of actions promotes acquisition of motor skills. Furthermore, studies demonstrated that early sleep after practice enhances motor learning through an offline stabilization process. Here, we investigated behavioral effects and neurodynamical correlates of early sleep after action observation and motor imagery training (AO + MI-training) on motor learning in terms of manual dexterity. Forty-five healthy participants were randomized into three groups receiving a 3 week intervention consisting of AO + MI-training immediately before sleeping or AO + MI-training at least 12 h before sleeping or a control stimulation. AO + MI-training implied the observation and motor imagery of transitive manual dexterity tasks, whereas the control stimulation consisted of landscape video-clips observation. Manual dexterity was assessed using functional tests, kinematic and neurophysiological outcomes before and after the training and at 1-month follow-up. AO + MI-training improved manual dexterity, but subjects performing AO + MI-training followed by early sleep had significantly larger improvements than those undergoing the same training at least 12 h before sleeping. Behavioral findings were supported by neurodynamical correlates during motor performance and additional sleep-dependent benefits were also detected at 1 month follow-up. These findings introduce a new approach to enhance the acquisition of new motor skills or facilitate recovery in patients with motor impairments.

Imagination as a fundamental function of the hippocampus

Imagination is a biological function that is vital to human experience and advanced cognition. Despite this importance, it remains unknown how imagination is realized in the brain. Substantial research focusing on the hippocampus, a brain structure traditionally linked to memory, indicates that firing patterns in spatially tuned neurons can represent previous and upcoming paths in space. This work has generally been interpreted under standard views that the hippocampus implements cognitive abilities primarily related to actual experience, whether in the past (e.g. recollection, consolidation), present (e.g. spatial mapping) or future (e.g. planning). However, relatively recent findings in rodents identify robust patterns of hippocampal firing corresponding to a variety of alternatives to actual experience, in many cases without overt reference to the past, present or future. Given these findings, and others on hippocampal contributions to human imagination, we suggest that a fundamental function of the hippocampus is to generate a wealth of hypothetical experiences and thoughts. Under this view, traditional accounts of hippocampal function in episodic memory and spatial navigation can be understood as particular applications of a more general system for imagination. This view also suggests that the hippocampus contributes to a wider range of cognitive abilities than previously thought. This article is part of the theme issue 'Thinking about possibilities: mechanisms, ontogeny, functions and phylogeny'.

A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI

Decoding neural responses from multimodal information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance hybrid brain-computer interfaces (hBCIs). However, existing modest performance improvement of hBCIs might be attributed to the lack of computational frameworks that exploit complementary synergistic properties in multimodal features. This study proposes a multimodal data fusion framework to represent and decode synergistic multimodal motor imagery (MI) neural responses. We hypothesize that exploiting EEG nonlinear dynamics adds a new informative dimension to the commonly combined EEG-fNIRS features and will ultimately increase the synergy between EEG and fNIRS features toward an enhanced hBCI. The EEG nonlinear dynamics were quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the commonly used spectral features for an enhanced multimodal configuration when combined with fNIRS. The high-dimensional multimodal features were further given to a feature selection algorithm relying on the least absolute shrinkage and selection operator (LASSO) for fused feature selection. Linear support vector machine (SVM) was then used to evaluate the framework. The mean hybrid classification performance improved by up to 15% and 4% compared to the unimodal EEG and fNIRS, respectively. The proposed graph-based framework substantially increased the contribution of EEG features for hBCI classification from 28.16% up to 52.9% when introduced the nonlinear dynamics and improved the performance by approximately 2%. These findings suggest that graph-based nonlinear dynamics can increase the synergy between EEG and fNIRS features for an enhanced MI response representation that is not dominated by a single modality.

The Effects of Bilateral Phase-Dependent Closed-Loop Vibration Stimulation With Motor Imagery Paradigm

Vibration stimulation has been shown to have the potential to improve the activation pattern of unilateral motor imagery (MI) and to promote motor recovery. However, in the widely used left and right hand MI brain-computer interface (BCI) paradigm, the vibration stimuli cannot be directly applied to the imaginary side due to the spontaneity of imagery. In this study, we proposed a method of phase-dependent closed-loop vibration stimulation to be applied on both hands, and explored the effects of different vibration stimuli on the left and right hand MI-BCI. Eighteen healthy subjects were recruited and asked to perform, in sequence, MI tasks under three different conditions of vibratory feedback, which were no vibration stimulus (MI), phase-dependent closed-loop vibration stimulus (PDS), and continuous vibration stimulus (CS). Then the performance of the left and right hand MI-BCI and the patterns of brain oscillation were compared and analyzed under these different stimulation conditions. The results showed that vibration stimulation effectively boosted the activation of the sensorimotor cortex and enhanced the functional connectivity among sensorimotor-related brain regions during MI. The closed-loop stimulation evoked stronger event-related desynchronization patterns on the contralateral side of the imagined hand compared to continuous stimulation. There was a more obvious distinction between left hand task and right hand task. In addition, phase-dependent closed-loop vibration stimulation increased classification accuracy by approximately 7% (paired t-test, p=0.004, n=18) compared to MI alone, while continuous vibration stimulation only increased it by 4% (paired t-test, p=0.067, n=18). This result further demonstrated the effectiveness of the phase-dependent closed-loop vibration stimulation method in improving the overall performance of the MI paradigm and is expected to be further applied in areas such as stroke rehabilitation in the future.

Improving classification and reconstruction of imagined images from EEG signals

Decoding brain activity related to specific tasks, such as imagining something, is important for brain computer interface (BCI) control. While decoding of brain signals, such as functional magnetic resonance imaging (fMRI) signals and electroencephalography (EEG) signals, during observing visual images and while imagining images has been previously reported, further development of methods for improving training, performance, and interpretation of brain data was the goal of this study. We applied a Sinc-EEGNet to decode brain activity during perception and imagination of visual stimuli, and added an attention module to extract the importance of each electrode or frequency band. We also reconstructed images from brain activity by using a generative adversarial network (GAN). By combining the EEG recorded during a visual task (perception) and an imagination task, we have successfully boosted the accuracy of classifying EEG data in the imagination task and improved the quality of reconstruction by GAN. Our result indicates that the brain activity evoked during the visual task is present in the imagination task and can be used for better classification of the imagined image. By using the attention module, we can derive the spatial weights in each frequency band and contrast spatial or frequency importance between tasks from our model. Imagination tasks are classified by low frequency EEG signals over temporal cortex, while perception tasks are classified by high frequency EEG signals over occipital and frontal cortex. Combining data sets in training results in a balanced model improving classification of the imagination task without significantly changing performance in the visual task. Our approach not only improves performance and interpretability but also potentially reduces the burden on training since we can improve the accuracy of classifying a relatively hard task with high variability (imagination) by combining with the data of the relatively easy task, observing visual images.