Learning to Associate Orientation with Color in Early Visual Areas by Associative Decoded fMRI Neurofeedback

Training in Use of Brain-Machine Interface-Controlled Robotic Hand Improves Accuracy Decoding Two Types of Hand Movements

Objective: Brain-machine interfaces (BMIs) are useful for inducing plastic changes in cortical representation. A BMI first decodes hand movements using cortical signals and then converts the decoded information into movements of a robotic hand. By using the BMI robotic hand, the cortical representation decoded by the BMI is modulated to improve decoding accuracy. We developed a BMI based on real-time magnetoencephalography (MEG) signals to control a robotic hand using decoded hand movements. Subjects were trained to use the BMI robotic hand freely for 10 min to evaluate plastic changes in the cortical representation due to the training.

Method: We trained nine young healthy subjects with normal motor function. In open-loop conditions, they were instructed to grasp or open their right hands during MEG recording. Time-averaged MEG signals were then used to train a real decoder to control the robotic arm in real time. Then, subjects were instructed to control the BMI-controlled robotic hand by moving their right hands for 10 min while watching the robot's movement. During this closed-loop session, subjects tried to improve their ability to control the robot. Finally, subjects performed the same offline task to compare cortical activities related to the hand movements. As a control, we used a random decoder trained by the MEG signals with shuffled movement labels. We performed the same experiments with the random decoder as a crossover trial. To evaluate the cortical representation, cortical currents were estimated using a source localization technique. Hand movements were also decoded by a support vector machine using the MEG signals during the offline task. The classification accuracy of the movements was compared among offline tasks.

Results: During the BMI training with the real decoder, the subjects succeeded in improving their accuracy in controlling the BMI robotic hand with correct rates of 0.28 ± 0.13 to 0.50 ± 0.11 (p = 0.017, n = 8, paired Student's t-test). Moreover, the classification accuracy of hand movements during the offline task was significantly increased after BMI training with the real decoder from 62.7 ± 6.5 to 70.0 ± 11.1% (p = 0.022, n = 8, t₍₇₎ = 2.93, paired Student's t-test), whereas accuracy did not significantly change after BMI training with the random decoder from 63.0 ± 8.8 to 66.4 ± 9.0% (p = 0.225, n = 8, t₍₇₎ = 1.33).

Conclusion: BMI training is a useful tool to train the cortical activity necessary for BMI control and to induce some plastic changes in the activity.

Time course of clinical change following neurofeedback

Neurofeedback - learning to modulate brain function through real-time monitoring of current brain state - is both a powerful method to perturb and probe brain function and an exciting potential clinical tool. For neurofeedback effects to be useful clinically, they must persist. Here we examine the time course of symptom change following neurofeedback in two clinical populations, combining data from two ongoing neurofeedback studies. This analysis reveals a shared pattern of symptom change, in which symptoms continue to improve for weeks after neurofeedback. This time course has several implications for future neurofeedback studies. Most neurofeedback studies are not designed to test an intervention with this temporal pattern of response. We recommend that new studies incorporate regular follow-up of subjects for weeks or months after the intervention to ensure that the time point of greatest effect is sampled. Furthermore, this time course of continuing clinical change has implications for crossover designs, which may attribute long-term, ongoing effects of real neurofeedback to the control intervention that follows. Finally, interleaving neurofeedback sessions with assessments and examining when clinical improvement peaks may not be an appropriate approach to determine the optimal number of sessions for an application.

Human V4 Activity Patterns Predict Behavioral Performance in Imagery of Object Color

Color is special among basic visual features in that it can form a defining part of objects that are engrained in our memory. Whereas most neuroimaging research on human color vision has focused on responses related to external stimulation, the present study investigated how sensory-driven color vision is linked to subjective color perception induced by object imagery. We recorded fMRI activity in male and female volunteers during viewing of abstract color stimuli that were red, green, or yellow in half of the runs. In the other half we asked them to produce mental images of colored, meaningful objects (such as tomato, grapes, banana) corresponding to the same three color categories. Although physically presented color could be decoded from all retinotopically mapped visual areas, only hV4 allowed predicting colors of imagined objects when classifiers were trained on responses to physical colors. Importantly, only neural signal in hV4 was predictive of behavioral performance in the color judgment task on a trial-by-trial basis. The commonality between neural representations of sensory-driven and imagined object color and the behavioral link to neural representations in hV4 identifies area hV4 as a perceptual hub linking externally triggered color vision with color in self-generated object imagery.SIGNIFICANCE STATEMENT Humans experience color not only when visually exploring the outside world, but also in the absence of visual input, for example when remembering, dreaming, and during imagery. It is not known where neural codes for sensory-driven and internally generated hue converge. In the current study we evoked matching subjective color percepts, one driven by physically presented color stimuli, the other by internally generated color imagery. This allowed us to identify area hV4 as the only site where neural codes of corresponding subjective color perception converged regardless of its origin. Color codes in hV4 also predicted behavioral performance in an imagery task, suggesting it forms a perceptual hub for color perception.

Towards an unconscious neural reinforcement intervention for common fears

Conventional therapies for the treatment of anxiety disorders are aversive, and as a result, many patients terminate treatment prematurely. We have developed an unconscious method to bypass the unpleasantness in conscious exposure using functional magnetic resonance imaging neural reinforcement. Using this method, participants learn to generate brain patterns similar to the multivariate brain pattern of a feared animal. We demonstrate in a double-blind placebo-controlled experiment that neural reinforcement can lead to reliable reductions in physiological fear responses. Crucially, this intervention can be achieved completely unconsciously and without any aversive reaction. Extending our approach to other forms of psychopathologies, such as posttraumatic stress disorders, might eventually provide another means of intervention for patients currently receiving insufficient exposure treatments.

Can “hardwired” physiological fear responses (e.g., for spiders and snakes) be reprogramed unconsciously in the human brain? Currently, exposure therapy is among the most effective treatments for anxiety disorders, but this intervention is subjectively aversive to patients, causing many to drop out of treatment prematurely. Here we introduce a method to bypass the subjective unpleasantness in conscious exposure, by directly pairing monetary reward with unconscious occurrences of decoded representations of naturally feared animals in the brain. To decode physiological fear representations without triggering excessively aversive reactions, we capitalize on recent advancements in functional magnetic resonance imaging decoding techniques, and use a method called hyperalignment to infer the relevant representations of feared animals for a designated participant based on data from other “surrogate” participants. In this way, the procedure completely bypasses the need for a conscious encounter with feared animals. We demonstrate that our method can lead to reliable reductions in physiological fear responses, as measured by skin conductance as well as amygdala hemodynamic activity. Not only do these results raise the intriguing possibility that naturally occurring fear responses can be “reprogrammed” outside of conscious awareness, importantly, they also create the rare opportunity to rigorously test a psychological intervention of this nature in a double-blind, placebo-controlled fashion. This may pave the way for a new approach combining the appealing rationale and proven efficacy of conventional psychotherapy with the rigor and leverage of clinical neuroscience.

Advances in fMRI Real-Time Neurofeedback

Functional magnetic resonance imaging (fMRI) neurofeedback is a type of biofeedback in which real-time online fMRI signals are used to self-regulate brain function. Since its advent in 2003 significant progress has been made in fMRI neurofeedback techniques. Specifically, the use of implicit protocols, external rewards, multivariate analysis, and connectivity analysis has allowed neuroscientists to explore a possible causal involvement of modified brain activity in modified behavior. These techniques have also been integrated into groundbreaking new neurofeedback technologies, specifically decoded neurofeedback (DecNef) and functional connectivity-based neurofeedback (FCNef). By modulating neural activity and behavior, DecNef and FCNef have substantially advanced both basic and clinical research.

The potential of real-time fMRI neurofeedback for stroke rehabilitation: A systematic review

Real-time functional magnetic resonance imaging (rt-fMRI) neurofeedback aids the modulation of neural functions by training self-regulation of brain activity through operant conditioning. This technique has been applied to treat several neurodevelopmental and neuropsychiatric disorders, but its effectiveness for stroke rehabilitation has not been examined yet. Here, we systematically review the effectiveness of rt-fMRI neurofeedback training in modulating motor and cognitive processes that are often impaired after stroke. Based on predefined search criteria, we selected and examined 33 rt-fMRI neurofeedback studies, including 651 healthy individuals and 15 stroke patients in total. The results of our systematic review suggest that rt-fMRI neurofeedback training can lead to a learned modulation of brain signals, with associated changes at both the neural and the behavioural level. However, more research is needed to establish how its use can be optimized in the context of stroke rehabilitation.

Direct modulation of aberrant brain network connectivity through real-time NeuroFeedback

The existence of abnormal connectivity patterns between resting state networks in neuropsychiatric disorders, including Autism Spectrum Disorder (ASD), has been well established. Traditional treatment methods in ASD are limited, and do not address the aberrant network structure. Using real-time fMRI neurofeedback, we directly trained three brain nodes in participants with ASD, in which the aberrant connectivity has been shown to correlate with symptom severity. Desired network connectivity patterns were reinforced in real-time, without participants’ awareness of the training taking place. This training regimen produced large, significant long-term changes in correlations at the network level, and whole brain analysis revealed that the greatest changes were focused on the areas being trained. These changes were not found in the control group. Moreover, changes in ASD resting state connectivity following the training were correlated to changes in behavior, suggesting that neurofeedback can be used to directly alter complex, clinically relevant network connectivity patterns.

Even when we are at rest, our brains are always active. For example, areas of the brain involved in vision remain active in complete darkness. Different brain regions that connect together to perform a given task often show coordinated activity at rest. Past studies have shown that these resting connections are different in people with conditions such as autism. Some brain regions are more weakly connected while others are more strongly connected in people with autism spectrum disorder compared to those without. Furthermore, people with more severe symptoms seem to have more abnormal connections.

“Neurofeedback training” is a method of changing the resting connections between different brain regions. Scientists measure a brain signal – the connection between different brain regions – from a person in real time. They then provide positive feedback to the person if this signal improves. For example, if a connection that is too weak becomes stronger, the scientists might reinforce this by providing feedback on the success. Previous work has shown that neurofeedback training may even change people’s behaviour. However, it has not yet been explored as a method of treating the abnormal connections seen in people with autism when their brains are at rest.

To address this, Ramot et al. used a technique known as “functional magnetic resonance imaging” (or fMRI for short) to measure brain activity in young men with autism. First, certain brain regions were identified as having abnormal resting connections with each other. The participants were then asked to look at a blank screen and to try to reveal a picture hidden underneath. Whenever the connections between the chosen brain regions improved, part of the picture was revealed on the screen, accompanied by an upbeat sound. The participants were unaware that it was their brain signals causing this positive feedback.

This form of neurofeedback training successfully changed the abnormal brain connections in most of the participants with autism, making their connections more similar to those seen in the wider population. These effects lasted up to a year after training. Early results also suggest that these changes were related to improvements in symptoms, although further work is needed to see if doctors could reliably use this method as a therapy. These findings show that neurofeedback training could potentially help treat not only autism spectrum disorder, but a range of other disorders that involve abnormal brain connections, including depression and schizophrenia.

Self-regulation strategy, feedback timing and hemodynamic properties modulate learning in a simulated fMRI neurofeedback environment

Direct manipulation of brain activity can be used to investigate causal brain-behavior relationships. Current noninvasive neural stimulation techniques are too coarse to manipulate behaviors that correlate with fine-grained spatial patterns recorded by fMRI. However, these activity patterns can be manipulated by having people learn to self-regulate their own recorded neural activity. This technique, known as fMRI neurofeedback, faces challenges as many participants are unable to self-regulate. The causes of this non-responder effect are not well understood due to the cost and complexity of such investigation in the MRI scanner. Here, we investigated the temporal dynamics of the hemodynamic response measured by fMRI as a potential cause of the non-responder effect. Learning to self-regulate the hemodynamic response involves a difficult temporal credit-assignment problem because this signal is both delayed and blurred over time. Two factors critical to this problem are the prescribed self-regulation strategy (cognitive or automatic) and feedback timing (continuous or intermittent). Here, we sought to evaluate how these factors interact with the temporal dynamics of fMRI without using the MRI scanner. We first examined the role of cognitive strategies by having participants learn to regulate a simulated neurofeedback signal using a unidimensional strategy: pressing one of two buttons to rotate a visual grating that stimulates a model of visual cortex. Under these conditions, continuous feedback led to faster regulation compared to intermittent feedback. Yet, since many neurofeedback studies prescribe implicit self-regulation strategies, we created a computational model of automatic reward-based learning to examine whether this result held true for automatic processing. When feedback was delayed and blurred based on the hemodynamics of fMRI, this model learned more reliably from intermittent feedback compared to continuous feedback. These results suggest that different self-regulation mechanisms prefer different feedback timings, and that these factors can be effectively explored and optimized via simulation prior to deployment in the MRI scanner.

By repeatedly stimulating fine-grained patterns of neural activity, it is possible to manipulate behaviors associated with these patterns. While millimeter-scale patterns cannot yet be targeted with noninvasive brain stimulation, some people can learn to self-stimulate these activity patterns if they receive real-time feedback of their own recorded brain activity through a procedure known as fMRI neurofeedback. Other ‘non-responders’ are, for reasons unknown, unable to learn how to self-regulate these patterns. Here, we investigate how the properties of the fMRI signal, feedback timing, and self-regulation strategies may lead to this non-responder effect. The signal recorded by fMRI is related to blood flow in the brain and can be delayed by up to six seconds relative to underlying neural activity, which makes it difficult to learn. Because experiments in the MRI scanner are costly and time-consuming, we created a simulated neurofeedback environment to compare continuous versus intermittent feedback timing and cognitive versus automatic self-regulation strategies. In a cognitive experiment with human participants playing a simple game with the simulated neurofeedback signal, we found continuous feedback led to faster learning. However, in a computer model of automatic reward-based learning, we found that intermittent feedback was more reliable. These results will help improve future fMRI neurofeedback experiments and treatments by improving the efficacy of neurofeedback training procedures.

Resting-State Functional Connectivity-Based Biomarkers and Functional MRI-Based Neurofeedback for Psychiatric Disorders: A Challenge for Developing Theranostic Biomarkers

Psychiatric research has been hampered by an explanatory gap between psychiatric symptoms and their neural underpinnings, which has resulted in poor treatment outcomes. This situation has prompted us to shift from symptom-based diagnosis to data-driven diagnosis, aiming to redefine psychiatric disorders as disorders of neural circuitry. Promising candidates for data-driven diagnosis include resting-state functional connectivity MRI (rs-fcMRI)-based biomarkers. Although biomarkers have been developed with the aim of diagnosing patients and predicting the efficacy of therapy, the focus has shifted to the identification of biomarkers that represent therapeutic targets, which would allow for more personalized treatment approaches. This type of biomarker (i.e., "theranostic biomarker") is expected to elucidate the disease mechanism of psychiatric conditions and to offer an individualized neural circuit-based therapeutic target based on the neural cause of a condition. To this end, researchers have developed rs-fcMRI-based biomarkers and investigated a causal relationship between potential biomarkers and disease-specific behavior using functional MRI (fMRI)-based neurofeedback on functional connectivity. In this review, we introduce a recent approach for creating a theranostic biomarker, which consists mainly of 2 parts: (1) developing an rs-fcMRI-based biomarker that can predict diagnosis and/or symptoms with high accuracy, and (2) the introduction of a proof-of-concept study investigating the relationship between normalizing the biomarker and symptom changes using fMRI-based neurofeedback. In parallel with the introduction of recent studies, we review rs-fcMRI-based biomarker and fMRI-based neurofeedback, focusing on the technological improvements and limitations associated with clinical use.

OpenNFT: An open-source Python/Matlab framework for real-time fMRI neurofeedback training based on activity, connectivity and multivariate pattern analysis

Neurofeedback based on real-time functional magnetic resonance imaging (rt-fMRI) is a novel and rapidly developing research field. It allows for training of voluntary control over localized brain activity and connectivity and has demonstrated promising clinical applications. Because of the rapid technical developments of MRI techniques and the availability of high-performance computing, new methodological advances in rt-fMRI neurofeedback become possible. Here we outline the core components of a novel open-source neurofeedback framework, termed Open NeuroFeedback Training (OpenNFT), which efficiently integrates these new developments. This framework is implemented using Python and Matlab source code to allow for diverse functionality, high modularity, and rapid extendibility of the software depending on the user's needs. In addition, it provides an easy interface to the functionality of Statistical Parametric Mapping (SPM) that is also open-source and one of the most widely used fMRI data analysis software. We demonstrate the functionality of our new framework by describing case studies that include neurofeedback protocols based on brain activity levels, effective connectivity models, and pattern classification approaches. This open-source initiative provides a suitable framework to actively engage in the development of novel neurofeedback approaches, so that local methodological developments can be easily made accessible to a wider range of users.

A network engineering perspective on probing and perturbing cognition with neurofeedback

Network science and engineering provide a flexible and generalizable tool set to describe and manipulate complex systems characterized by heterogeneous interaction patterns among component parts. While classically applied to social systems, these tools have recently proven to be particularly useful in the study of the brain. In this review, we describe the nascent use of these tools to understand human cognition, and we discuss their utility in informing the meaningful and predictable perturbation of cognition in combination with the emerging capabilities of neurofeedback. To blend these disparate strands of research, we build on emerging conceptualizations of how the brain functions (as a complex network) and how we can develop and target interventions or modulations (as a form of network control). We close with an outline of current frontiers that bridge neurofeedback, connectomics, and network control theory to better understand human cognition.

Decoded fMRI neurofeedback can induce bidirectional confidence changes within single participants

Neurofeedback studies using real-time functional magnetic resonance imaging (rt-fMRI) have recently incorporated the multi-voxel pattern decoding approach, allowing for fMRI to serve as a tool to manipulate fine-grained neural activity embedded in voxel patterns. Because of its tremendous potential for clinical applications, certain questions regarding decoded neurofeedback (DecNef) must be addressed. Specifically, can the same participants learn to induce neural patterns in opposite directions in different sessions? If so, how does previous learning affect subsequent induction effectiveness? These questions are critical because neurofeedback effects can last for months, but the short- to mid-term dynamics of such effects are unknown. Here we employed a within-subjects design, where participants underwent two DecNef training sessions to induce behavioural changes of opposing directionality (up or down regulation of perceptual confidence in a visual discrimination task), with the order of training counterbalanced across participants. Behavioral results indicated that the manipulation was strongly influenced by the order and the directionality of neurofeedback training. We applied nonlinear mathematical modeling to parametrize four main consequences of DecNef: main effect of change in confidence, strength of down-regulation of confidence relative to up-regulation, maintenance of learning effects, and anterograde learning interference. Modeling results revealed that DecNef successfully induced bidirectional confidence changes in different sessions within single participants. Furthermore, the effect of up- compared to down-regulation was more prominent, and confidence changes (regardless of the direction) were largely preserved even after a week-long interval. Lastly, the effect of the second session was markedly diminished as compared to the effect of the first session, indicating strong anterograde learning interference. These results are interpreted in the framework of reinforcement learning and provide important implications for its application to basic neuroscience, to occupational and sports training, and to therapy.

Multivoxel neurofeedback selectively modulates confidence without changing perceptual performance

A central controversy in metacognition studies concerns whether subjective confidence directly reflects the reliability of perceptual or cognitive processes, as suggested by normative models based on the assumption that neural computations are generally optimal. This view enjoys popularity in the computational and animal literatures, but it has also been suggested that confidence may depend on a late-stage estimation dissociable from perceptual processes. Yet, at least in humans, experimental tools have lacked the power to resolve these issues convincingly. Here, we overcome this difficulty by using the recently developed method of decoded neurofeedback (DecNef) to systematically manipulate multivoxel correlates of confidence in a frontoparietal network. Here we report that bi-directional changes in confidence do not affect perceptual accuracy. Further psychophysical analyses rule out accounts based on simple shifts in reporting strategy. Our results provide clear neuroscientific evidence for the systematic dissociation between confidence and perceptual performance, and thereby challenge current theoretical thinking.

Confidence associated with perceptual judgements is generally seen as directly reflecting the reliability of perceptual processes. Here the authors use fMRI-based decoded neurofeedback to manipulate confidence and show that it does not affect perceptual performance.

The Anatomy of Non-conscious Recognition Memory

Cortical regions as early as primary visual cortex have been implicated in recognition memory. Here, we outline the challenges that this presents for neurobiological accounts of recognition memory. We conclude that understanding the role of early visual cortex (EVC) in this process will require the use of protocols that mask stimuli from visual awareness.

Differential Activation Patterns in the Same Brain Region Led to Opposite Emotional States

In human studies, how averaged activation in a brain region relates to human behavior has been extensively investigated. This approach has led to the finding that positive and negative facial preferences are represented by different brain regions. However, using a functional magnetic resonance imaging (fMRI) decoded neurofeedback (DecNef) method, we found that different patterns of neural activations within the cingulate cortex (CC) play roles in representing opposite directions of facial preference. In the present study, while neutrally preferred faces were presented, multi-voxel activation patterns in the CC that corresponded to higher (or lower) preference were repeatedly induced by fMRI DecNef. As a result, previously neutrally preferred faces became more (or less) preferred. We conclude that a different activation pattern in the CC, rather than averaged activation in a different area, represents and suffices to determine positive or negative facial preference. This new approach may reveal the importance of an activation pattern within a brain region in many cognitive functions.

A newly developed fMRI method, decoded neurofeedback (DecNef), reveals that specific activation patterns in the cingulate cortex are largely responsible for determining human facial preferences.

Although it is well studied how averaged activation of a brain region relates to behavior, it is still unclear if specific patterns of activation within regions also relate to cognitive function. In recent years, several methods have been developed for manipulating brain activity in humans. Real-time functional magnetic resonance imaging decoded neurofeedback (fMRI DecNef) is a method that allows the induction of specific patterns of brain activity by measuring the current pattern, comparing this to the pattern to be induced, and giving the subjects feedback on how close the two patterns of neuronal activity are. Using fMRI DecNef, we manipulated the pattern of activation in the cingulate cortex—a part of the cerebral cortex that plays a role in preference to different categories including faces and daily items—and tested whether we could change these preferences. In the experiment, a specific activation pattern in the cingulate cortex corresponding to higher (or lower) preference was induced by fMRI DecNef while subjects were seeing a neutrally preferred face. As a result, these neutrally preferred faces became more (or less) preferred. Our finding suggests that different patterns of activation in the cingulate cortex represent, and are sufficient to determine, different emotional states. Our new approach using fMRI DecNef may reveal the importance of activation patterns within a brain region, rather than activation in a whole region, in many cognitive functions.