Direct speech reconstruction from sensorimotor brain activity with optimized deep learning models

Objective.Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field.Approach.In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task.Main results.We show that (1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; (2) individual word decoding in reconstructed speech achieves 92%-100% accuracy (chance level is 8%); (3) direct reconstruction from sensorimotor brain activity produces intelligible speech.Significance.These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

PhenoScore quantifies phenotypic variation for rare genetic diseases by combining facial analysis with other clinical features using a machine-learning framework

Several molecular and phenotypic algorithms exist that establish genotype-phenotype correlations, including facial recognition tools. However, no unified framework that investigates both facial data and other phenotypic data directly from individuals exists. We developed PhenoScore: an open-source, artificial intelligence-based phenomics framework, combining facial recognition technology with Human Phenotype Ontology data analysis to quantify phenotypic similarity. Here we show PhenoScore's ability to recognize distinct phenotypic entities by establishing recognizable phenotypes for 37 of 40 investigated syndromes against clinical features observed in individuals with other neurodevelopmental disorders and show it is an improvement on existing approaches. PhenoScore provides predictions for individuals with variants of unknown significance and enables sophisticated genotype-phenotype studies by testing hypotheses on possible phenotypic (sub)groups. PhenoScore confirmed previously known phenotypic subgroups caused by variants in the same gene for SATB1, SETBP1 and DEAF1 and provides objective clinical evidence for two distinct ADNP-related phenotypes, already established functionally.

The neuroconnectionist research programme

Artificial neural networks (ANNs) inspired by biology are beginning to be widely used to model behavioural and neural data, an approach we call 'neuroconnectionism'. ANNs have been not only lauded as the current best models of information processing in the brain but also criticized for failing to account for basic cognitive functions. In this Perspective article, we propose that arguing about the successes and failures of a restricted set of current ANNs is the wrong approach to assess the promise of neuroconnectionism for brain science. Instead, we take inspiration from the philosophy of science, and in particular from Lakatos, who showed that the core of a scientific research programme is often not directly falsifiable but should be assessed by its capacity to generate novel insights. Following this view, we present neuroconnectionism as a general research programme centred around ANNs as a computational language for expressing falsifiable theories about brain computation. We describe the core of the programme, the underlying computational framework and its tools for testing specific neuroscientific hypotheses and deriving novel understanding. Taking a longitudinal view, we review past and present neuroconnectionist projects and their responses to challenges and argue that the research programme is highly progressive, generating new and otherwise unreachable insights into the workings of the brain.

Towards Naturalistic Speech Decoding from Intracranial Brain Data

Speech decoding from brain activity can enable development of brain-computer interfaces (BCIs) to restore naturalistic communication in paralyzed patients. Previous work has focused on development of decoding models from isolated speech data with a clean background and multiple repetitions of the material. In this study, we describe a novel approach to speech decoding that relies on a generative adversarial neural network (GAN) to reconstruct speech from brain data recorded during a naturalistic speech listening task (watching a movie). We compared the GAN-based approach, where reconstruction was done from the compressed latent representation of sound decoded from the brain, with several baseline models that reconstructed sound spectrogram directly. We show that the novel approach provides more accurate reconstructions compared to the baselines. These results underscore the potential of GAN models for speech decoding in naturalistic noisy environments and further advancing of BCIs for naturalistic communication. Clinical Relevance - This study presents a novel speech decoding paradigm that combines advances in deep learning, speech synthesis and neural engineering, and has the potential to advance the field of BCI for severely paralyzed individuals.

Bayesian model averaging for nonparametric discontinuity design

Quasi-experimental research designs, such as regression discontinuity and interrupted time series, allow for causal inference in the absence of a randomized controlled trial, at the cost of additional assumptions. In this paper, we provide a framework for discontinuity-based designs using Bayesian model averaging and Gaussian process regression, which we refer to as ‘Bayesian nonparametric discontinuity design’, or BNDD for short. BNDD addresses the two major shortcomings in most implementations of such designs: overconfidence due to implicit conditioning on the alleged effect, and model misspecification due to reliance on overly simplistic regression models. With the appropriate Gaussian process covariance function, our approach can detect discontinuities of any order, and in spectral features. We demonstrate the usage of BNDD in simulations, and apply the framework to determine the effect of running for political positions on longevity, of the effect of an alleged historical phantom border in the Netherlands on Dutch voting behaviour, and of Kundalini Yoga meditation on heart rate.

Phenotype based prediction of exome sequencing outcome using machine learning for neurodevelopmental disorders

PURPOSE

Although the introduction of exome sequencing (ES) has led to the diagnosis of a significant portion of patients with neurodevelopmental disorders (NDDs), the diagnostic yield in actual clinical practice has remained stable at approximately 30%. We hypothesized that improving the selection of patients to test on the basis of their phenotypic presentation will increase diagnostic yield and therefore reduce unnecessary genetic testing.

METHODS

We tested 4 machine learning methods and developed PredWES from these: a statistical model predicting the probability of a positive ES result solely on the basis of the phenotype of the patient.

RESULTS

We first trained the tool on 1663 patients with NDDs and subsequently showed that diagnostic ES on the top 10% of patients with the highest probability of a positive ES result would provide a diagnostic yield of 56%, leading to a notable 114% increase. Inspection of our model revealed that for patients with NDDs, comorbid abnormal (lower) muscle tone and microcephaly positively correlated with a conclusive ES diagnosis, whereas autism was negatively associated with a molecular diagnosis.

CONCLUSION

In conclusion, PredWES allows prioritizing patients with NDDs eligible for diagnostic ES on the basis of their phenotypic presentation to increase the diagnostic yield, making a more efficient use of health care resources.

No Evidence for Neural Overlap between Unconsciously Processed and Imagined Stimuli

Visual representations can be generated via feedforward or feedback processes. The extent to which these processes result in overlapping representations remains unclear. Previous work has shown that imagined stimuli elicit similar representations as perceived stimuli throughout the visual cortex. However, while representations during imagery are indeed only caused by feedback processing, neural processing during perception is an interplay of both feedforward and feedback processing. This means that any representational overlap could be because of overlap in feedback processes. In the current study, we aimed to investigate this issue by characterizing the overlap between feedforward- and feedback-initiated category representations during imagined stimuli, conscious perception, and unconscious processing using fMRI in humans of either sex. While all three conditions elicited stimulus representations in left lateral occipital cortex (LOC), significant similarities were observed only between imagery and conscious perception in this area. Furthermore, connectivity analyses revealed stronger connectivity between frontal areas and left LOC during conscious perception and in imagery compared with unconscious processing. Together, these findings can be explained by the idea that long-range feedback modifies visual representations, thereby reducing representational overlap between purely feedforward- and feedback-initiated stimulus representations measured by fMRI. Neural representations influenced by feedback, either stimulus driven (perception) or purely internally driven (imagery), are, however, relatively similar.

Cortical network responses map onto data-driven features that capture visual semantics of movie fragments

Research on how the human brain extracts meaning from sensory input relies in principle on methodological reductionism. In the present study, we adopt a more holistic approach by modeling the cortical responses to semantic information that was extracted from the visual stream of a feature film, employing artificial neural network models. Advances in both computer vision and natural language processing were utilized to extract the semantic representations from the film by combining perceptual and linguistic information. We tested whether these representations were useful in studying the human brain data. To this end, we collected electrocorticography responses to a short movie from 37 subjects and fitted their cortical patterns across multiple regions using the semantic components extracted from film frames. We found that individual semantic components reflected fundamental semantic distinctions in the visual input, such as presence or absence of people, human movement, landscape scenes, human faces, etc. Moreover, each semantic component mapped onto a distinct functional cortical network involving high-level cognitive regions in occipitotemporal, frontal and parietal cortices. The present work demonstrates the potential of the data-driven methods from information processing fields to explain patterns of cortical responses, and contributes to the overall discussion about the encoding of high-level perceptual information in the human brain.

Adaptive time scales in recurrent neural networks

Recent experiments have revealed a hierarchy of time scales in the visual cortex, where different stages of the visual system process information at different time scales. Recurrent neural networks are ideal models to gain insight in how information is processed by such a hierarchy of time scales and have become widely used to model temporal dynamics both in machine learning and computational neuroscience. However, in the derivation of such models as discrete time approximations of the firing rate of a population of neurons, the time constants of the neuronal process are generally ignored. Learning these time constants could inform us about the time scales underlying temporal processes in the brain and enhance the expressive capacity of the network. To investigate the potential of adaptive time constants, we compare the standard approximations to a more lenient one that accounts for the time scales at which processes unfold. We show that such a model performs better on predicting simulated neural data and allows recovery of the time scales at which the underlying processes unfold. A hierarchy of time scales emerges when adapting to data with multiple underlying time scales, underscoring the importance of such a hierarchy in processing complex temporal information.

Brain-optimized extraction of complex sound features that drive continuous auditory perception

Understanding how the human brain processes auditory input remains a challenge. Traditionally, a distinction between lower- and higher-level sound features is made, but their definition depends on a specific theoretical framework and might not match the neural representation of sound. Here, we postulate that constructing a data-driven neural model of auditory perception, with a minimum of theoretical assumptions about the relevant sound features, could provide an alternative approach and possibly a better match to the neural responses. We collected electrocorticography recordings from six patients who watched a long-duration feature film. The raw movie soundtrack was used to train an artificial neural network model to predict the associated neural responses. The model achieved high prediction accuracy and generalized well to a second dataset, where new participants watched a different film. The extracted bottom-up features captured acoustic properties that were specific to the type of sound and were associated with various response latency profiles and distinct cortical distributions. Specifically, several features encoded speech-related acoustic properties with some features exhibiting shorter latency profiles (associated with responses in posterior perisylvian cortex) and others exhibiting longer latency profiles (associated with responses in anterior perisylvian cortex). Our results support and extend the current view on speech perception by demonstrating the presence of temporal hierarchies in the perisylvian cortex and involvement of cortical sites outside of this region during audiovisual speech perception.

Evidence for confounding eye movements under attempted fixation and active viewing in cognitive neuroscience

Eye movements can have serious confounding effects in cognitive neuroscience experiments. Therefore, participants are commonly asked to fixate. Regardless, participants will make so-called fixational eye movements under attempted fixation, which are thought to be necessary to prevent perceptual fading. Neural changes related to these eye movements could potentially explain previously reported neural decoding and neuroimaging results under attempted fixation. In previous work, under attempted fixation and passive viewing, we found no evidence for systematic eye movements. Here, however, we show that participants' eye movements are systematic under attempted fixation when active viewing is demanded by the task. Since eye movements directly affect early visual cortex activity, commonly used for neural decoding, our findings imply alternative explanations for previously reported results in neural decoding.

Validation of structural brain connectivity networks: The impact of scanning parameters

Evaluation of the structural connectivity (SC) of the brain based on tractography has mainly focused on the choice of diffusion model, tractography algorithm, and their respective parameter settings. Here, we systematically validate SC derived from a post mortem monkey brain, while varying key acquisition parameters such as the b-value, gradient angular resolution and image resolution. As gold standard we use the connectivity matrix obtained invasively with histological tracers by Markov et al. (2014). As performance metric, we use cross entropy as a measure that enables comparison of the relative tracer labeled neuron counts to the streamline counts from tractography. We find that high angular resolution and high signal-to-noise ratio are important to estimate SC, and that SC derived from low image resolution (1.0³ mm³) are in better agreement with the tracer network, than those derived from high image resolution (0.5³ mm³) or at an even lower image resolution (2.0³ mm³). In contradiction, sensitivity and specificity analyses suggest that if the angular resolution is sufficient, the balanced compromise in which sensitivity and specificity are identical remains 60-64% regardless of the other scanning parameters. Interestingly, the tracer graph is assumed to be the gold standard but by thresholding, the balanced compromise increases to 70-75%. Hence, by using performance metrics based on binarized tracer graphs, one risks losing important information, changing the performance of SC graphs derived by tractography and their dependence of different scanning parameters.

Neuroimaging Findings on Amodal Completion: A Review

Amodal completion is the phenomenon of perceiving completed objects even though physically they are partially occluded. In this review, we provide an extensive overview of the results obtained from a variety of neuroimaging studies on the neural correlates of amodal completion. We discuss whether low-level and high-level cortical areas are implicated in amodal completion; provide an overview of how amodal completion unfolds over time while dissociating feedforward, recurrent, and feedback processes; and discuss how amodal completion is represented at the neuronal level. The involvement of low-level visual areas such as V1 and V2 is not yet clear, while several high-level structures such as the lateral occipital complex and fusiform face area seem invariant to occlusion of objects and faces, respectively, and several motor areas seem to code for object permanence. The variety of results on the timing of amodal completion hints to a mixture of feedforward, recurrent, and feedback processes. We discuss whether the invisible parts of the occluded object are represented as if they were visible, contrary to a high-level representation. While plenty of questions on amodal completion remain, this review presents an overview of the neuroimaging findings reported to date, summarizes several insights from computational models, and connects research of other perceptual completion processes such as modal completion. In all, it is suggested that amodal completion is the solution to deal with various types of incomplete retinal information, and highly depends on stimulus complexity and saliency, and therefore also give rise to a variety of observed neural patterns.

Dynamic decomposition of spatiotemporal neural signals

Neural signals are characterized by rich temporal and spatiotemporal dynamics that reflect the organization of cortical networks. Theoretical research has shown how neural networks can operate at different dynamic ranges that correspond to specific types of information processing. Here we present a data analysis framework that uses a linearized model of these dynamic states in order to decompose the measured neural signal into a series of components that capture both rhythmic and non-rhythmic neural activity. The method is based on stochastic differential equations and Gaussian process regression. Through computer simulations and analysis of magnetoencephalographic data, we demonstrate the efficacy of the method in identifying meaningful modulations of oscillatory signals corrupted by structured temporal and spatiotemporal noise. These results suggest that the method is particularly suitable for the analysis and interpretation of complex temporal and spatiotemporal neural signals.

In neuroscience, researchers are often interested in the modulations of specific signal components (e.g., oscillations in a particular frequency band), that have to be extracted from a background of both rhythmic and non-rhythmic activity. As the interfering background signals often have higher amplitude than the component of interest, it is crucial to develop methods that are able to perform some sort of signal decomposition. In this paper, we introduce a Bayesian decomposition method that exploits a prior dynamical model of the neural temporal dynamics in order to extract signal components with well-defined dynamic features. The method is based on Gaussian process regression with prior distributions determined by the covariance functions of linear stochastic differential equations. Using simulations and analysis of real MEG data, we show that these informed prior distributions allow for the extraction of interpretable dynamic components and the estimation of relevant signal modulations. We generalize the method to the analysis of spatiotemporal cortical activity and show that the framework is intimately related to well-established source-reconstruction techniques.

Modeling the Dynamics of Human Brain Activity with Recurrent Neural Networks

Encoding models are used for predicting brain activity in response to sensory stimuli with the objective of elucidating how sensory information is represented in the brain. Encoding models typically comprise a nonlinear transformation of stimuli to features (feature model) and a linear convolution of features to responses (response model). While there has been extensive work on developing better feature models, the work on developing better response models has been rather limited. Here, we investigate the extent to which recurrent neural network models can use their internal memories for nonlinear processing of arbitrary feature sequences to predict feature-evoked response sequences as measured by functional magnetic resonance imaging. We show that the proposed recurrent neural network models can significantly outperform established response models by accurately estimating long-term dependencies that drive hemodynamic responses. The results open a new window into modeling the dynamics of brain activity in response to sensory stimuli.

The missing link: Predicting connectomes from noisy and partially observed tract tracing data

Our understanding of the wiring map of the brain, known as the connectome, has increased greatly in the last decade, mostly due to technological advancements in neuroimaging techniques and improvements in computational tools to interpret the vast amount of available data. Despite this, with the exception of the C. elegans roundworm, no definitive connectome has been established for any species. In order to obtain this, tracer studies are particularly appealing, as these have proven highly reliable. The downside of tract tracing is that it is costly to perform, and can only be applied ex vivo. In this paper, we suggest that instead of probing all possible connections, hitherto unknown connections may be predicted from the data that is already available. Our approach uses a 'latent space model' that embeds the connectivity in an abstract physical space. Regions that are close in the latent space have a high chance of being connected, while regions far apart are most likely disconnected in the connectome. After learning the latent embedding from the connections that we did observe, the latent space allows us to predict connections that have not been probed previously. We apply the methodology to two connectivity data sets of the macaque, where we demonstrate that the latent space model is successful in predicting unobserved connectivity, outperforming two baselines and an alternative model in nearly all cases. Furthermore, we show how the latent spatial embedding may be used to integrate multimodal observations (i.e. anterograde and retrograde tracers) for the mouse neocortex. Finally, our probabilistic approach enables us to make explicit which connections are easy to predict and which prove difficult, allowing for informed follow-up studies.

Let's Not Waste Time: Using Temporal Information in Clustered Activity Estimation with Spatial Adjacency Restrictions (CAESAR) for Parcellating FMRI Data

We have proposed a Bayesian approach for functional parcellation of whole-brain FMRI measurements which we call Clustered Activity Estimation with Spatial Adjacency Restrictions (CAESAR). We use distance-dependent Chinese restaurant processes (dd-CRPs) to define a flexible prior which partitions the voxel measurements into clusters whose number and shapes are unknown a priori. With dd-CRPs we can conveniently implement spatial constraints to ensure that our parcellations remain spatially contiguous and thereby physiologically meaningful. In the present work, we extend CAESAR by using Gaussian process (GP) priors to model the temporally smooth haemodynamic signals that give rise to the measured FMRI data. A challenge for GP inference in our setting is the cubic scaling with respect to the number of time points, which can become computationally prohibitive with FMRI measurements, potentially consisting of long time series. As a solution we describe an efficient implementation that is practically as fast as the corresponding time-independent non-GP model with typically-sized FMRI data sets. We also employ a population Monte-Carlo algorithm that can significantly speed up convergence compared to traditional single-chain methods. First we illustrate the benefits of CAESAR and the GP priors with simulated experiments. Next, we demonstrate our approach by parcellating resting state FMRI data measured from twenty participants as taken from the Human Connectome Project data repository. Results show that CAESAR affords highly robust and scalable whole-brain clustering of FMRI timecourses.

McGurk illusion recalibrates subsequent auditory perception

Visual information can alter auditory perception. This is clearly illustrated by the well-known McGurk illusion, where an auditory/aba/ and a visual /aga/ are merged to the percept of ‘ada’. It is less clear however whether such a change in perception may recalibrate subsequent perception. Here we asked whether the altered auditory perception due to the McGurk illusion affects subsequent auditory perception, i.e. whether this process of fusion may cause a recalibration of the auditory boundaries between phonemes. Participants categorized auditory and audiovisual speech stimuli as /aba/, /ada/ or /aga/ while activity patterns in their auditory cortices were recorded using fMRI. Interestingly, following a McGurk illusion, an auditory /aba/ was more often misperceived as ‘ada’. Furthermore, we observed a neural counterpart of this recalibration in the early auditory cortex. When the auditory input /aba/ was perceived as ‘ada’, activity patterns bore stronger resemblance to activity patterns elicited by /ada/ sounds than when they were correctly perceived as /aba/. Our results suggest that upon experiencing the McGurk illusion, the brain shifts the neural representation of an /aba/ sound towards /ada/, culminating in a recalibration in perception of subsequent auditory input.

Spatiotemporal Dynamics of Cortical Representations during and after Stimulus Presentation

Visual perception is a spatiotemporally complex process. In this study, we investigated cortical dynamics during and after stimulus presentation. We observed that visual category information related to the difference between faces and objects became apparent in the occipital lobe after 63 ms. Within the next 110 ms, activation spread out to include the temporal lobe before returning to residing mainly in the occipital lobe again. After stimulus offset, a peak in information was observed, comparable to the peak after stimulus onset. Moreover, similar processes, albeit not identical, seemed to underlie both peaks. Information about the categorical identity of the stimulus remained present until 677 ms after stimulus offset, during which period the stimulus had to be retained in working memory. Activation patterns initially resembled those observed during stimulus presentation. After about 200 ms, however, this representation changed and class-specific activity became more equally distributed over the four lobes. These results show that, although there are common processes underlying stimulus representation both during and after stimulus presentation, these representations change depending on the specific stage of perception and maintenance.

The spatiotemporal dynamics of binocular rivalry: evidence for increased top-down flow prior to a perceptual switch

According to most theories, perceptual switching during binocular rivalry is caused by competition between the neural representations of the two input images. It remains unclear whether competition is resolved already at the early stages of visual processing and that information about the dominant percept is then fed forward to more high-level areas or whether competition is first resolved in high-level areas and then fed back to lower levels. This study aimed to dissociate between these theories by investigating the direction of information flow prior to a perceptual switch, using Granger causality on classifier output originating from occipital, temporal, parietal and frontal regions of interest. The results point toward increased top-down information flow between temporal and occipital areas before a switch in dominance. These findings do not support a low-level account of binocular rivalry but are in line with high-level and hybrid explanations.

Preference for Audiovisual Speech Congruency in Superior Temporal Cortex

Auditory speech perception can be altered by concurrent visual information. The superior temporal cortex is an important combining site for this integration process. This area was previously found to be sensitive to audiovisual congruency. However, the direction of this congruency effect (i.e., stronger or weaker activity for congruent compared to incongruent stimulation) has been more equivocal. Here, we used fMRI to look at the neural responses of human participants during the McGurk illusion--in which auditory /aba/ and visual /aga/ inputs are fused to perceived /ada/--in a large homogenous sample of participants who consistently experienced this illusion. This enabled us to compare the neuronal responses during congruent audiovisual stimulation with incongruent audiovisual stimulation leading to the McGurk illusion while avoiding the possible confounding factor of sensory surprise that can occur when McGurk stimuli are only occasionally perceived. We found larger activity for congruent audiovisual stimuli than for incongruent (McGurk) stimuli in bilateral superior temporal cortex, extending into the primary auditory cortex. This finding suggests that superior temporal cortex prefers when auditory and visual input support the same representation.

Probabilistic clustering of the human connectome identifies communities and hubs

A fundamental assumption in neuroscience is that brain function is constrained by its structural properties. This motivates the idea that the brain can be parcellated into functionally coherent regions based on anatomical connectivity patterns that capture how different areas are interconnected. Several studies have successfully implemented this idea in humans using diffusion weighted MRI, allowing parcellation to be conducted in vivo. Two distinct approaches to connectivity-based parcellation can be identified. The first uses the connection profiles of brain regions as a feature vector, and groups brain regions with similar connection profiles together. Alternatively, one may adopt a network perspective that aims to identify clusters of brain regions that show dense within-cluster and sparse between-cluster connectivity. In this paper, we introduce a probabilistic model for connectivity-based parcellation that unifies both approaches. Using the model we are able to obtain a parcellation of the human brain whose clusters may adhere to either interpretation. We find that parts of the connectome consistently cluster as densely connected components, while other parts consistently result in clusters with similar connections. Interestingly, the densely connected components consist predominantly of major cortical areas, while the clusters with similar connection profiles consist of regions that have previously been identified as the 'rich club'; regions known for their integrative role in connectivity. Furthermore, the probabilistic model allows quantification of the uncertainty in cluster assignments. We show that, while most clusters are clearly delineated, some regions are more difficult to assign. These results indicate that care should be taken when interpreting connectivity-based parcellations obtained using alternative deterministic procedures.

Predicting the Semantic Category of Internally Generated Words from Neuromagnetic Recordings

In this study, we explore the possibility to predict the semantic category of words from brain signals in a free word generation task. Participants produced single words from different semantic categories in a modified semantic fluency task. A Bayesian logistic regression classifier was trained to predict the semantic category of words from single-trial MEG data. Significant classification accuracies were achieved using sensor-level MEG time series at the time interval of conceptual preparation. Semantic category prediction was also possible using source-reconstructed time series, based on minimum norm estimates of cortical activity. Brain regions that contributed most to classification on the source level were identified. These were the left inferior frontal gyrus, left middle frontal gyrus, and left posterior middle temporal gyrus. Additionally, the temporal dynamics of brain activity underlying the semantic preparation during word generation was explored. These results provide important insights about central aspects of language production.

Quantifying uncertainty in brain network measures using Bayesian connectomics

The wiring diagram of the human brain can be described in terms of graph measures that characterize structural regularities. These measures require an estimate of whole-brain structural connectivity for which one may resort to deterministic or thresholded probabilistic streamlining procedures. While these procedures have provided important insights about the characteristics of human brain networks, they ultimately rely on unwarranted assumptions such as those of noise-free data or the use of an arbitrary threshold. Therefore, resulting structural connectivity estimates as well as derived graph measures fail to fully take into account the inherent uncertainty in the structural estimate. In this paper, we illustrate an easy way of obtaining posterior distributions over graph metrics using Bayesian inference. It is shown that this posterior distribution can be used to quantify uncertainty about graph-theoretical measures at the single subject level, thereby providing a more nuanced view of the graph-theoretical properties of human brain connectivity. We refer to this model-based approach to connectivity analysis as Bayesian connectomics.

Modality-specific alpha modulations facilitate long-term memory encoding in the presence of distracters

It has been proposed that long-term memory encoding is not only dependent on engaging task-relevant regions but also on disengaging task-irrelevant regions. In particular, oscillatory alpha activity has been shown to be involved in shaping the functional architecture of the working brain because it reflects the functional disengagement of specific regions in attention and memory tasks. We here ask if such allocation of resources by alpha oscillations generalizes to long-term memory encoding in a cross-modal setting in which we acquired the ongoing brain activity using magnetoencephalography. Participants were asked to encode pictures while ignoring simultaneously presented words and vice versa. We quantified the brain activity during rehearsal reflecting subsequent memory in the different attention conditions. The key finding was that successful long-term memory encoding is reflected by alpha power decreases in the sensory region of the to-be-attended modality and increases in the sensory region of the to-be-ignored modality to suppress distraction during rehearsal period. Our results corroborate related findings from attention studies by demonstrating that alpha activity is also important for the allocation of resources during long-term memory encoding in the presence of distracters.

Unsupervised feature learning improves prediction of human brain activity in response to natural images

Encoding and decoding in functional magnetic resonance imaging has recently emerged as an area of research to noninvasively characterize the relationship between stimulus features and human brain activity. To overcome the challenge of formalizing what stimulus features should modulate single voxel responses, we introduce a general approach for making directly testable predictions of single voxel responses to statistically adapted representations of ecologically valid stimuli. These representations are learned from unlabeled data without supervision. Our approach is validated using a parsimonious computational model of (i) how early visual cortical representations are adapted to statistical regularities in natural images and (ii) how populations of these representations are pooled by single voxels. This computational model is used to predict single voxel responses to natural images and identify natural images from stimulus-evoked multiple voxel responses. We show that statistically adapted low-level sparse and invariant representations of natural images better span the space of early visual cortical representations and can be more effectively exploited in stimulus identification than hand-designed Gabor wavelets. Our results demonstrate the potential of our approach to better probe unknown cortical representations.

An important but difficult problem in contemporary cognitive neuroscience is to find what stimulus features best drive responses in the human brain. The conventional approach to solve this problem is to use descriptive encoding models that predict responses to stimulus features that are known a priori. In this study, we introduce an alternative to this approach that is independent of a priori knowledge. Instead, we use a normative encoding model that predicts responses to stimulus features that are learned from unlabeled data. We show that this normative encoding model learns sparse, topographic and invariant stimulus features from tens of thousands of grayscale natural image patches without supervision, and reproduces the population behavior of simple and complex cells. We find that these stimulus features significantly better drive blood-oxygen-level dependent hemodynamic responses in early visual areas than Gabor wavelets–the fundamental building blocks of the conventional approach. Our approach will improve our understanding of how sensory information is represented beyond early visual areas since it can theoretically find what stimulus features best drive responses in other sensory areas.

Online decoding of object-based attention using real-time fMRI

Visual attention is used to selectively filter relevant information depending on current task demands and goals. Visual attention is called object-based attention when it is directed to coherent forms or objects in the visual field. This study used real-time functional magnetic resonance imaging for moment-to-moment decoding of attention to spatially overlapped objects belonging to two different object categories. First, a whole-brain classifier was trained on pictures of faces and places. Subjects then saw transparently overlapped pictures of a face and a place, and attended to only one of them while ignoring the other. The category of the attended object, face or place, was decoded on a scan-by-scan basis using the previously trained decoder. The decoder performed at 77.6% accuracy indicating that despite competing bottom-up sensory input, object-based visual attention biased neural patterns towards that of the attended object. Furthermore, a comparison between different classification approaches indicated that the representation of faces and places is distributed rather than focal. This implies that real-time decoding of object-based attention requires a multivariate decoding approach that can detect these distributed patterns of cortical activity.

Bayesian sparse partial least squares

Partial least squares (PLS) is a class of methods that makes use of a set of latent or unobserved variables to model the relation between (typically) two sets of input and output variables, respectively. Several flavors, depending on how the latent variables or components are computed, have been developed over the last years. In this letter, we propose a Bayesian formulation of PLS along with some extensions. In a nutshell, we provide sparsity at the input space level and an automatic estimation of the optimal number of latent components. We follow the variational approach to infer the parameter distributions. We have successfully tested the proposed methods on a synthetic data benchmark and on electrocorticogram data associated with several motor outputs in monkeys.

Detecting semantic priming at the single-trial level

Semantic priming is usually studied by examining ERPs over many trials and subjects. This article aims at detecting semantic priming at the single-trial level. By using machine learning techniques it is possible to analyse and classify short traces of brain activity, which could, for example, be used to build a Brain Computer Interface (BCI). This article describes an experiment where subjects were presented with word pairs and asked to decide whether the words were related or not. A classifier was trained to determine whether the subjects judged words as related or unrelated based on one second of EEG data. The results show that the classifier accuracy when training per subject varies between 54% and 67%, and is significantly above chance level for all subjects (N  = 12) and the accuracy when training over subjects varies between 51% and 63%, and is significantly above chance level for 11 subjects, pointing to a general effect.

Decoding the memorization of individual stimuli with direct human brain recordings

Through decades of research, neuroscientists and clinicians have identified an array of brain areas that each activate when a person views a certain category of stimuli. However, we do not have a detailed understanding of how the brain represents individual stimuli within a category. Here we used direct human brain recordings and machine-learning algorithms to characterize the distributed patterns that distinguish specific cognitive states. Epilepsy patients with surgically implanted electrodes performed a working-memory task and we used machine-learning algorithms to predict the identity of each viewed stimulus. We found that the brain's representation of stimulus-specific information is distributed across neural activity at multiple frequencies, electrodes, and timepoints. Stimulus-specific neuronal activity was most prominent in the high-gamma (65-128 Hz) and theta/alpha (4-16 Hz) bands, but the properties of these signals differed significantly between individuals and for novel stimuli compared to common ones. Our findings are helpful for understanding the neural basis of memory and developing brain-computer interfaces by showing that the brain distinguishes specific cognitive states by diverse spatiotemporal patterns of neuronal.

Bayesian inference of structural brain networks

Structural brain networks are used to model white-matter connectivity between spatially segregated brain regions. The presence, location and orientation of these white matter tracts can be derived using diffusion-weighted magnetic resonance imaging in combination with probabilistic tractography. Unfortunately, as of yet, none of the existing approaches provide an undisputed way of inferring brain networks from the streamline distributions which tractography produces. State-of-the-art methods rely on an arbitrary threshold or, alternatively, yield weighted results that are difficult to interpret. In this paper, we provide a generative model that explicitly describes how structural brain networks lead to observed streamline distributions. This allows us to draw principled conclusions about brain networks, which we validate using simultaneously acquired resting-state functional MRI data. Inference may be further informed by means of a prior which combines connectivity estimates from multiple subjects. Based on this prior, we obtain networks that significantly improve on the conventional approach.

On the decoding of intracranial data using sparse orthonormalized partial least squares

It has recently been shown that robust decoding of motor output from electrocorticogram signals in monkeys over prolonged periods of time has become feasible (Chao et al 2010 Front. Neuroeng. 3 1-10). In order to achieve these results, multivariate partial least-squares (PLS) regression was used. PLS uses a set of latent variables, referred to as components, to model the relationship between the input and the output data and is known to handle high-dimensional and possibly strongly correlated inputs and outputs well. We developed a new decoding method called sparse orthonormalized partial least squares (SOPLS) which was tested on a subset of the data used in Chao et al (2010) (freely obtainable from neurotycho.org (Nagasaka et al 2011 PLoS ONE 6 e22561)). We show that SOPLS reaches the same decoding performance as PLS using just two sparse components which can each be interpreted as encoding particular combinations of motor parameters. Furthermore, the sparse solution afforded by the SOPLS model allowed us to show the functional involvement of beta and gamma band responses in premotor and motor cortex for predicting the first component. Based on the literature, we conjecture that this first component is involved in the encoding of movement direction. Hence, the sparse and compact representation afforded by the SOPLS model facilitates interpretation of which spectral, spatial and temporal components are involved in successful decoding. These advantages make the proposed decoding method an important new tool in neuroprosthetics.

Using brain-computer interfaces and brain-state dependent stimulation as tools in cognitive neuroscience

Large efforts are currently being made to develop and improve online analysis of brain activity which can be used, e.g., for brain-computer interfacing (BCI). A BCI allows a subject to control a device by willfully changing his/her own brain activity. BCI therefore holds the promise as a tool for aiding the disabled and for augmenting human performance. While technical developments obviously are important, we will here argue that new insight gained from cognitive neuroscience can be used to identify signatures of neural activation which reliably can be modulated by the subject at will. This review will focus mainly on oscillatory activity in the alpha band which is strongly modulated by changes in covert attention. Besides developing BCIs for their traditional purpose, they might also be used as a research tool for cognitive neuroscience. There is currently a strong interest in how brain-state fluctuations impact cognition. These state fluctuations are partly reflected by ongoing oscillatory activity. The functional role of the brain state can be investigated by introducing stimuli in real-time to subjects depending on the actual state of the brain. This principle of brain-state dependent stimulation may also be used as a practical tool for augmenting human behavior. In conclusion, new approaches based on online analysis of ongoing brain activity are currently in rapid development. These approaches are amongst others informed by new insight gained from electroencephalography/magnetoencephalography studies in cognitive neuroscience and hold the promise of providing new ways for investigating the brain at work.

Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention

Background

Visual brain-computer interfaces (BCIs) often yield high performance only when targets are fixated with the eyes. Furthermore, many paradigms use intense visual stimulation, which can be irritating especially in long BCI sessions. However, BCIs can more directly directly tap the neural processes underlying visual attention. Covert shifts of visual attention induce changes in oscillatory alpha activity in posterior cortex, even in the absence of visual stimulation. The aim was to investigate whether different pairs of directions of attention shifts can be reliably differentiated based on the electroencephalogram. To this end, healthy participants (N = 8) had to strictly fixate a central dot and covertly shift visual attention to one out of six cued directions.

Results

Covert attention shifts induced a prolonged alpha synchronization over posterior electrode sites (PO and O electrodes). Spectral changes had specific topographies so that different pairs of directions could be differentiated. There was substantial variation across participants with respect to the direction pairs that could be reliably classified. Mean accuracy for the best-classifiable pair amounted to 74.6%. Furthermore, an alpha power index obtained during a relaxation measurement showed to be predictive of peak BCI performance (r = .66).

Conclusions

Results confirm posterior alpha power modulations as a viable input modality for gaze-independent EEG-based BCIs. The pair of directions yielding optimal performance varies across participants. Consequently, participants with low control for standard directions such as left-right might resort to other pairs of directions including top and bottom. Additionally, a simple alpha index was shown to predict prospective BCI performance.

Lateralized responses during covert attention are modulated by target eccentricity

Various studies have demonstrated that covert attention to different locations in the visual field can be used as a control signal for brain computer interfacing. It is well known that when covert attention is directed to the left visual hemifield, posterior alpha activity decreases in the right hemisphere while simultaneously increasing in the left hemisphere and vice versa. However, it remains unknown if and how the classical lateralization pattern depends on the eccentricity of the locations to which one attends. In this paper we study the effect of target eccentricity on the performance of a brain computer interface system that is driven by covert attention. Results show that the lateralization pattern becomes more pronounced as target eccentricity increases and suggest that in the current design the minimum eccentricity for having an acceptable classification performance for two targets at equal distance from fixation in opposite hemifields is about 6° of visual angle.

Neural Decoding with Hierarchical Generative Models

Recent research has shown that reconstruction of perceived images based on hemodynamic response as measured with functional magnetic resonance imaging (fMRI) is starting to become feasible. In this letter, we explore reconstruction based on a learned hierarchy of features by employing a hierarchical generative model that consists of conditional restricted Boltzmann machines. In an unsupervised phase, we learn a hierarchy of features from data, and in a supervised phase, we learn how brain activity predicts the states of those features. Reconstruction is achieved by sampling from the model, conditioned on brain activity. We show that by using the hierarchical generative model, we can obtain good-quality reconstructions of visual images of handwritten digits presented during an fMRI scanning session.

Dynamic Bayesian networks as prognostic models for clinical patient management

Prognostic models in medicine are usually been built using simple decision rules, proportional hazards models, or Markov models. Dynamic Bayesian networks (DBNs) offer an approach that allows for the incorporation of the causal and temporal nature of medical domain knowledge as elicited from domain experts, thereby allowing for detailed prognostic predictions. The aim of this paper is to describe the considerations that must be taken into account when constructing a DBN for complex medical domains and to demonstrate their usefulness in practice. To this end, we focus on the construction of a DBN for prognosis of carcinoid patients, compare performance with that of a proportional hazards model, and describe predictions for three individual patients. We show that the DBN can make detailed predictions, about not only patient survival, but also other variables of interest, such as disease progression, the effect of treatment, and the development of complications. Strengths and limitations of our approach are discussed and compared with those offered by traditional methods.

Selecting treatment strategies with dynamic limited-memory influence diagrams

OBJECTIVE

The development of dynamic limited-memory influence diagrams as a framework for representing factorized infinite-horizon partially observable Markov decision processes (POMDPs), the introduction of algorithms for their (approximate) solution, and the application to a dynamic decision problem in clinical oncology.

MATERIALS AND METHODS

A dynamic limited-memory influence diagram for high-grade carcinoid tumor pathophysiology was developed in collaboration with an expert physician. Three algorithms, known as single policy updating, single rule updating, and simulated annealing have been examined for approximating the optimal treatment strategy from a space of 10(19) possible strategies.

RESULTS

Single policy updating proved intractable for finding a treatment strategy for carcinoid tumors. Single rule updating and simulated annealing both found the treatment strategy that is applied by physicians in practice.

CONCLUSIONS

Dynamic limited-memory influence diagrams are a suitable framework for the representation of factorized infinite-horizon POMDPs, and the developed algorithms find acceptable solutions under the assumption of limited memory about past observations. The framework allows for finding reasonable treatment strategies for complex dynamic decision problems in medicine.

Predicting carcinoid heart disease with the noisy-threshold classifier

OBJECTIVE

To predict the development of carcinoid heart disease (CHD), which is a life-threatening complication of certain neuroendocrine tumors. To this end, a novel type of Bayesian classifier, known as the noisy-threshold classifier, is applied.

MATERIALS AND METHODS

Fifty-four cases of patients that suffered from a low-grade midgut carcinoid tumor, of which 22 patients developed CHD, were obtained from the Netherlands Cancer Institute (NKI). Eleven attributes that are known at admission have been used to classify whether the patient develops CHD. Classification accuracy and area under the receiver operating characteristics (ROC) curve of the noisy-threshold classifier are compared with those of the naive-Bayes classifier, logistic regression, the decision-tree learning algorithm C4.5, and a decision rule, as formulated by an expert physician.

RESULTS

The noisy-threshold classifier showed the best classification accuracy of 72% correctly classified cases, although differences were significant only for logistic regression and C4.5. An area under the ROC curve of 0.66 was attained for the noisy-threshold classifier, and equaled that of the physician's decision-rule.

CONCLUSIONS

The noisy-threshold classifier performed favorably to other state-of-the-art classification algorithms, and equally well as a decision-rule that was formulated by the physician. Furthermore, the semantics of the noisy-threshold classifier make it a useful machine learning technique in domains where multiple causes influence a common effect.