Identifying natural images from human brain activity

Automated classification of fMRI data employing trial-based imagery tasks

Automated interpretation and classification of functional MRI (fMRI) data is an emerging research field that enables the characterization of underlying cognitive processes with minimal human intervention. In this work, we present a method for the automated classification of human thoughts reflected on a trial-based paradigm using fMRI with a significantly shortened data acquisition time (less than one minute). Based on our preliminary experience with various cognitive imagery tasks, six characteristic thoughts were chosen as target tasks for the present work: right-hand motor imagery, left-hand motor imagery, right foot motor imagery, mental calculation, internal speech/word generation, and visual imagery. These six tasks were performed by five healthy volunteers and functional images were obtained using a T(*)(2)-weighted echo planar imaging (EPI) sequence. Feature vectors from activation maps, necessary for the classification of neural activity, were automatically extracted from the regions that were consistently and exclusively activated for a given task during the training process. Extracted feature vectors were classified using the support vector machine (SVM) algorithm. Parameter optimization, using a k-fold cross validation scheme, allowed the successful recognition of the six different categories of administered thought tasks with an accuracy of 74.5% (mean)+/-14.3% (standard deviation) across all five subjects. Our proposed study for the automated classification of fMRI data may be utilized in further investigations to monitor/identify human thought processes and their potential link to hardware/computer control.

Multivariate patterns in object-selective cortex dissociate perceptual and physical shape similarity

Prior research has identified the lateral occipital complex (LOC) as a critical cortical region for the representation of object shape in humans. However, little is known about the nature of the representations contained in the LOC and their relationship to the perceptual experience of shape. We used human functional MRI to measure the physical, behavioral, and neural similarity between pairs of novel shapes to ask whether the representations of shape contained in subregions of the LOC more closely reflect the physical stimuli themselves, or the perceptual experience of those stimuli. Perceptual similarity measures for each pair of shapes were obtained from a psychophysical same-different task; physical similarity measures were based on stimulus parameters; and neural similarity measures were obtained from multivoxel pattern analysis methods applied to anterior LOC (pFs) and posterior LOC (LO). We found that the pattern of pairwise shape similarities in LO most closely matched physical shape similarities, whereas shape similarities in pFs most closely matched perceptual shape similarities. Further, shape representations were similar across participants in LO but highly variable across participants in pFs. Together, these findings indicate that activation patterns in subregions of object-selective cortex encode objects according to a hierarchy, with stimulus-based representations in posterior regions and subjective and observer-specific representations in anterior regions.

As early as 1031 a.d., the Arab scholar Ibn al-Haytham suggested that visual experience was not veridical, but inherently subjective. During the last few decades, this observation has given rise to one of the core questions in visual neuroscience: how does the subjective experience of visual stimuli relate to their neural representations in the brain? It is well-known that visual shape is represented in a brain region called lateral occipital complex (LOC). However, do these representations reflect physical or perceptual stimulus characteristics? We presented observers with a set of complex visual stimuli and obtained three measures of similarity for these stimuli: a physical similarity measure based on stimulus parameters; a behavioral similarity measure based on discrimination performance; and finally a neural similarity measure based on multivariate pattern analyses in LOC. We found that in anterior LOC, neural stimulus similarities correlated with subjective perceptual similarities, but not with physical stimulus similarities; the reverse was true in posterior LOC. In addition, neural similarities were consistent across participants in posterior LOC, but highly variable across participants in anterior LOC. Together these findings suggest a two-part answer to the question of how cortical object representations relate to subjective experience: anterior regions appear to contain subjective, individually variable shape representations, whereas posterior regions contain stimulus-based shape representations.

How does the subjective experience of visual shapes relate to the neural representations of these shapes in the brain? Using psychophysics, functional MRI, and multivariate pattern analysis methods, this study shows that activation patterns in anterior, shape-selective brain regions reflect perceptual shape similarities, whereas patterns in posterior regions reflect physical similarities.

Science of the conscious mind

Human beings have direct access to their own mental states, but can only indirectly observe cosmic radiation and enzyme kinetics. Why then can we measure the temperature of far away galaxies and the activation constant of kinases to the third digit, yet we only gauge our happiness on a scale from 1 to 7? Here we propose a radical research paradigm shift to embrace the subjective conscious mind into the realm of objective empirical science. Key steps are the axiomatic acceptance of first-person experiences as scientific observables; the definition of a quantitative, reliable metric system based on natural language; and the careful distinction of subjective mental states (e.g., interpretation and intent) from physically measurable sensory and motor behaviors (input and output). Using this approach, we propose a series of reproducible experiments that may help define a still largely unexplored branch of science. We speculate that the development of this new discipline will be initially parallel to, and eventually converging with, neurobiology and physics.

Representational similarity analysis - connecting the branches of systems neuroscience

A fundamental challenge for systems neuroscience is to quantitatively relate its three major branches of research: brain-activity measurement, behavioral measurement, and computational modeling. Using measured brain-activity patterns to evaluate computational network models is complicated by the need to define the correspondency between the units of the model and the channels of the brain-activity data, e.g., single-cell recordings or voxels from functional magnetic resonance imaging (fMRI). Similar correspondency problems complicate relating activity patterns between different modalities of brain-activity measurement (e.g., fMRI and invasive or scalp electrophysiology), and between subjects and species. In order to bridge these divides, we suggest abstracting from the activity patterns themselves and computing representational dissimilarity matrices (RDMs), which characterize the information carried by a given representation in a brain or model. Building on a rich psychological and mathematical literature on similarity analysis, we propose a new experimental and data-analytical framework called representational similarity analysis (RSA), in which multi-channel measures of neural activity are quantitatively related to each other and to computational theory and behavior by comparing RDMs. We demonstrate RSA by relating representations of visual objects as measured with fMRI in early visual cortex and the fusiform face area to computational models spanning a wide range of complexities. The RDMs are simultaneously related via second-level application of multidimensional scaling and tested using randomization and bootstrap techniques. We discuss the broad potential of RSA, including novel approaches to experimental design, and argue that these ideas, which have deep roots in psychology and neuroscience, will allow the integrated quantitative analysis of data from all three branches, thus contributing to a more unified systems neuroscience.

Machine learning classifiers and fMRI: a tutorial overview

Interpreting brain image experiments requires analysis of complex, multivariate data. In recent years, one analysis approach that has grown in popularity is the use of machine learning algorithms to train classifiers to decode stimuli, mental states, behaviours and other variables of interest from fMRI data and thereby show the data contain information about them. In this tutorial overview we review some of the key choices faced in using this approach as well as how to derive statistically significant results, illustrating each point from a case study. Furthermore, we show how, in addition to answering the question of 'is there information about a variable of interest' (pattern discrimination), classifiers can be used to tackle other classes of question, namely 'where is the information' (pattern localization) and 'how is that information encoded' (pattern characterization).

Executed and observed movements have different distributed representations in human aIPS

How similar are the representations of executed and observed hand movements in the human brain? We used functional magnetic resonance imaging (fMRI) and multivariate pattern classification analysis to compare spatial distributions of cortical activity in response to several observed and executed movements. Subjects played the rock-paper-scissors game against a videotaped opponent, freely choosing their movement on each trial and observing the opponent's hand movement after a short delay. The identities of executed movements were correctly classified from fMRI responses in several areas of motor cortex, observed movements were classified from responses in visual cortex, and both observed and executed movements were classified from responses in either left or right anterior intraparietal sulcus (aIPS). We interpret above chance classification as evidence for reproducible, distributed patterns of cortical activity that were unique for execution and/or observation of each movement. Responses in aIPS enabled accurate classification of movement identity within each modality (visual or motor), but did not enable accurate classification across modalities (i.e., decoding observed movements from a classifier trained on executed movements and vice versa). These results support theories regarding the central role of aIPS in the perception and execution of movements. However, the spatial pattern of activity for a particular observed movement was distinctly different from that for the same movement when executed, suggesting that observed and executed movements are mostly represented by distinctly different subpopulations of neurons in aIPS.

Neuroimaging of cognition: past, present, and future

Neuroimaging, particularly that based upon functional magnetic resonance (fMRI), has become a dominant tool in cognitive neuroscience. This review provides a personal and selective perspective on its past, present, and future. Two trends currently characterize the field that broadly reflect a pursuit of "where"- and "how"-type questions. The latter addresses basic mechanisms related to the expression of task-induced neural activity and is likely to be an increasingly important theme in the future. This trend entails an enhanced symbiosis among investigators pursuing similar questions in fields such as computational and theoretical neuroscience as well as through the detailed analysis of microcircuitry.

Predicting human brain activity associated with the meanings of nouns

The question of how the human brain represents conceptual knowledge has been debated in many scientific fields. Brain imaging studies have shown that different spatial patterns of neural activation are associated with thinking about different semantic categories of pictures and words (for example, tools, buildings, and animals). We present a computational model that predicts the functional magnetic resonance imaging (fMRI) neural activation associated with words for which fMRI data are not yet available. This model is trained with a combination of data from a trillion-word text corpus and observed fMRI data associated with viewing several dozen concrete nouns. Once trained, the model predicts fMRI activation for thousands of other concrete nouns in the text corpus, with highly significant accuracies over the 60 nouns for which we currently have fMRI data.

Au naturel

Although adaptation is a ubiquitous property of neurons in the early visual pathway, the functional consequences in the natural visual environment are unknown. In this issue of Neuron, Mante et al. show, through a comprehensive set of in vivo experiments in the visual thalamus, that the basic functional mechanisms of adaptation that have been well studied with artificial probes capture the neuronal response in the natural environment and are predictable from properties of the visual scene that may be represented by local neural ensembles.

In the Spotlight: Neuroengineering

This article reviews current researches in the field of neuroengineering. Special focus is given to neural prosthesis, neuroprosthetic control and brain-computer interfaces (BCIs) for anthropomorphic and sensory prosthetic control.