Decoding visual object categories from temporal correlations of ECoG signals

Evidence for Multiscale Multiplexed Representation of Visual Features in EEG

Distinct neural processes such as sensory and memory processes are often encoded over distinct timescales of neural activations. Animal studies have shown that this multiscale coding strategy is also implemented for individual components of a single process, such as individual features of a multifeature stimulus in sensory coding. However, the generalizability of this encoding strategy to the human brain has remained unclear. We asked if individual features of visual stimuli were encoded over distinct timescales. We applied a multiscale time-resolved decoding method to electroencephalography (EEG) collected from human subjects presented with grating visual stimuli to estimate the timescale of individual stimulus features. We observed that the orientation and color of the stimuli were encoded in shorter timescales, whereas spatial frequency and the contrast of the same stimuli were encoded in longer timescales. The stimulus features appeared in temporally overlapping windows along the trial supporting a multiplexed coding strategy. These results provide evidence for a multiplexed, multiscale coding strategy in the human visual system.

When the Whole Is Less Than the Sum of Its Parts: Maximum Object Category Information and Behavioral Prediction in Multiscale Activation Patterns

Neural codes are reflected in complex neural activation patterns. Conventional electroencephalography (EEG) decoding analyses summarize activations by averaging/down-sampling signals within the analysis window. This diminishes informative fine-grained patterns. While previous studies have proposed distinct statistical features capable of capturing variability-dependent neural codes, it has been suggested that the brain could use a combination of encoding protocols not reflected in any one mathematical feature alone. To check, we combined 30 features using state-of-the-art supervised and unsupervised feature selection procedures (n = 17). Across three datasets, we compared decoding of visual object category between these 17 sets of combined features, and between combined and individual features. Object category could be robustly decoded using the combined features from all of the 17 algorithms. However, the combination of features, which were equalized in dimension to the individual features, were outperformed across most of the time points by the multiscale feature of Wavelet coefficients. Moreover, the Wavelet coefficients also explained the behavioral performance more accurately than the combined features. These results suggest that a single but multiscale encoding protocol may capture the EEG neural codes better than any combination of protocols. Our findings put new constraints on the models of neural information encoding in EEG.

An engram of intentionally forgotten information

Successful forgetting of unwanted memories is crucial for goal-directed behavior and mental wellbeing. While memory retention strengthens memory traces, it is unclear what happens to memory traces of events that are actively forgotten. Using intracranial EEG recordings from lateral temporal cortex, we find that memory traces for actively forgotten information are partially preserved and exhibit unique neural signatures. Memory traces of successfully remembered items show stronger encoding-retrieval similarity in gamma frequency patterns. By contrast, encoding-retrieval similarity of item-specific memory traces of actively forgotten items depend on activity at alpha/beta frequencies commonly associated with functional inhibition. Additional analyses revealed selective modification of item-specific patterns of connectivity and top-down information flow from dorsolateral prefrontal cortex to lateral temporal cortex in memory traces of intentionally forgotten items. These results suggest that intentional forgetting relies more on inhibitory top-down connections than intentional remembering, resulting in inhibitory memory traces with unique neural signatures and representational formats.

Quantum-inspired canonical correlation analysis for exponentially large dimensional data

Canonical correlation analysis (CCA) serves to identify statistical dependencies between pairs of multivariate data. However, its application to high-dimensional data is limited due to considerable computational complexity. As an alternative to the conventional CCA approach that requires polynomial computational time, we propose an algorithm that approximates CCA using quantum-inspired computations with computational time proportional to the logarithm of the input dimensionality. The computational efficiency and performance of the proposed quantum-inspired CCA (qiCCA) algorithm are experimentally evaluated on synthetic and real datasets. Furthermore, the fast computation provided by qiCCA allows directly applying CCA even after nonlinearly mapping raw input data into high-dimensional spaces. The conducted experiments demonstrate that, as a result of mapping raw input data into the high-dimensional spaces with the use of second-order monomials, qiCCA extracts more correlations compared with the linear CCA and achieves comparable performance with state-of-the-art nonlinear variants of CCA on several datasets. These results confirm the appropriateness of the proposed qiCCA and the high potential of quantum-inspired computations in analyzing high-dimensional data.

Electrocorticogram (ECoG) Is Highly Informative in Primate Visual Cortex

Neural signals recorded at different scales contain information about environment and behavior and have been used to control Brain Machine Interfaces with varying degrees of success. However, a direct comparison of their efficacy has not been possible due to different recording setups, tasks, species, etc. To address this, we implanted customized arrays having both microelectrodes and electrocorticogram (ECoG) electrodes in the primary visual cortex of 2 female macaque monkeys, and also recorded electroencephalogram (EEG), while they viewed a variety of naturalistic images and parametric gratings. Surprisingly, ECoG had higher information and decodability than all other signals. Combining a few ECoG electrodes allowed more accurate decoding than combining a much larger number of microelectrodes. Control analyses showed that higher decoding accuracy of ECoG compared with local field potential was not because of differences in low-level visual features captured by them but instead because of larger spatial summation of the ECoG. Information was high in the 30-80 Hz range and at lower frequencies. Information in different frequencies and scales was nonredundant. These results have strong implications for Brain Machine Interface applications and for study of population representation of visual stimuli.SIGNIFICANCE STATEMENT Electrophysiological signals captured across scales by different recording electrodes are regularly used for Brain Machine Interfaces, but the information content varies due to electrode size and location. A systematic comparison of their efficiency for Brain Machine Interfaces is important but technically challenging. Here, we recorded simultaneous signals across four scales: spikes, local field potential, electrocorticogram (ECoG), and EEG, and compared their information and decoding accuracy for a large variety of naturalistic stimuli. We found that ECoGs were highly informative and outperformed other signals in information content and decoding accuracy.

Heterogeneous Redistribution of Facial Subcategory Information Within and Outside the Face-Selective Domain in Primate Inferior Temporal Cortex

The inferior temporal cortex (ITC) contains neurons selective to multiple levels of visual categories. However, the mechanisms by which these neurons collectively construct hierarchical category percepts remain unclear. By comparing decoding accuracy with simultaneously acquired electrocorticogram (ECoG), local field potentials (LFPs), and multi-unit activity in the macaque ITC, we show that low-frequency LFPs/ECoG in the early evoked visual response phase contain sufficient coarse category (e.g., face) information, which is homogeneous and enhanced by spatial summation of up to several millimeters. Late-induced high-frequency LFPs additionally carry spike-coupled finer category (e.g., species, view, and identity of the face) information, which is heterogeneous and reduced by spatial summation. Face-encoding neural activity forms a cluster in similar cortical locations regardless of whether it is defined by early evoked low-frequency signals or late-induced high-gamma signals. By contrast, facial subcategory-encoding activity is distributed, not confined to the face cluster, and dynamically increases its heterogeneity from the early evoked to late-induced phases. These findings support a view that, in contrast to the homogeneous and static coarse category-encoding neural cluster, finer category-encoding clusters are heterogeneously distributed even outside their parent category cluster and dynamically increase heterogeneity along with the local cortical processing in the ITC.

Decoding Speech With Integrated Hybrid Signals Recorded From the Human Ventral Motor Cortex

Restoration of speech communication for locked-in patients by means of brain computer interfaces (BCIs) is currently an important area of active research. Among the neural signals obtained from intracranial recordings, single/multi-unit activity (SUA/MUA), local field potential (LFP), and electrocorticography (ECoG) are good candidates for an input signal for BCIs. However, the question of which signal or which combination of the three signal modalities is best suited for decoding speech production remains unverified. In order to record SUA, LFP, and ECoG simultaneously from a highly localized area of human ventral sensorimotor cortex (vSMC), we fabricated an electrode the size of which was 7 by 13 mm containing sparsely arranged microneedle and conventional macro contacts. We determined which signal modality is the most capable of decoding speech production, and tested if the combination of these signals could improve the decoding accuracy of spoken phonemes. Feature vectors were constructed from spike frequency obtained from SUAs and event-related spectral perturbation derived from ECoG and LFP signals, then input to the decoder. The results showed that the decoding accuracy for five spoken vowels was highest when features from multiple signals were combined and optimized for each subject, and reached 59% when averaged across all six subjects. This result suggests that multi-scale signals convey complementary information for speech articulation. The current study demonstrated that simultaneous recording of multi-scale neuronal activities could raise decoding accuracy even though the recording area is limited to a small portion of cortex, which is advantageous for future implementation of speech-assisting BCIs.

Real-time detection and discrimination of visual perception using electrocorticographic signals

OBJECTIVE

Several neuroimaging studies have demonstrated that the ventral temporal cortex contains specialized regions that process visual stimuli. This study investigated the spatial and temporal dynamics of electrocorticographic (ECoG) responses to different types and colors of visual stimulation that were presented to four human participants, and demonstrated a real-time decoder that detects and discriminates responses to untrained natural images.

APPROACH

ECoG signals from the participants were recorded while they were shown colored and greyscale versions of seven types of visual stimuli (images of faces, objects, bodies, line drawings, digits, and kanji and hiragana characters), resulting in 14 classes for discrimination (experiment I). Additionally, a real-time system asynchronously classified ECoG responses to faces, kanji and black screens presented via a monitor (experiment II), or to natural scenes (i.e. the face of an experimenter, natural images of faces and kanji, and a mirror) (experiment III). Outcome measures in all experiments included the discrimination performance across types based on broadband γ activity.

MAIN RESULTS

Experiment I demonstrated an offline classification accuracy of 72.9% when discriminating among the seven types (without color separation). Further discrimination of grey versus colored images reached an accuracy of 67.1%. Discriminating all colors and types (14 classes) yielded an accuracy of 52.1%. In experiment II and III, the real-time decoder correctly detected 73.7% responses to face, kanji and black computer stimuli and 74.8% responses to presented natural scenes.

SIGNIFICANCE

Seven different types and their color information (either grey or color) could be detected and discriminated using broadband γ activity. Discrimination performance maximized for combined spatial-temporal information. The discrimination of stimulus color information provided the first ECoG-based evidence for color-related population-level cortical broadband γ responses in humans. Stimulus categories can be detected by their ECoG responses in real time within 500 ms with respect to stimulus onset.

Associative-memory representations emerge as shared spatial patterns of theta activity spanning the primate temporal cortex

Highly localized neuronal spikes in primate temporal cortex can encode associative memory; however, whether memory formation involves area-wide reorganization of ensemble activity, which often accompanies rhythmicity, or just local microcircuit-level plasticity, remains elusive. Using high-density electrocorticography, we capture local-field potentials spanning the monkey temporal lobes, and show that the visual pair-association (PA) memory is encoded in spatial patterns of theta activity in areas TE, 36, and, partially, in the parahippocampal cortex, but not in the entorhinal cortex. The theta patterns elicited by learned paired associates are distinct between pairs, but similar within pairs. This pattern similarity, emerging through novel PA learning, allows a machine-learning decoder trained on theta patterns elicited by a particular visual item to correctly predict the identity of those elicited by its paired associate. Our results suggest that the formation and sharing of widespread cortical theta patterns via learning-induced reorganization are involved in the mechanisms of associative memory representation.

Representing object categories by connections: Evidence from a mutivariate connectivity pattern classification approach

The representation of object categories is a classical question in cognitive neuroscience and compelling evidence has identified specific brain regions showing preferential activation to categories of evolutionary significance. However, the potential contributions to category processing by tuning the connectivity patterns are largely unknown. Adopting a continuous multicategory paradigm, we obtained whole-brain functional connectivity (FC) patterns of each of four categories (faces, scenes, animals and tools) in healthy human adults and applied multivariate connectivity pattern classification analyses. We found that the whole-brain FC patterns made high-accuracy predictions of which category was being viewed. The decoding was successful even after the contributions of regions showing classical category-selective activations were excluded. We further identified the discriminative network for each category, which span way beyond the classical category-selective regions. Together, these results reveal novel mechanisms about how categorical information is represented in large-scale FC patterns, with general implications for the interactive nature of distributed brain areas underlying high-level cognition. Hum Brain Mapp 37:3685-3697, 2016. © 2016 Wiley Periodicals, Inc.

Categorical discrimination of human body parts by magnetoencephalography

Humans recognize body parts in categories. Previous studies have shown that responses in the fusiform body area (FBA) and extrastriate body area (EBA) are evoked by the perception of the human body, when presented either as whole or as isolated parts. These responses occur approximately 190 ms after body images are visualized. The extent to which body-sensitive responses show specificity for different body part categories remains to be largely clarified. We used a decoding method to quantify neural responses associated with the perception of different categories of body parts. Nine subjects underwent measurements of their brain activities by magnetoencephalography (MEG) while viewing 14 images of feet, hands, mouths, and objects. We decoded categories of the presented images from the MEG signals using a support vector machine (SVM) and calculated their accuracy by 10-fold cross-validation. For each subject, a response that appeared to be a body-sensitive response was observed and the MEG signals corresponding to the three types of body categories were classified based on the signals in the occipitotemporal cortex. The accuracy in decoding body-part categories (with a peak at approximately 48%) was above chance (33.3%) and significantly higher than that for random categories. According to the time course and location, the responses are suggested to be body-sensitive and to include information regarding the body-part category. Finally, this non-invasive method can decode category information of a visual object with high temporal and spatial resolution and this result may have a significant impact in the field of brain-machine interface research.

Shifts of Gamma Phase across Primary Visual Cortical Sites Reflect Dynamic Stimulus-Modulated Information Transfer

Distributed neural processing likely entails the capability of networks to reconfigure dynamically the directionality and strength of their functional connections. Yet, the neural mechanisms that may allow such dynamic routing of the information flow are not yet fully understood. We investigated the role of gamma band (50-80 Hz) oscillations in transient modulations of communication among neural populations by using measures of direction-specific causal information transfer. We found that the local phase of gamma-band rhythmic activity exerted a stimulus-modulated and spatially-asymmetric directed effect on the firing rate of spatially separated populations within the primary visual cortex. The relationships between gamma phases at different sites (phase shifts) could be described as a stimulus-modulated gamma-band wave propagating along the spatial directions with the largest information transfer. We observed transient stimulus-related changes in the spatial configuration of phases (compatible with changes in direction of gamma wave propagation) accompanied by a relative increase of the amount of information flowing along the instantaneous direction of the gamma wave. These effects were specific to the gamma-band and suggest that the time-varying relationships between gamma phases at different locations mark, and possibly causally mediate, the dynamic reconfiguration of functional connections.

Phase-amplitude coupling supports phase coding in human ECoG

Prior studies have shown that high-frequency activity (HFA) is modulated by the phase of low-frequency activity. This phenomenon of phase-amplitude coupling (PAC) is often interpreted as reflecting phase coding of neural representations, although evidence for this link is still lacking in humans. Here, we show that PAC indeed supports phase-dependent stimulus representations for categories. Six patients with medication-resistant epilepsy viewed images of faces, tools, houses, and scenes during simultaneous acquisition of intracranial recordings. Analyzing 167 electrodes, we observed PAC at 43% of electrodes. Further inspection of PAC revealed that category specific HFA modulations occurred at different phases and frequencies of the underlying low-frequency rhythm, permitting decoding of categorical information using the phase at which HFA events occurred. These results provide evidence for categorical phase-coded neural representations and are the first to show that PAC coincides with phase-dependent coding in the human brain.

Neural population coding: combining insights from microscopic and mass signals

Behavior relies on the distributed and coordinated activity of neural populations. Population activity can be measured using multi-neuron recordings and neuroimaging. Neural recordings reveal how the heterogeneity, sparseness, timing, and correlation of population activity shape information processing in local networks, whereas neuroimaging shows how long-range coupling and brain states impact on local activity and perception. To obtain an integrated perspective on neural information processing we need to combine knowledge from both levels of investigation. We review recent progress of how neural recordings, neuroimaging, and computational approaches begin to elucidate how interactions between local neural population activity and large-scale dynamics shape the structure and coding capacity of local information representations, make them state-dependent, and control distributed populations that collectively shape behavior.

Sensitivity to object impossibility in the human visual cortex: evidence from functional connectivity

Processing spatial configuration is a fundamental requirement for object recognition. Using fMRI, the neural basis underlying this ability was examined while human participants viewed possible and visually similar, but spatially impossible, objects presented for either long or short exposure duration. Response profiles in object-selective cortical regions exhibited sensitivity to object possibility, but only for the long exposure duration. Contrary, functional connectivity, indexed by the pairwise correlations between activation profiles across ROIs, revealed sensitivity to possibility, evident in enhanced correlations for impossible compared with possible objects. Such sensitivity was found even following a brief exposure duration, which allowed only minimal awareness of possibility. Importantly, this sensitivity was correlated with participants' general spatial ability as assessed by an independent neuropsychological test. These results suggest that the visual system is highly susceptible to objects' 3-D structural information even with minimal perceptual awareness. Such sensitivity is captured at the level of functional connectivity between object-selective regions, rather than the absolute level of within-region activity, implicating the role of interregional synchronization in the representation of objects' 3-D structure.

More than spikes: common oscillatory mechanisms for content specific neural representations during perception and memory

Although previous research into the mechanisms underlying sensory and episodic representations has primarily focused on changes in neural firing rate, more recent evidence suggests that neural oscillations also contribute to these representations. Here, we argue that multiplexed oscillatory power and phase contribute to neural representations at the mesoscopic scale, complementary to neuronal firing. Reviewing recent studies which used oscillatory activity to decipher content-specific neural representations, we identify oscillatory mechanisms common to both sensory and episodic memory representations and incorporate these into a model of episodic encoding and retrieval. This model advances the idea that oscillations provide a reference frame for phase-coded item representations during memory encoding and that shifts in oscillatory frequency and phase coordinate ensemble activity during memory retrieval.