Frequency-dependent spatiotemporal profiles of visual responses recorded with subdural ECoG electrodes in awake monkeys: Differences between high- and low-frequency activity

Learning neural decoders without labels using multiple data streams

OBJECTIVE

Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding.

APPROACH

We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Specifically, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models.

MAIN RESULTS

We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Significance: We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

Behavioral and Neural Variability of Naturalistic Arm Movements

Motor behaviors are central to many functions and dysfunctions of the brain, and understanding their neural basis has consequently been a major focus in neuroscience. However, most studies of motor behaviors have been restricted to artificial, repetitive paradigms, far removed from natural movements performed "in the wild." Here, we leveraged recent advances in machine learning and computer vision to analyze intracranial recordings from 12 human subjects during thousands of spontaneous, unstructured arm reach movements, observed over several days for each subject. These naturalistic movements elicited cortical spectral power patterns consistent with findings from controlled paradigms, but with considerable neural variability across subjects and events. We modeled interevent variability using 10 behavioral and environmental features; the most important features explaining this variability were reach angle and day of recording. Our work is among the first studies connecting behavioral and neural variability across cortex in humans during unstructured movements and contributes to our understanding of long-term naturalistic behavior.

Generalized neural decoders for transfer learning across participants and recording modalities

OBJECTIVE

Advances in neural decoding have enabled brain-computer interfaces to perform increasingly complex and clinically-relevant tasks. However, such decoders are often tailored to specific participants, days, and recording sites, limiting their practical long-term usage. Therefore, a fundamental challenge is to develop neural decoders that can robustly train on pooled, multi-participant data and generalize to new participants.

APPROACH

We introduce a new decoder, HTNet, which uses a convolutional neural network with two innovations: (1) a Hilbert transform that computes spectral power at data-driven frequencies and (2) a layer that projects electrode-level data onto predefined brain regions. The projection layer critically enables applications with intracranial electrocorticography (ECoG), where electrode locations are not standardized and vary widely across participants. We trained HTNet to decode arm movements using pooled ECoG data from 11 of 12 participants and tested performance on unseen ECoG or electroencephalography (EEG) participants; these pretrained models were also subsequently fine-tuned to each test participant.

MAIN RESULTS

HTNet outperformed state-of-the-art decoders when tested on unseen participants, even when a different recording modality was used. By fine-tuning these generalized HTNet decoders, we achieved performance approaching the best tailored decoders with as few as 50 ECoG or 20 EEG events. We were also able to interpret HTNet's trained weights and demonstrate its ability to extract physiologically-relevant features.

SIGNIFICANCE

By generalizing to new participants and recording modalities, robustly handling variations in electrode placement, and allowing participant-specific fine-tuning with minimal data, HTNet is applicable across a broader range of neural decoding applications compared to current state-of-the-art decoders.

Hierarchical cortical gradients in somatosensory processing

Sensory information is processed in the visual cortex in distinct streams of different anatomical and functional properties. A comparable organizational principle has also been proposed to underlie auditory processing. This raises the question of whether a similar principle characterize the somatosensory domain. One property of a cortical stream is a hierarchical organization of the neuronal response properties along an anatomically distinct pathway. Indeed, several hierarchies between specific somatosensory cortical regions have been identified, primarily using electrophysiology, in non-human primates. However, it has been unclear how these local hierarchies are organized throughout the cortex. Here we used phase-encoded bilateral full-body light touch stimulation in healthy humans under functional MRI to study the large-scale organization of hierarchies in the somatosensory domain. We quantified two measures of hierarchy of BOLD responses, selectivity and laterality. We measured how selectivity and laterality change as we move away from the central sulcus within four gross anatomically-distinct regions. We found that both selectivity and laterality decrease in three directions: parietal, posteriorly along the parietal lobe, frontal, anteriorly along the frontal lobe and medial, inferiorly-anteriorly along the medial wall. The decline of selectivity and laterality along these directions provides evidence for hierarchical gradients. In view of the anatomical segregation of these three directions, the multiplicity of body representations in each region and the hierarchical gradients in our findings, we propose that as in the visual and auditory domains, these directions are streams of somatosensory information processing.

A multiple-response frequency-tagging paradigm measures graded changes in consciousness during perceptual filling-in

Perceptual filling-in (PFI) occurs when a physically present visual target disappears from conscious perception, with its location filled-in by the surrounding visual background. These perceptual changes are complete, near instantaneous, and can occur for multiple separate locations simultaneously. Here, we show that contrasting neural activity during the presence or absence of multi-target PFI can complement other findings from multistable phenomena to reveal the neural correlates of consciousness (NCC). We presented four peripheral targets over a background dynamically updating at 20 Hz. While participants reported on target disappearances/reappearances via button press/release, we tracked neural activity entrained by the background during PFI using steady-state visually evoked potentials (SSVEPs) recorded in the electroencephalogram. We found background SSVEPs closely correlated with subjective report, and increased with an increasing amount of PFI. Unexpectedly, we found that as the number of filled-in targets increased, the duration of target disappearances also increased, suggesting that facilitatory interactions exist between targets in separate visual quadrants. We also found distinct spatiotemporal correlates for the background SSVEP harmonics. Prior to genuine PFI, the response at the second harmonic (40 Hz) increased before the first (20 Hz), which we tentatively link to an attentional effect, while no such difference between harmonics was observed for physically removed stimuli. These results demonstrate that PFI can be used to study multi-object perceptual suppression when frequency-tagging the background of a visual display, and because there are distinct neural correlates for endogenously and exogenously induced changes in consciousness, that it is ideally suited to study the NCC.

Visual epidural field potentials possess high functional specificity in single trials

Recordings of epidural field potentials (EFPs) allow neuronal activity to be acquired over a large region of cortical tissue with minimal invasiveness. Because electrodes are placed on top of the dura and do not enter the neuronal tissue, EFPs offer intriguing options for both clinical and basic science research. On the other hand, EFPs represent the integrated activity of larger neuronal populations and possess a higher trial-by-trial variability and a reduced signal-to-noise ratio due the additional barrier of the dura. It is thus unclear whether and to what extent EFPs have sufficient spatial selectivity to allow for conclusions about the underlying functional cortical architecture, and whether single EFP trials provide enough information on the short timescales relevant for many clinical and basic neuroscience purposes. We used the high spatial resolution of primary visual cortex to address these issues and investigated the extent to which very short EFP traces allow reliable decoding of spatial information. We briefly presented different visual objects at one of nine closely adjacent locations and recorded neuronal activity with a high-density epidural multielectrode array in three macaque monkeys. With the use of receiver operating characteristics (ROC) to identify the most informative data, machine-learning algorithms provided close-to-perfect classification rates for all 27 stimulus conditions. A binary classifier applying a simple max function on ROC-selected data further showed that single trials might be classified with 100% performance even without advanced offline classifiers. Thus, although highly variable, EFPs constitute an extremely valuable source of information and offer new perspectives for minimally invasive recording of large-scale networks.NEW & NOTEWORTHY Epidural field potential (EFP) recordings provide a minimally invasive approach to investigate large-scale neural networks, but little is known about whether they possess the required specificity for basic and clinical neuroscience. By making use of the spatial selectivity of primary visual cortex, we show that single-trial information can be decoded with close-to-perfect performance, even without using advanced classifiers and based on very few data. This labels EFPs as a highly attractive and widely usable signal.

Correlation Structure in Micro-ECoG Recordings is Described by Spatially Coherent Components

Electrocorticography (ECoG) is becoming more prevalent due to improvements in fabrication and recording technology as well as its ease of implantation compared to intracortical electrophysiology, larger cortical coverage, and potential advantages for use in long term chronic implantation. Given the flexibility in the design of ECoG grids, which is only increasing, it remains an open question what geometry of the electrodes is optimal for an application. Conductive polymer, PEDOT:PSS, coated microelectrodes have an advantage that they can be made very small without losing low impedance. This makes them suitable for evaluating the required granularity of ECoG recording in humans and experimental animals. We used two-dimensional (2D) micro-ECoG grids to record intra-operatively in humans and during acute implantations in mouse with separation distance between neighboring electrodes (i.e., pitch) of 0.4 mm and 0.2/0.25 mm respectively. To assess the spatial properties of the signals, we used the average correlation between electrodes as a function of the pitch. In agreement with prior studies, we find a strong frequency dependence in the spatial scale of correlation. By applying independent component analysis (ICA), we find that the spatial pattern of correlation is largely due to contributions from multiple spatially extended, time-locked sources present at any given time. Our analysis indicates the presence of spatially structured activity down to the sub-millimeter spatial scale in ECoG despite the effects of volume conduction, justifying the use of dense micro-ECoG grids.

Electrocorticography (ECoG) is a type of electrophysiological monitoring that uses electrodes placed directly on the exposed surface of the brain. ECoG is a promising technique for studying the brain, and EcoG signals can be used to control brain-computer interfaces. Advances have made it possible to record simultaneously with an increasing number of smaller, and more closely spaced electrodes. However, a property of electrical recording from outside the brain is that common signals appear on different electrodes at different locations, and this affects decisions about how to best distribute a limited number of electrodes to maximize the information that can be gathered. Large spacing of electrodes around one centimeter apart on the brain’s surface has proven useful for clinical and research use, but how much benefit there is to recording from more locations in a smaller area remains to be answered. We found that we can explain the commonality between the different locations as the combination of different patterns of brain activity that are present at multiple electrode locations, and that signals recorded from very closely spaced electrodes, around a millimeter or less apart, are able to identify patterns that are at this small scale.

Mapping Brain Activity with Electrocorticography: Resolution Properties and Robustness of Inverse Solutions

Electrocorticography (ECoG) is an electrophysiological technique that records brain activity directly from the cortical surface with high temporal (ms) and spatial (mm) resolution. Its major limitations are in the high invasiveness and in the restricted field-of-view of the electrode grid, which partially covers the cortex. To infer brain activity at locations different from just below the electrodes, it is necessary to solve the electromagnetic inverse problem. Limitations in the performance of source reconstruction algorithms from ECoG have been, to date, only partially addressed in the literature, and a systematic evaluation is still lacking. The main goal of this study is to provide a quantitative evaluation of resolution properties of widely used inverse methods (eLORETA and MNE) for various ECoG grid sizes, in terms of localization error, spatial dispersion, and overall amplitude. Additionally, this study aims at evaluating how the use of simultaneous electroencephalography (EEG) affects the above properties. For these purposes, we take advantage of a unique dataset in which a monkey underwent a simultaneous recording with a 128 channel ECoG grid and an 18 channel EEG grid. Our results show that, in general conditions, the reconstruction of cortical activity located more than 1 cm away from the ECoG grid is not accurate, since the localization error increases linearly with the distance from the electrodes. This problem can be partially overcome by recording simultaneously ECoG and EEG. However, this analysis enlightens the necessity to design inverse algorithms specifically targeted at taking into account the limited field-of-view of the ECoG grid.