Spatiotemporal dynamics of electrocorticographic high gamma activity during overt and covert word repetition

Differential Representation of Articulatory Gestures and Phonemes in Precentral and Inferior Frontal Gyri

Speech is a critical form of human communication and is central to our daily lives. Yet, despite decades of study, an understanding of the fundamental neural control of speech production remains incomplete. Current theories model speech production as a hierarchy from sentences and phrases down to words, syllables, speech sounds (phonemes), and the actions of vocal tract articulators used to produce speech sounds (articulatory gestures). Here, we investigate the cortical representation of articulatory gestures and phonemes in ventral precentral and inferior frontal gyri in men and women. Our results indicate that ventral precentral cortex represents gestures to a greater extent than phonemes, while inferior frontal cortex represents both gestures and phonemes. These findings suggest that speech production shares a common cortical representation with that of other types of movement, such as arm and hand movements. This has important implications both for our understanding of speech production and for the design of brain-machine interfaces to restore communication to people who cannot speak.SIGNIFICANCE STATEMENT Despite being studied for decades, the production of speech by the brain is not fully understood. In particular, the most elemental parts of speech, speech sounds (phonemes) and the movements of vocal tract articulators used to produce these sounds (articulatory gestures), have both been hypothesized to be encoded in motor cortex. Using direct cortical recordings, we found evidence that primary motor and premotor cortices represent gestures to a greater extent than phonemes. Inferior frontal cortex (part of Broca's area) appears to represent both gestures and phonemes. These findings suggest that speech production shares a similar cortical organizational structure with the movement of other body parts.

Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

A direct-speech brain-computer interface (DS-BCI) acquires neural signals corresponding to imagined speech, then processes and decodes these signals to produce a linguistic output in the form of phonemes, words, or sentences. Recent research has shown the potential of neurolinguistics to enhance decoding approaches to imagined speech with the inclusion of semantics and phonology in experimental procedures. As neurolinguistics research findings are beginning to be incorporated within the scope of DS-BCI research, it is our view that a thorough understanding of imagined speech, and its relationship with overt speech, must be considered an integral feature of research in this field. With a focus on imagined speech, we provide a review of the most important neurolinguistics research informing the field of DS-BCI and suggest how this research may be utilized to improve current experimental protocols and decoding techniques. Our review of the literature supports a cross-disciplinary approach to DS-BCI research, in which neurolinguistics concepts and methods are utilized to aid development of a naturalistic mode of communication.

Repeated Vowel Production Affects Features of Neural Activity in Sensorimotor Cortex

The sensorimotor cortex is responsible for the generation of movements and interest in the ability to use this area for decoding speech by brain–computer interfaces has increased recently. Speech decoding is challenging however, since the relationship between neural activity and motor actions is not completely understood. Non-linearity between neural activity and movement has been found for instance for simple finger movements. Despite equal motor output, neural activity amplitudes are affected by preceding movements and the time between movements. It is unknown if neural activity is also affected by preceding motor actions during speech. We addressed this issue, using electrocorticographic high frequency band (HFB; 75–135 Hz) power changes in the sensorimotor cortex during discrete vowel generation. Three subjects with temporarily implanted electrode grids produced the /i/ vowel at repetition rates of 1, 1.33 and 1.66 Hz. For every repetition, the HFB power amplitude was determined. During the first utterance, most electrodes showed a large HFB power peak, which decreased for subsequent utterances. This result could not be explained by differences in performance. With increasing duration between utterances, more electrodes showed an equal response to all repetitions, suggesting that the duration between vowel productions influences the effect of previous productions on sensorimotor cortex activity. Our findings correspond with previous studies for finger movements and bear relevance for the development of brain-computer interfaces that employ speech decoding based on brain signals, in that past utterances will need to be taken into account for these systems to work accurately.

Decoding Inner Speech Using Electrocorticography: Progress and Challenges Toward a Speech Prosthesis

Certain brain disorders resulting from brainstem infarcts, traumatic brain injury, cerebral palsy, stroke, and amyotrophic lateral sclerosis, limit verbal communication despite the patient being fully aware. People that cannot communicate due to neurological disorders would benefit from a system that can infer internal speech directly from brain signals. In this review article, we describe the state of the art in decoding inner speech, ranging from early acoustic sound features, to higher order speech units. We focused on intracranial recordings, as this technique allows monitoring brain activity with high spatial, temporal, and spectral resolution, and therefore is a good candidate to investigate inner speech. Despite intense efforts, investigating how the human cortex encodes inner speech remains an elusive challenge, due to the lack of behavioral and observable measures. We emphasize various challenges commonly encountered when investigating inner speech decoding, and propose potential solutions in order to get closer to a natural speech assistive device.

fMRI connectivity of expressive language in young children and adolescents

Studies of language representation in development have shown a bilateral distributed pattern of activation that becomes increasingly left-lateralized and focal from young childhood to adulthood. However, the level by which canonical and extra-canonical regions, including subcortical and cerebellar regions, contribute to language during development has not been well-characterized. In this study, we employed fMRI connectivity analyses (fcMRI) to characterize the distributed network supporting expressive language in a group of young children (age 4-6) and adolescents (age 16-18). We conducted an fcMRI analysis using seed-to-voxel and seed-to-ROI (region of interest) strategies to investigate interactions of left pars triangularis with other brain areas. The analyses showed significant interhemispheric connectivity in young children, with a minimal connectivity of the left pars triangularis to subcortical and cerebellar regions. In contrast, adolescents showed significant connectivity between the left IFG seed and left perisylvian cortex, left caudate and putamen, and regions of the right cerebellum. Importantly, fcMRI analyses indicated significant differences between groups at 3 anatomical clusters, including left IFG, left supramarginal gyrus, and right cerebellar crura, suggesting a role in the functional development of language.

Real-time detection and discrimination of visual perception using electrocorticographic signals

OBJECTIVE

Several neuroimaging studies have demonstrated that the ventral temporal cortex contains specialized regions that process visual stimuli. This study investigated the spatial and temporal dynamics of electrocorticographic (ECoG) responses to different types and colors of visual stimulation that were presented to four human participants, and demonstrated a real-time decoder that detects and discriminates responses to untrained natural images.

APPROACH

ECoG signals from the participants were recorded while they were shown colored and greyscale versions of seven types of visual stimuli (images of faces, objects, bodies, line drawings, digits, and kanji and hiragana characters), resulting in 14 classes for discrimination (experiment I). Additionally, a real-time system asynchronously classified ECoG responses to faces, kanji and black screens presented via a monitor (experiment II), or to natural scenes (i.e. the face of an experimenter, natural images of faces and kanji, and a mirror) (experiment III). Outcome measures in all experiments included the discrimination performance across types based on broadband γ activity.

MAIN RESULTS

Experiment I demonstrated an offline classification accuracy of 72.9% when discriminating among the seven types (without color separation). Further discrimination of grey versus colored images reached an accuracy of 67.1%. Discriminating all colors and types (14 classes) yielded an accuracy of 52.1%. In experiment II and III, the real-time decoder correctly detected 73.7% responses to face, kanji and black computer stimuli and 74.8% responses to presented natural scenes.

SIGNIFICANCE

Seven different types and their color information (either grey or color) could be detected and discriminated using broadband γ activity. Discrimination performance maximized for combined spatial-temporal information. The discrimination of stimulus color information provided the first ECoG-based evidence for color-related population-level cortical broadband γ responses in humans. Stimulus categories can be detected by their ECoG responses in real time within 500 ms with respect to stimulus onset.

A novel onset detection technique for brain-computer interfaces using sound-production related cognitive tasks in simulated-online system

OBJECTIVE

Self-paced EEG-based BCIs (SP-BCIs) have traditionally been avoided due to two sources of uncertainty: (1) precisely when an intentional command is sent by the brain, i.e., the command onset detection problem, and (2) how different the intentional command is when compared to non-specific (or idle) states. Performance evaluation is also a problem and there are no suitable standard metrics available. In this paper we attempted to tackle these issues.

APPROACH

Self-paced covert sound-production cognitive tasks (i.e., high pitch and siren-like sounds) were used to distinguish between intentional commands (IC) and idle states. The IC states were chosen for their ease of execution and negligible overlap with common cognitive states. Band power and a digital wavelet transform were used for feature extraction, and the Davies-Bouldin index was used for feature selection. Classification was performed using linear discriminant analysis.

MAIN RESULTS

Performance was evaluated under offline and simulated-online conditions. For the latter, a performance score called true-false-positive (TFP) rate, ranging from 0 (poor) to 100 (perfect), was created to take into account both classification performance and onset timing errors. Averaging the results from the best performing IC task for all seven participants, an 77.7% true-positive (TP) rate was achieved in offline testing. For simulated-online analysis the best IC average TFP score was 76.67% (87.61% TP rate, 4.05% false-positive rate).

SIGNIFICANCE

Results were promising when compared to previous IC onset detection studies using motor imagery, in which best TP rates were reported as 72.0% and 79.7%, and which, crucially, did not take timing errors into account. Moreover, based on our literature review, there is no previous covert sound-production onset detection system for spBCIs. Results showed that the proposed onset detection technique and TFP performance metric have good potential for use in SP-BCIs.

Cooperative cortical network for categorical processing of Chinese lexical tone

The variation of pitch in speech not only creates the intonation for affective communication but also signals different meaning of a word in tonal languages, like Chinese. Due to its subtle and brisk pitch contour distinction between tone categories, the underlying neural processing mechanism is largely unknown. Using direct recordings of the human brain, we found categorical neural responses to lexical tones over a distributed cooperative network that included not only the auditory areas in the temporal cortex but also motor areas in the frontal cortex. Strong causal links from the temporal cortex to the motor cortex were discovered, which provides new evidence of top-down influence and sensory–motor interaction during speech perception.

In tonal languages such as Chinese, lexical tone with varying pitch contours serves as a key feature to provide contrast in word meaning. Similar to phoneme processing, behavioral studies have suggested that Chinese tone is categorically perceived. However, its underlying neural mechanism remains poorly understood. By conducting cortical surface recordings in surgical patients, we revealed a cooperative cortical network along with its dynamics responsible for this categorical perception. Based on an oddball paradigm, we found amplified neural dissimilarity between cross-category tone pairs, rather than between within-category tone pairs, over cortical sites covering both the ventral and dorsal streams of speech processing. The bilateral superior temporal gyrus (STG) and the middle temporal gyrus (MTG) exhibited increased response latencies and enlarged neural dissimilarity, suggesting a ventral hierarchy that gradually differentiates the acoustic features of lexical tones. In addition, the bilateral motor cortices were also found to be involved in categorical processing, interacting with both the STG and the MTG and exhibiting a response latency in between. Moreover, the motor cortex received enhanced Granger causal influence from the semantic hub, the anterior temporal lobe, in the right hemisphere. These unique data suggest that there exists a distributed cooperative cortical network supporting the categorical processing of lexical tone in tonal language speakers, not only encompassing a bilateral temporal hierarchy that is shared by categorical processing of phonemes but also involving intensive speech–motor interactions over the right hemisphere, which might be the unique machinery responsible for the reliable discrimination of tone identities.

Non-invasive detection of language-related prefrontal high gamma band activity with beamforming MEG

High gamma band (>50 Hz) activity is a key oscillatory phenomenon of brain activation. However, there has not been a non-invasive method established to detect language-related high gamma band activity. We used a 160-channel whole-head magnetoencephalography (MEG) system equipped with superconducting quantum interference device (SQUID) gradiometers to non-invasively investigate neuromagnetic activities during silent reading and verb generation tasks in 15 healthy participants. Individual data were divided into alpha (8–13 Hz), beta (13–25 Hz), low gamma (25–50 Hz), and high gamma (50–100 Hz) bands and analysed with the beamformer method. The time window was consecutively moved. Group analysis was performed to delineate common areas of brain activation. In the verb generation task, transient power increases in the high gamma band appeared in the left middle frontal gyrus (MFG) at the 550–750 ms post-stimulus window. We set a virtual sensor on the left MFG for time-frequency analysis, and high gamma event-related synchronization (ERS) induced by a verb generation task was demonstrated at 650 ms. In contrast, ERS in the high gamma band was not detected in the silent reading task. Thus, our study successfully non-invasively measured language-related prefrontal high gamma band activity.

Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids

For people who cannot communicate due to severe paralysis or involuntary movements, technology that decodes intended speech from the brain may offer an alternative means of communication. If decoding proves to be feasible, intracranial Brain-Computer Interface systems can be developed which are designed to translate decoded speech into computer generated speech or to instructions for controlling assistive devices. Recent advances suggest that such decoding may be feasible from sensorimotor cortex, but it is not clear how this challenge can be approached best. One approach is to identify and discriminate elements of spoken language, such as phonemes. We investigated feasibility of decoding four spoken phonemes from the sensorimotor face area, using electrocorticographic signals obtained with high-density electrode grids. Several decoding algorithms including spatiotemporal matched filters, spatial matched filters and support vector machines were compared. Phonemes could be classified correctly at a level of over 75% with spatiotemporal matched filters. Support Vector machine analysis reached a similar level, but spatial matched filters yielded significantly lower scores. The most informative electrodes were clustered along the central sulcus. Highest scores were achieved from time windows centered around voice onset time, but a 500 ms window before onset time could also be classified significantly. The results suggest that phoneme production involves a sequence of robust and reproducible activity patterns on the cortical surface. Importantly, decoding requires inclusion of temporal information to capture the rapid shifts of robust patterns associated with articulator muscle group contraction during production of a phoneme. The high classification scores are likely to be enabled by the use of high density grids, and by the use of discrete phonemes. Implications for use in Brain-Computer Interfaces are discussed.

Evaluating the roles of left middle frontal gyrus in word production using electrocorticography

To assess the specific roles of left middle frontal gyrus (LMFG) in word production, electrocorticography signals were recorded from an epilepsy patient when he participated in language tasks. We found three sites of LMFG showed high-gamma perturbations with distinct patterns across tasks; and neural activities elicited in the same tasks shared similar patterns, while those elicited by stimuli leading to the same articulations did not. These findings confirmed that the LMFG takes active parts in word production, and suggested that it may serve as a temporal perceptual information storage space, supporting the hierarchical state feedback control model of word production.

Key considerations in designing a speech brain-computer interface

Restoring communication in case of aphasia is a key challenge for neurotechnologies. To this end, brain-computer strategies can be envisioned to allow artificial speech synthesis from the continuous decoding of neural signals underlying speech imagination. Such speech brain-computer interfaces do not exist yet and their design should consider three key choices that need to be made: the choice of appropriate brain regions to record neural activity from, the choice of an appropriate recording technique, and the choice of a neural decoding scheme in association with an appropriate speech synthesis method. These key considerations are discussed here in light of (1) the current understanding of the functional neuroanatomy of cortical areas underlying overt and covert speech production, (2) the available literature making use of a variety of brain recording techniques to better characterize and address the challenge of decoding cortical speech signals, and (3) the different speech synthesis approaches that can be considered depending on the level of speech representation (phonetic, acoustic or articulatory) envisioned to be decoded at the core of a speech BCI paradigm.

ECoG data analyses to inform closed-loop BCI experiments for speech-based prosthetic applications

Brain Computer Interfaces (BCIs) assist individuals with motor disabilities by enabling them to control prosthetic devices with their neural activity. Performance of closed-loop BCI systems can be improved by using design strategies that leverage structured and task-relevant neural activity. We use data from high density electrocorticography (ECoG) grids implanted in three subjects to study sensory-motor activity during an instructed speech task in which the subjects vocalized three cardinal vowel phonemes. We show how our findings relate to the current understanding of speech physiology and functional organization of human sensory-motor cortex. We investigate the effect of behavioral variations on parameters and performance of the decoding model. Our analyses suggest experimental design strategies that may be critical for speech-based BCI performance.

Instantaneous voltage as an alternative to power- and phase-based interpretation of oscillatory brain activity

For decades, oscillatory brain activity has been characterized primarily by measurements of power and phase. While many studies have linked those measurements to cortical excitability, their relationship to each other and to the physiological underpinnings of excitability is unclear. The recently proposed Function-through-Biased-Oscillations (FBO) hypothesis (Schalk, 2015) addressed these issues by suggesting that the voltage potential at the cortical surface directly reflects the excitability of cortical populations, that this voltage is rhythmically driven away from a low resting potential (associated with depolarized cortical populations) towards positivity (associated with hyperpolarized cortical populations). This view explains how oscillatory power and phase together influence the instantaneous voltage potential that directly regulates cortical excitability. This implies that the alternative measurement of instantaneous voltage of oscillatory activity should better predict cortical excitability compared to either of the more traditional measurements of power or phase. Using electrocorticographic (ECoG) data from 28 human subjects, the results of our study confirm this prediction: compared to oscillatory power and phase, the instantaneous voltage explained 20% and 31% more of the variance in broadband gamma, respectively, and power and phase together did not produce better predictions than the instantaneous voltage. These results synthesize the previously separate power- and phase-based interpretations and associate oscillatory activity directly with a physiological interpretation of cortical excitability. This alternative view has implications for the interpretation of studies of oscillatory activity and for current theories of cortical information transmission.

Auditory processing in the human cortex: An intracranial electrophysiology perspective

Objective

Direct electrophysiological recordings in epilepsy patients offer an opportunity to study human auditory cortical processing with unprecedented spatiotemporal resolution. This review highlights recent intracranial studies of human auditory cortex and focuses on its basic response properties as well as modulation of cortical activity during the performance of active behavioral tasks.

Data Sources: Literature review.

Review Methods: A review of the literature was conducted to summarize the functional organization of human auditory and auditory‐related cortex as revealed using intracranial recordings.

Results

The tonotopically organized core auditory cortex within the posteromedial portion of Heschl's gyrus represents spectrotemporal features of sounds with high temporal precision and short response latencies. At this level of processing, high gamma (70–150 Hz) activity is minimally modulated by task demands. Non‐core cortex on the lateral surface of the superior temporal gyrus also maintains representation of stimulus acoustic features and, for speech, subserves transformation of acoustic inputs into phonemic representations. High gamma responses in this region are modulated by task requirements. Prefrontal cortex exhibits complex response patterns, related to stimulus intelligibility and task relevance. At this level of auditory processing, activity is strongly modulated by task requirements and reflects behavioral performance.

Conclusions

Direct recordings from the human brain reveal hierarchical organization of sound processing within auditory and auditory‐related cortex.

Level of Evidence

Level V

Mapping Critical Language Sites in Children Performing Verb Generation: Whole-Brain Connectivity and Graph Theoretical Analysis in MEG

A classic left frontal-temporal brain network is known to support language processes. However, the level of participation of constituent regions, and the contribution of extra-canonical areas, is not fully understood; this is particularly true in children, and in individuals who have experienced early neurological insult. In the present work, we propose whole-brain connectivity and graph-theoretical analysis of magnetoencephalography (MEG) source estimates to provide robust maps of the pediatric expressive language network. We examined neuromagnetic data from a group of typically-developing young children (n = 15, ages 4–6 years) and adolescents (n = 14, 16–18 years) completing an auditory verb generation task in MEG. All source analyses were carried out using a linearly-constrained minimum-variance (LCMV) beamformer. Conventional differential analyses revealed significant (p < 0.05, corrected) low-beta (13–23 Hz) event related desynchrony (ERD) focused in the left inferior frontal region (Broca’s area) in both groups, consistent with previous studies. Connectivity analyses were carried out in broadband (3–30 Hz) on time-course estimates obtained at the voxel level. Patterns of connectivity were characterized by phase locking value (PLV), and network hubs identified through eigenvector centrality (EVC). Hub analysis revealed the importance of left perisylvian sites, i.e., Broca’s and Wernicke’s areas, across groups. The hemispheric distribution of frontal and temporal lobe EVC values was asymmetrical in most subjects; left dominant EVC was observed in 20% of young children, and 71% of adolescents. Interestingly, the adolescent group demonstrated increased critical sites in the right cerebellum, left inferior frontal gyrus (IFG) and left putamen. Here, we show that whole brain connectivity and network analysis can be used to map critical language sites in typical development; these methods may be useful for defining the margins of eloquent tissue in neurosurgical candidates.

A Spiking Neurocomputational Model of High-Frequency Oscillatory Brain Responses to Words and Pseudowords

Experimental evidence indicates that neurophysiological responses to well-known meaningful sensory items and symbols (such as familiar objects, faces, or words) differ from those to matched but novel and senseless materials (unknown objects, scrambled faces, and pseudowords). Spectral responses in the high beta- and gamma-band have been observed to be generally stronger to familiar stimuli than to unfamiliar ones. These differences have been hypothesized to be caused by the activation of distributed neuronal circuits or cell assemblies, which act as long-term memory traces for learned familiar items only. Here, we simulated word learning using a biologically constrained neurocomputational model of the left-hemispheric cortical areas known to be relevant for language and conceptual processing. The 12-area spiking neural-network architecture implemented replicates physiological and connectivity features of primary, secondary, and higher-association cortices in the frontal, temporal, and occipital lobes of the human brain. We simulated elementary aspects of word learning in it, focussing specifically on semantic grounding in action and perception. As a result of spike-driven Hebbian synaptic plasticity mechanisms, distributed, stimulus-specific cell-assembly (CA) circuits spontaneously emerged in the network. After training, presentation of one of the learned "word" forms to the model correlate of primary auditory cortex induced periodic bursts of activity within the corresponding CA, leading to oscillatory phenomena in the entire network and spontaneous across-area neural synchronization. Crucially, Morlet wavelet analysis of the network's responses recorded during presentation of learned meaningful "word" and novel, senseless "pseudoword" patterns revealed stronger induced spectral power in the gamma-band for the former than the latter, closely mirroring differences found in neurophysiological data. Furthermore, coherence analysis of the simulated responses uncovered dissociated category specific patterns of synchronous oscillations in distant cortical areas, including indirectly connected primary sensorimotor areas. Bridging the gap between cellular-level mechanisms, neuronal-population behavior, and cognitive function, the present model constitutes the first spiking, neurobiologically, and anatomically realistic model able to explain high-frequency oscillatory phenomena indexing language processing on the basis of dynamics and competitive interactions of distributed cell-assembly circuits which emerge in the brain as a result of Hebbian learning and sensorimotor experience.

Spatio-Temporal Progression of Cortical Activity Related to Continuous Overt and Covert Speech Production in a Reading Task

How the human brain plans, executes, and monitors continuous and fluent speech has remained largely elusive. For example, previous research has defined the cortical locations most important for different aspects of speech function, but has not yet yielded a definition of the temporal progression of involvement of those locations as speech progresses either overtly or covertly. In this paper, we uncovered the spatio-temporal evolution of neuronal population-level activity related to continuous overt speech, and identified those locations that shared activity characteristics across overt and covert speech. Specifically, we asked subjects to repeat continuous sentences aloud or silently while we recorded electrical signals directly from the surface of the brain (electrocorticography (ECoG)). We then determined the relationship between cortical activity and speech output across different areas of cortex and at sub-second timescales. The results highlight a spatio-temporal progression of cortical involvement in the continuous speech process that initiates utterances in frontal-motor areas and ends with the monitoring of auditory feedback in superior temporal gyrus. Direct comparison of cortical activity related to overt versus covert conditions revealed a common network of brain regions involved in speech that may implement orthographic and phonological processing. Our results provide one of the first characterizations of the spatiotemporal electrophysiological representations of the continuous speech process, and also highlight the common neural substrate of overt and covert speech. These results thereby contribute to a refined understanding of speech functions in the human brain.

Automatic Speech Recognition from Neural Signals: A Focused Review

Speech interfaces have become widely accepted and are nowadays integrated in various real-life applications and devices. They have become a part of our daily life. However, speech interfaces presume the ability to produce intelligible speech, which might be impossible due to either loud environments, bothering bystanders or incapabilities to produce speech (i.e., patients suffering from locked-in syndrome). For these reasons it would be highly desirable to not speak but to simply envision oneself to say words or sentences. Interfaces based on imagined speech would enable fast and natural communication without the need for audible speech and would give a voice to otherwise mute people. This focused review analyzes the potential of different brain imaging techniques to recognize speech from neural signals by applying Automatic Speech Recognition technology. We argue that modalities based on metabolic processes, such as functional Near Infrared Spectroscopy and functional Magnetic Resonance Imaging, are less suited for Automatic Speech Recognition from neural signals due to low temporal resolution but are very useful for the investigation of the underlying neural mechanisms involved in speech processes. In contrast, electrophysiologic activity is fast enough to capture speech processes and is therefor better suited for ASR. Our experimental results indicate the potential of these signals for speech recognition from neural data with a focus on invasively measured brain activity (electrocorticography). As a first example of Automatic Speech Recognition techniques used from neural signals, we discuss the Brain-to-text system.

Trends and Challenges in Neuroengineering: Toward "Intelligent" Neuroprostheses through Brain-"Brain Inspired Systems" Communication

Future technologies aiming at restoring and enhancing organs function will intimately rely on near-physiological and energy-efficient communication between living and artificial biomimetic systems. Interfacing brain-inspired devices with the real brain is at the forefront of such emerging field, with the term "neurobiohybrids" indicating all those systems where such interaction is established. We argue that achieving a "high-level" communication and functional synergy between natural and artificial neuronal networks in vivo, will allow the development of a heterogeneous world of neurobiohybrids, which will include "living robots" but will also embrace "intelligent" neuroprostheses for augmentation of brain function. The societal and economical impact of intelligent neuroprostheses is likely to be potentially strong, as they will offer novel therapeutic perspectives for a number of diseases, and going beyond classical pharmaceutical schemes. However, they will unavoidably raise fundamental ethical questions on the intermingling between man and machine and more specifically, on how deeply it should be allowed that brain processing is affected by implanted "intelligent" artificial systems. Following this perspective, we provide the reader with insights on ongoing developments and trends in the field of neurobiohybrids. We address the topic also from a "community building" perspective, showing through a quantitative bibliographic analysis, how scientists working on the engineering of brain-inspired devices and brain-machine interfaces are increasing their interactions. We foresee that such trend preludes to a formidable technological and scientific revolution in brain-machine communication and to the opening of new avenues for restoring or even augmenting brain function for therapeutic purposes.

Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity

OBJECTIVE

The superior temporal gyrus (STG) and neighboring brain regions play a key role in human language processing. Previous studies have attempted to reconstruct speech information from brain activity in the STG, but few of them incorporate the probabilistic framework and engineering methodology used in modern speech recognition systems. In this work, we describe the initial efforts toward the design of a neural speech recognition (NSR) system that performs continuous phoneme recognition on English stimuli with arbitrary vocabulary sizes using the high gamma band power of local field potentials in the STG and neighboring cortical areas obtained via electrocorticography.

APPROACH

The system implements a Viterbi decoder that incorporates phoneme likelihood estimates from a linear discriminant analysis model and transition probabilities from an n-gram phonemic language model. Grid searches were used in an attempt to determine optimal parameterizations of the feature vectors and Viterbi decoder.

MAIN RESULTS

The performance of the system was significantly improved by using spatiotemporal representations of the neural activity (as opposed to purely spatial representations) and by including language modeling and Viterbi decoding in the NSR system.

SIGNIFICANCE

These results emphasize the importance of modeling the temporal dynamics of neural responses when analyzing their variations with respect to varying stimuli and demonstrate that speech recognition techniques can be successfully leveraged when decoding speech from neural signals. Guided by the results detailed in this work, further development of the NSR system could have applications in the fields of automatic speech recognition and neural prosthetics.

A method to establish the spatiotemporal evolution of task-related cortical activity from electrocorticographic signals in single trials

BACKGROUND

Progress in neuroscience depends substantially on the ability to establish the detailed spatial and temporal sequence of neuronal population-level activity across large areas of the brain. Because there is substantial inter-trial variability in neuronal activity, traditional techniques that rely on signal averaging obscure where and when neuronal activity occurs. Thus, up to the present, it has been difficult to examine the detailed progression of neuronal activity across large areas of the brain.

NEW METHOD

Here we describe a method for establishing the spatiotemporal evolution of neuronal population-level activity across large brain regions by determining exactly where and when neural activity occurs during a behavioral task in individual trials. We validate the efficacy of the method, examine the effects of its parameterization, and demonstrate its utility by highlighting two sets of results that could not readily be achieved with traditional methods.

RESULTS

Our results reveal the precise spatiotemporal evolution of neuronal population activity that unfolds during a sensorimotor task in individual trials, and establishes the relationship between neuronal oscillations and the onset of this activity.

CONCLUSIONS

The ability to identify the spatiotemporal evolution of neuronal population activity onsets in single trials gives investigators a powerful new tool with which to study large-scale cortical processes.

Alpha power indexes task-related networks on large and small scales: A multimodal ECoG study in humans and a non-human primate

Performing different tasks, such as generating motor movements or processing sensory input, requires the recruitment of specific networks of neuronal populations. Previous studies suggested that power variations in the alpha band (8-12Hz) may implement such recruitment of task-specific populations by increasing cortical excitability in task-related areas while inhibiting population-level cortical activity in task-unrelated areas (Klimesch et al., 2007; Jensen and Mazaheri, 2010). However, the precise temporal and spatial relationships between the modulatory function implemented by alpha oscillations and population-level cortical activity remained undefined. Furthermore, while several studies suggested that alpha power indexes task-related populations across large and spatially separated cortical areas, it was largely unclear whether alpha power also differentially indexes smaller networks of task-related neuronal populations. Here we addressed these questions by investigating the temporal and spatial relationships of electrocorticographic (ECoG) power modulations in the alpha band and in the broadband gamma range (70-170Hz, indexing population-level activity) during auditory and motor tasks in five human subjects and one macaque monkey. In line with previous research, our results confirm that broadband gamma power accurately tracks task-related behavior and that alpha power decreases in task-related areas. More importantly, they demonstrate that alpha power suppression lags population-level activity in auditory areas during the auditory task, but precedes it in motor areas during the motor task. This suppression of alpha power in task-related areas was accompanied by an increase in areas not related to the task. In addition, we show for the first time that these differential modulations of alpha power could be observed not only across widely distributed systems (e.g., motor vs. auditory system), but also within the auditory system. Specifically, alpha power was suppressed in the locations within the auditory system that most robustly responded to particular sound stimuli. Altogether, our results provide experimental evidence for a mechanism that preferentially recruits task-related neuronal populations by increasing cortical excitability in task-related cortical areas and decreasing cortical excitability in task-unrelated areas. This mechanism is implemented by variations in alpha power and is common to humans and the non-human primate under study. These results contribute to an increasingly refined understanding of the mechanisms underlying the selection of the specific neuronal populations required for task execution.

Word pair classification during imagined speech using direct brain recordings

People that cannot communicate due to neurological disorders would benefit from an internal speech decoder. Here, we showed the ability to classify individual words during imagined speech from electrocorticographic signals. In a word imagery task, we used high gamma (70–150 Hz) time features with a support vector machine model to classify individual words from a pair of words. To account for temporal irregularities during speech production, we introduced a non-linear time alignment into the SVM kernel. Classification accuracy reached 88% in a two-class classification framework (50% chance level), and average classification accuracy across fifteen word-pairs was significant across five subjects (mean = 58%; p < 0.05). We also compared classification accuracy between imagined speech, overt speech and listening. As predicted, higher classification accuracy was obtained in the listening and overt speech conditions (mean = 89% and 86%, respectively; p < 0.0001), where speech stimuli were directly presented. The results provide evidence for a neural representation for imagined words in the temporal lobe, frontal lobe and sensorimotor cortex, consistent with previous findings in speech perception and production. These data represent a proof of concept study for basic decoding of speech imagery, and delineate a number of key challenges to usage of speech imagery neural representations for clinical applications.

Restoring motor control and sensory feedback in people with upper extremity amputations using arrays of 96 microelectrodes implanted in the median and ulnar nerves

OBJECTIVE

An important goal of neuroprosthetic research is to establish bidirectional communication between the user and new prosthetic limbs that are capable of controlling >20 different movements. One strategy for achieving this goal is to interface the prosthetic limb directly with efferent and afferent fibres in the peripheral nervous system using an array of intrafascicular microelectrodes. This approach would provide access to a large number of independent neural pathways for controlling high degree-of-freedom prosthetic limbs, as well as evoking multiple-complex sensory percepts.

APPROACH

Utah Slanted Electrode Arrays (USEAs, 96 recording/stimulating electrodes) were implanted for 30 days into the median (Subject 1-M, 31 years post-amputation) or ulnar (Subject 2-U, 1.5 years post-amputation) nerves of two amputees. Neural activity was recorded during intended movements of the subject's phantom fingers and a linear Kalman filter was used to decode the neural data. Microelectrode stimulation of varying amplitudes and frequencies was delivered via single or multiple electrodes to investigate the number, size and quality of sensory percepts that could be evoked. Device performance over time was assessed by measuring: electrode impedances, signal-to-noise ratios (SNRs), stimulation thresholds, number and stability of evoked percepts.

MAIN RESULTS

The subjects were able to proportionally, control individual fingers of a virtual robotic hand, with 13 different movements decoded offline (r = 0.48) and two movements decoded online. Electrical stimulation across one USEA evoked >80 sensory percepts. Varying the stimulation parameters modulated percept quality. Devices remained intrafascicularly implanted for the duration of the study with no significant changes in the SNRs or percept thresholds.

SIGNIFICANCE

This study demonstrated that an array of 96 microelectrodes can be implanted into the human peripheral nervous system for up to 1 month durations. Such an array could provide intuitive control of a virtual prosthetic hand with broad sensory feedback.

Intracranial EEG potentials estimated from MEG sources: A new approach to correlate MEG and iEEG data in epilepsy

Detection of epileptic spikes in MagnetoEncephaloGraphy (MEG) requires synchronized neuronal activity over a minimum of 4cm2. We previously validated the Maximum Entropy on the Mean (MEM) as a source localization able to recover the spatial extent of the epileptic spike generators. The purpose of this study was to evaluate quantitatively, using intracranial EEG (iEEG), the spatial extent recovered from MEG sources by estimating iEEG potentials generated by these MEG sources. We evaluated five patients with focal epilepsy who had a pre-operative MEG acquisition and iEEG with MRI-compatible electrodes. Individual MEG epileptic spikes were localized along the cortical surface segmented from a pre-operative MRI, which was co-registered with the MRI obtained with iEEG electrodes in place for identification of iEEG contacts. An iEEG forward model estimated the influence of every dipolar source of the cortical surface on each iEEG contact. This iEEG forward model was applied to MEG sources to estimate iEEG potentials that would have been generated by these sources. MEG-estimated iEEG potentials were compared with measured iEEG potentials using four source localization methods: two variants of MEM and two standard methods equivalent to minimum norm and LORETA estimates. Our results demonstrated an excellent MEG/iEEG correspondence in the presumed focus for four out of five patients. In one patient, the deep generator identified in iEEG could not be localized in MEG. MEG-estimated iEEG potentials is a promising method to evaluate which MEG sources could be retrieved and validated with iEEG data, providing accurate results especially when applied to MEM localizations. Hum Brain Mapp 37:1661-1683, 2016. © 2016 Wiley Periodicals, Inc.

Sparse Spectro-Temporal Receptive Fields Based on Multi-Unit and High-Gamma Responses in Human Auditory Cortex

Spectro-Temporal Receptive Fields (STRFs) were estimated from both multi-unit sorted clusters and high-gamma power responses in human auditory cortex. Intracranial electrophysiological recordings were used to measure responses to a random chord sequence of Gammatone stimuli. Traditional methods for estimating STRFs from single-unit recordings, such as spike-triggered-averages, tend to be noisy and are less robust to other response signals such as local field potentials. We present an extension to recently advanced methods for estimating STRFs from generalized linear models (GLM). A new variant of regression using regularization that penalizes non-zero coefficients is described, which results in a sparse solution. The frequency-time structure of the STRF tends toward grouping in different areas of frequency-time and we demonstrate that group sparsity-inducing penalties applied to GLM estimates of STRFs reduces the background noise while preserving the complex internal structure. The contribution of local spiking activity to the high-gamma power signal was factored out of the STRF using the GLM method, and this contribution was significant in 85 percent of the cases. Although the GLM methods have been used to estimate STRFs in animals, this study examines the detailed structure directly from auditory cortex in the awake human brain. We used this approach to identify an abrupt change in the best frequency of estimated STRFs along posteromedial-to-anterolateral recording locations along the long axis of Heschl's gyrus. This change correlates well with a proposed transition from core to non-core auditory fields previously identified using the temporal response properties of Heschl's gyrus recordings elicited by click-train stimuli.

Frequency-dependent spatiotemporal profiles of visual responses recorded with subdural ECoG electrodes in awake monkeys: Differences between high- and low-frequency activity

Electrocorticography (ECoG) constitutes a powerful and promising neural recording modality in humans and animals. ECoG signals are often decomposed into several frequency bands, among which the so-called high-gamma band (80-250Hz) has been proposed to reflect local cortical functions near the cortical surface below the ECoG electrodes. It is typically assumed that the lower the frequency bands, the lower the spatial resolution of the signals; thus, there is not much to gain by analyzing the event-related changes of the ECoG signals in the lower-frequency bands. However, differences across frequency bands have not been systematically investigated. To address this issue, we recorded ECoG activity from two awake monkeys performing a retinotopic mapping task. We characterized the spatiotemporal profiles of the visual responses in the time-frequency domain. We defined the preferred spatial position, receptive field (RF), and response latencies of band-limited power (BLP) (i.e., alpha [3.9-11.7Hz], beta [15.6-23.4Hz], low [30-80Hz] and high [80-250Hz] gamma) for each electrode and compared them across bands and time-domain visual evoked potentials (VEPs). At the population level, we found that the spatial preferences were comparable across bands and VEPs. The high-gamma power showed a smaller RF than the other bands and VEPs. The response latencies for the alpha band were always longer than the latencies for the other bands and fastest in VEPs. Comparing the response profiles in both space and time for each cortical region (V1, V4+, and TEO/TE) revealed regional idiosyncrasies. Although the latencies of visual responses in the beta, low-, and high-gamma bands were almost identical in V1 and V4+, beta and low-gamma BLP occurred about 17ms earlier than high-gamma power in TEO/TE. Furthermore, TEO/TE exhibited a unique pattern in the spatial response profile: the alpha and high-gamma responses tended to prefer the foveal regions, whereas the beta and low-gamma responses preferred the peripheral visual fields with larger RFs. This suggests that neurons in TEO/TE first receive less selective spatial information via beta and low-gamma BLP but later receive more fine-tuned spatial foveal information via high-gamma power. This result is consistent with a hypothesis previously proposed by Nakamura et al. (1993) that states that visual processing in TEO/TE starts with coarse-grained information, which primes subsequent fine-grained information. Collectively, our results demonstrate that ECoG can be a potent tool for investigating the nature of the neural computations in each cortical region that cannot be fully understood by measuring only the spiking activity, through the incorporation of the knowledge of the spatiotemporal characteristics across all frequency bands.

Brain-to-text: decoding spoken phrases from phone representations in the brain

It has long been speculated whether communication between humans and machines based on natural speech related cortical activity is possible. Over the past decade, studies have suggested that it is feasible to recognize isolated aspects of speech from neural signals, such as auditory features, phones or one of a few isolated words. However, until now it remained an unsolved challenge to decode continuously spoken speech from the neural substrate associated with speech and language processing. Here, we show for the first time that continuously spoken speech can be decoded into the expressed words from intracranial electrocorticographic (ECoG) recordings.Specifically, we implemented a system, which we call Brain-To-Text that models single phones, employs techniques from automatic speech recognition (ASR), and thereby transforms brain activity while speaking into the corresponding textual representation. Our results demonstrate that our system can achieve word error rates as low as 25% and phone error rates below 50%. Additionally, our approach contributes to the current understanding of the neural basis of continuous speech production by identifying those cortical regions that hold substantial information about individual phones. In conclusion, the Brain-To-Text system described in this paper represents an important step toward human-machine communication based on imagined speech.

NeuralAct: A Tool to Visualize Electrocortical (ECoG) Activity on a Three-Dimensional Model of the Cortex

Electrocorticography (ECoG) records neural signals directly from the surface of the cortex. Due to its high temporal and favorable spatial resolution, ECoG has emerged as a valuable new tool in acquiring cortical activity in cognitive and systems neuroscience. Many studies using ECoG visualized topographies of cortical activity or statistical tests on a three-dimensional model of the cortex, but a dedicated tool for this function has not yet been described. In this paper, we describe the NeuralAct package that serves this purpose. This package takes as input the 3D coordinates of the recording sensors, a cortical model in the same coordinate system (e.g., Talairach), and the activation data to be visualized at each sensor. It then aligns the sensor coordinates with the cortical model, convolves the activation data with a spatial kernel, and renders the resulting activations in color on the cortical model. The NeuralAct package can plot cortical activations of an individual subject as well as activations averaged over subjects. It is capable to render single images as well as sequences of images. The software runs under Matlab and is stable and robust. We here provide the tool and describe its visualization capabilities and procedures. The provided package contains thoroughly documented code and includes a simple demo that guides the researcher through the functionality of the tool.

Electrocorticographic representations of segmental features in continuous speech

Acoustic speech output results from coordinated articulation of dozens of muscles, bones and cartilages of the vocal mechanism. While we commonly take the fluency and speed of our speech productions for granted, the neural mechanisms facilitating the requisite muscular control are not completely understood. Previous neuroimaging and electrophysiology studies of speech sensorimotor control has typically concentrated on speech sounds (i.e., phonemes, syllables and words) in isolation; sentence-length investigations have largely been used to inform coincident linguistic processing. In this study, we examined the neural representations of segmental features (place and manner of articulation, and voicing status) in the context of fluent, continuous speech production. We used recordings from the cortical surface [electrocorticography (ECoG)] to simultaneously evaluate the spatial topography and temporal dynamics of the neural correlates of speech articulation that may mediate the generation of hypothesized gestural or articulatory scores. We found that the representation of place of articulation involved broad networks of brain regions during all phases of speech production: preparation, execution and monitoring. In contrast, manner of articulation and voicing status were dominated by auditory cortical responses after speech had been initiated. These results provide a new insight into the articulatory and auditory processes underlying speech production in terms of their motor requirements and acoustic correlates.

Redefining the role of Broca's area in speech

For over a century neuroscientists have debated the dynamics by which human cortical language networks allow words to be spoken. Although it is widely accepted that Broca's area in the left inferior frontal gyrus plays an important role in this process, it was not possible, until recently, to detail the timing of its recruitment relative to other language areas, nor how it interacts with these areas during word production. Using direct cortical surface recordings in neurosurgical patients, we studied the evolution of activity in cortical neuronal populations, as well as the Granger causal interactions between them. We found that, during the cued production of words, a temporal cascade of neural activity proceeds from sensory representations of words in temporal cortex to their corresponding articulatory gestures in motor cortex. Broca's area mediates this cascade through reciprocal interactions with temporal and frontal motor regions. Contrary to classic notions of the role of Broca's area in speech, while motor cortex is activated during spoken responses, Broca's area is surprisingly silent. Moreover, when novel strings of articulatory gestures must be produced in response to nonword stimuli, neural activity is enhanced in Broca's area, but not in motor cortex. These unique data provide evidence that Broca's area coordinates the transformation of information across large-scale cortical networks involved in spoken word production. In this role, Broca's area formulates an appropriate articulatory code to be implemented by motor cortex.

Sequential activation of premotor, primary somatosensory and primary motor areas in humans during cued finger movements

OBJECTIVE

Human voluntary movements are a final product of complex interactions between multiple sensory, cognitive and motor areas of central nervous system. The objective was to investigate temporal sequence of activation of premotor (PM), primary motor (M1) and somatosensory (S1) areas during cued finger movements.

METHODS

Electrocorticography (ECoG) was used to measure activation timing in human PM, S1, and M1 neurons in preparation for finger movements in 5 subjects with subdural grids for seizure localization. Cortical activation was determined by the onset of high gamma (HG) oscillation (70-150Hz). The three cortical regions were mapped anatomically using a common brain atlas and confirmed independently with direct electrical cortical stimulation, somatosensory evoked potentials and detection of HG response to tactile stimulation. Subjects were given visual cues to flex each finger or pinch the thumb and index finger. Movements were captured with a dataglove and time-locked with ECoG. A windowed covariance metric was used to identify the rising slope of HG power between two electrodes and compute time lag. Statistical constraints were applied to the time estimates to combat the noise. Rank sum testing was used to verify the sequential activation of cortical regions across 5 subjects.

RESULTS

In all 5 subjects, HG activation in PM preceded S1 by an average of 53±13ms (P=0.03), PM preceded M1 by 180±40ms (P=0.001) and S1 activation preceded M1 by 136±40ms (P=0.04).

CONCLUSIONS

Sequential HG activation of PM, S1 and M1 regions in preparation for movements is reported. Activity in S1 prior to any overt body movements supports the notion that these neurons may encode sensory information in anticipation of movements, i.e., an efference copy. Our analysis suggests that S1 modulation likely originates from PM.

SIGNIFICANCE

First electrophysiological evidence of efference copy in humans.

Electrocorticographic language mapping in children by high-gamma synchronization during spontaneous conversation: comparison with conventional electrical cortical stimulation

INTRODUCTION

This study describes development of a novel language mapping approach using high-γ modulation in electrocorticograph (ECoG) during spontaneous conversation, and its comparison with electrical cortical stimulation (ECS) in childhood-onset drug-resistant epilepsy.

METHODS

Patients undergoing invasive pre-surgical monitoring and able to converse with the investigator were eligible. ECoG signals and synchronized audio were acquired during quiet baseline and during natural conversation between investigator and the patient. Using Signal Modeling for Real-time Identification and Event Detection (SIGFRIED) procedure, a statistical model for baseline high-γ (70-116 Hz) power, and a single score for each channel representing the probability that the power features in the experimental signal window belonged to the baseline model, were calculated. Electrodes with significant high-γ responses (HGS) were plotted on the 3D cortical model. Sensitivity, specificity, positive and negative predictive values (PPV, NPV), and classification accuracy were calculated compared to ECS.

RESULTS

Seven patients were included (4 males, mean age 10.28 ± 4.07 years). Significant high-γ responses were observed in classic language areas in the left hemisphere plus in some homologous right hemispheric areas. Compared with clinical standard ECS mapping, the sensitivity and specificity of HGS mapping was 88.89% and 63.64%, respectively, and PPV and NPV were 35.29% and 96.25%, with an overall accuracy of 68.24%. HGS mapping was able to correctly determine all ECS+ sites in 6 of 7 patients and all false-sites (ECS+, HGS- for visual naming, n = 3) were attributable to only 1 patient.

CONCLUSIONS

This study supports the feasibility of language mapping with ECoG HGS during spontaneous conversation, and its accuracy compared to traditional ECS. Given long-standing concerns about ecological validity of ECS mapping of cued language tasks, and difficulties encountered with its use in children, ECoG mapping of spontaneous language may provide a valid alternative for clinical use.

Control of spoken vowel acoustics and the influence of phonetic context in human speech sensorimotor cortex

Speech production requires the precise control of vocal tract movements to generate individual speech sounds (phonemes) which, in turn, are rapidly organized into complex sequences. Multiple productions of the same phoneme can exhibit substantial variability, some of which is inherent to control of the vocal tract and its biomechanics, and some of which reflects the contextual effects of surrounding phonemes ("coarticulation"). The role of the CNS in these aspects of speech motor control is not well understood. To address these issues, we recorded multielectrode cortical activity directly from human ventral sensory-motor cortex (vSMC) during the production of consonant-vowel syllables. We analyzed the relationship between the acoustic parameters of vowels (pitch and formants) and cortical activity on a single-trial level. We found that vSMC activity robustly predicted acoustic parameters across vowel categories (up to 80% of variance), as well as different renditions of the same vowel (up to 25% of variance). Furthermore, we observed significant contextual effects on vSMC representations of produced phonemes that suggest active control of coarticulation: vSMC representations for vowels were biased toward the representations of the preceding consonant, and conversely, representations for consonants were biased toward upcoming vowels. These results reveal that vSMC activity for phonemes are not invariant and provide insight into the cortical mechanisms of coarticulation.

The neural correlates of speech motor sequence learning

Speech is perhaps the most sophisticated example of a species-wide movement capability in the animal kingdom, requiring split-second sequencing of approximately 100 muscles in the respiratory, laryngeal, and oral movement systems. Despite the unique role speech plays in human interaction and the debilitating impact of its disruption, little is known about the neural mechanisms underlying speech motor learning. Here, we studied the behavioral and neural correlates of learning new speech motor sequences. Participants repeatedly produced novel, meaningless syllables comprising illegal consonant clusters (e.g., GVAZF) over 2 days of practice. Following practice, participants produced the sequences with fewer errors and shorter durations, indicative of motor learning. Using fMRI, we compared brain activity during production of the learned illegal sequences and novel illegal sequences. Greater activity was noted during production of novel sequences in brain regions linked to non-speech motor sequence learning, including the BG and pre-SMA. Activity during novel sequence production was also greater in brain regions associated with learning and maintaining speech motor programs, including lateral premotor cortex, frontal operculum, and posterior superior temporal cortex. Measures of learning success correlated positively with activity in left frontal operculum and white matter integrity under left posterior superior temporal sulcus. These findings indicate speech motor sequence learning relies not only on brain areas involved generally in motor sequencing learning but also those associated with feedback-based speech motor learning. Furthermore, learning success is modulated by the integrity of structural connectivity between these motor and sensory brain regions.

ECoG high gamma activity reveals distinct cortical representations of lyrics passages, harmonic and timbre-related changes in a rock song

Listening to music moves our minds and moods, stirring interest in its neural underpinnings. A multitude of compositional features drives the appeal of natural music. How such original music, where a composer's opus is not manipulated for experimental purposes, engages a listener's brain has not been studied until recently. Here, we report an in-depth analysis of two electrocorticographic (ECoG) data sets obtained over the left hemisphere in ten patients during presentation of either a rock song or a read-out narrative. First, the time courses of five acoustic features (intensity, presence/absence of vocals with lyrics, spectral centroid, harmonic change, and pulse clarity) were extracted from the audio tracks and found to be correlated with each other to varying degrees. In a second step, we uncovered the specific impact of each musical feature on ECoG high-gamma power (70-170 Hz) by calculating partial correlations to remove the influence of the other four features. In the music condition, the onset and offset of vocal lyrics in ongoing instrumental music was consistently identified within the group as the dominant driver for ECoG high-gamma power changes over temporal auditory areas, while concurrently subject-individual activation spots were identified for sound intensity, timbral, and harmonic features. The distinct cortical activations to vocal speech-related content embedded in instrumental music directly demonstrate that song integrated in instrumental music represents a distinct dimension in complex music. In contrast, in the speech condition, the full sound envelope was reflected in the high gamma response rather than the onset or offset of the vocal lyrics. This demonstrates how the contributions of stimulus features that modulate the brain response differ across the two examples of a full-length natural stimulus, which suggests a context-dependent feature selection in the processing of complex auditory stimuli.

Localizing ECoG electrodes on the cortical anatomy without post-implantation imaging

INTRODUCTION

Electrocorticographic (ECoG) grids are placed subdurally on the cortex in people undergoing cortical resection to delineate eloquent cortex. ECoG signals have high spatial and temporal resolution and thus can be valuable for neuroscientific research. The value of these data is highest when they can be related to the cortical anatomy. Existing methods that establish this relationship rely either on post-implantation imaging using computed tomography (CT), magnetic resonance imaging (MRI) or X-Rays, or on intra-operative photographs. For research purposes, it is desirable to localize ECoG electrodes on the brain anatomy even when post-operative imaging is not available or when intra-operative photographs do not readily identify anatomical landmarks.

METHODS

We developed a method to co-register ECoG electrodes to the underlying cortical anatomy using only a pre-operative MRI, a clinical neuronavigation device (such as BrainLab VectorVision), and fiducial markers. To validate our technique, we compared our results to data collected from six subjects who also had post-grid implantation imaging available. We compared the electrode coordinates obtained by our fiducial-based method to those obtained using existing methods, which are based on co-registering pre- and post-grid implantation images.

RESULTS

Our fiducial-based method agreed with the MRI-CT method to within an average of 8.24 mm (mean, median = 7.10 mm) across 6 subjects in 3 dimensions. It showed an average discrepancy of 2.7 mm when compared to the results of the intra-operative photograph method in a 2D coordinate system. As this method does not require post-operative imaging such as CTs, our technique should prove useful for research in intra-operative single-stage surgery scenarios. To demonstrate the use of our method, we applied our method during real-time mapping of eloquent cortex during a single-stage surgery. The results demonstrated that our method can be applied intra-operatively in the absence of post-operative imaging to acquire ECoG signals that can be valuable for neuroscientific investigations.

Direct classification of all American English phonemes using signals from functional speech motor cortex

OBJECTIVE

Although brain-computer interfaces (BCIs) can be used in several different ways to restore communication, communicative BCI has not approached the rate or efficiency of natural human speech. Electrocorticography (ECoG) has precise spatiotemporal resolution that enables recording of brain activity distributed over a wide area of cortex, such as during speech production. In this study, we sought to decode elements of speech production using ECoG.

APPROACH

We investigated words that contain the entire set of phonemes in the general American accent using ECoG with four subjects. Using a linear classifier, we evaluated the degree to which individual phonemes within each word could be correctly identified from cortical signal.

MAIN RESULTS

We classified phonemes with up to 36% accuracy when classifying all phonemes and up to 63% accuracy for a single phoneme. Further, misclassified phonemes follow articulation organization described in phonology literature, aiding classification of whole words. Precise temporal alignment to phoneme onset was crucial for classification success.

SIGNIFICANCE

We identified specific spatiotemporal features that aid classification, which could guide future applications. Word identification was equivalent to information transfer rates as high as 3.0 bits s(-1) (33.6 words min(-1)), supporting pursuit of speech articulation for BCI control.

Decoding spectrotemporal features of overt and covert speech from the human cortex

Auditory perception and auditory imagery have been shown to activate overlapping brain regions. We hypothesized that these phenomena also share a common underlying neural representation. To assess this, we used electrocorticography intracranial recordings from epileptic patients performing an out loud or a silent reading task. In these tasks, short stories scrolled across a video screen in two conditions: subjects read the same stories both aloud (overt) and silently (covert). In a control condition the subject remained in a resting state. We first built a high gamma (70–150 Hz) neural decoding model to reconstruct spectrotemporal auditory features of self-generated overt speech. We then evaluated whether this same model could reconstruct auditory speech features in the covert speech condition. Two speech models were tested: a spectrogram and a modulation-based feature space. For the overt condition, reconstruction accuracy was evaluated as the correlation between original and predicted speech features, and was significant in each subject (p < 10⁻⁵; paired two-sample t-test). For the covert speech condition, dynamic time warping was first used to realign the covert speech reconstruction with the corresponding original speech from the overt condition. Reconstruction accuracy was then evaluated as the correlation between original and reconstructed speech features. Covert reconstruction accuracy was compared to the accuracy obtained from reconstructions in the baseline control condition. Reconstruction accuracy for the covert condition was significantly better than for the control condition (p < 0.005; paired two-sample t-test). The superior temporal gyrus, pre- and post-central gyrus provided the highest reconstruction information. The relationship between overt and covert speech reconstruction depended on anatomy. These results provide evidence that auditory representations of covert speech can be reconstructed from models that are built from an overt speech data set, supporting a partially shared neural substrate.

Assessing dynamics, spatial scale, and uncertainty in task-related brain network analyses

The brain is a complex network of interconnected elements, whose interactions evolve dynamically in time to cooperatively perform specific functions. A common technique to probe these interactions involves multi-sensor recordings of brain activity during a repeated task. Many techniques exist to characterize the resulting task-related activity, including establishing functional networks, which represent the statistical associations between brain areas. Although functional network inference is commonly employed to analyze neural time series data, techniques to assess the uncertainty-both in the functional network edges and the corresponding aggregate measures of network topology-are lacking. To address this, we describe a statistically principled approach for computing uncertainty in functional networks and aggregate network measures in task-related data. The approach is based on a resampling procedure that utilizes the trial structure common in experimental recordings. We show in simulations that this approach successfully identifies functional networks and associated measures of confidence emergent during a task in a variety of scenarios, including dynamically evolving networks. In addition, we describe a principled technique for establishing functional networks based on predetermined regions of interest using canonical correlation. Doing so provides additional robustness to the functional network inference. Finally, we illustrate the use of these methods on example invasive brain voltage recordings collected during an overt speech task. The general strategy described here-appropriate for static and dynamic network inference and different statistical measures of coupling-permits the evaluation of confidence in network measures in a variety of settings common to neuroscience.

Neural decoding of single vowels during covert articulation using electrocorticography

The human brain has important abilities for manipulating phonemes, the basic building blocks of speech; these abilities represent phonological processing. Previous studies have shown change in the activation levels of broad cortical areas such as the premotor cortex, the inferior frontal gyrus, and the superior temporal gyrus during phonological processing. However, whether these areas actually convey signals to representations related to individual phonemes remains unclear. This study focused on single vowels and investigated cortical areas important for representing single vowels using electrocorticography (ECoG) during covert articulation. To identify such cortical areas, we used a neural decoding approach in which machine learning models identify vowels. A decoding model was trained on the ECoG signals from individual electrodes placed on the subjects' cortices. We then statistically evaluated whether each decoding model showed accurate identification of vowels, and we found cortical areas such as the premotor cortex and the superior temporal gyrus. These cortical areas were consistent with previous findings. On the other hand, no electrodes over Broca's area showed significant decoding accuracies. This was inconsistent with findings from a previous study showing that vowels within the phonemic sequence of words can be decoded using ECoG signals from Broca's area. Our results therefore suggest that Broca's area is involved in the processing of vowels within phonemic sequences, but not in the processing of single vowels.

Electrocorticographic correlates of overt articulation of 44 English phonemes: intracranial recording in children with focal epilepsy

OBJECTIVE

We determined the temporal-spatial patterns of electrocorticography (ECoG) signal modulation during overt articulation of 44 American English phonemes.

METHODS

We studied two children with focal epilepsy who underwent extraoperative ECoG recording. Using animation movies, we delineated 'when' and 'where' gamma- (70-110 Hz) and low-frequency-band activities (10-30 Hz) were modulated during self-paced articulation.

RESULTS

Regardless of the classes of phoneme articulated, gamma-augmentation initially involved a common site within the left inferior Rolandic area. Subsequently, gamma-augmentation and/or attenuation involved distinct sites within the left oral-sensorimotor area with a timing variable across phonemes. Finally, gamma-augmentation in a larynx-sensorimotor area took place uniformly at the onset of sound generation, and effectively distinguished voiced and voiceless phonemes. Gamma-attenuation involved the left inferior-frontal and superior-temporal regions simultaneously during articulation. Low-frequency band attenuation involved widespread regions including the frontal, temporal, and parietal regions.

CONCLUSIONS

Our preliminary results support the notion that articulation of distinct phonemes recruits specific sensorimotor activation and deactivation. Gamma attenuation in the left inferior-frontal and superior-temporal regions may reflect transient functional suppression in these cortical regions during automatic, self-paced vocalization of phonemes containing no semantic or syntactic information.

SIGNIFICANCE

Further studies are warranted to determine if measurement of event-related modulations of gamma-band activity, compared to that of the low-frequency-band, is more useful for decoding the underlying articulatory functions.

Using in vivo probabilistic tractography to reveal two segregated dorsal 'language-cognitive' pathways in the human brain

Primate studies have recently identified the dorsal stream as constituting multiple dissociable pathways associated with a range of specialized cognitive functions. To elucidate the nature and number of dorsal pathways in the human brain, the current study utilized in vivo probabilistic tractography to map the structural connectivity associated with subdivisions of the left supramarginal gyrus (SMG). The left SMG is a prominent region within the dorsal stream, which has recently been parcellated into five structurally-distinct regions which possess a dorsal-ventral (and rostral-caudal) organisation, postulated to reflect areas of functional specialisation. The connectivity patterns reveal a dissociation of the arcuate fasciculus into at least two segregated pathways connecting frontal-parietal-temporal regions. Specifically, the connectivity of the inferior SMG, implicated as an acoustic-motor speech interface, is carried by an inner/ventro-dorsal arc of fibres, whilst the pathways of the posterior superior SMG, implicated in object use and cognitive control, forms a parallel outer/dorso-dorsal crescent.