Decoding speech perception from single cell activity in humans

Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces

Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of eight neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.

Neural manifolds carry reactivation of phonetic representations during semantic processing

Traditional models of speech perception posit that neural activity encodes speech through a hierarchy of cognitive processes, from low-level representations of acoustic and phonetic features to high-level semantic encoding. Yet it remains unknown how neural representations are transformed across levels of the speech hierarchy. Here, we analyzed unique microelectrode array recordings of neuronal spiking activity from the human left anterior superior temporal gyrus, a brain region at the interface between phonetic and semantic speech processing, during a semantic categorization task and natural speech perception. We identified distinct neural manifolds for semantic and phonetic features, with a functional separation of the corresponding low-dimensional trajectories. Moreover, phonetic and semantic representations were encoded concurrently and reflected in power increases in the beta and low-gamma local field potentials, suggesting top-down predictive and bottom-up cumulative processes. Our results are the first to demonstrate mechanisms for hierarchical speech transformations that are specific to neuronal population dynamics.

Neural modulations in the auditory cortex during internal and external attention tasks: A single-patient intracranial recording study

Brain sensory processing is not passive, but is rather modulated by our internal state. Different research methods such as non-invasive imaging methods and intracranial recording of the local field potential (LFP) have been used to study to what extent sensory processing and the auditory cortex in particular are modulated by selective attention. However, at the level of the single- or multi-units the selective attention in humans has not been tested. In addition, most previous research on selective attention has explored externally-oriented attention, but attention can be also directed inward (i.e., internal attention), like spontaneous self-generated thoughts and mind-wandering. In the present study we had a rare opportunity to record multi-unit activity (MUA) in the auditory cortex of a patient. To complement, we also analyzed the LFP signal of the macro-contact in the auditory cortex. Our experiment consisted of two conditions with periodic beeping sounds. The participants were asked either to count the beeps (i.e., an "external attention" condition) or to recall the events of the previous day (i.e., an "internal attention" condition). We found that the four out of seven recorded units in the auditory cortex showed increased firing rates in "external attention" compared to "internal attention" condition. The beginning of this attentional modulation varied across multi-units between 30-50 msec and 130-150 msec from stimulus onset, a result that is compatible with an early selection view. The LFP evoked potential and induced high gamma activity both showed attentional modulation starting at about 70-80 msec. As the control, for the same experiment we recorded MUA activity in the amygdala and hippocampus of two additional patients. No major attentional modulation was found in the control regions. Overall, we believe that our results provide new empirical information and support for existing theoretical views on selective attention and spontaneous self-generated cognition.

Recruitment and Differential Firing Patterns of Single Units During Conditioning to a Tone in a Mute Locked-In Human

Single units that are not related to the desired task can become related to the task by conditioning their firing rates. We theorized that, during conditioning of firing rates to a tone, (a) unrelated single units would be recruited to the task; (b) the recruitment would depend on the phase of the task; (c) tones of different frequencies would produce different patterns of single unit recruitment. In our mute locked-in participant, we conditioned single units using tones of different frequencies emitted from a tone generator. The conditioning task had three phases: Listen to the tone for 20 s, then silently sing the tone for 10 s, with a prior control period of resting for 10 s. Twenty single units were recorded simultaneously while feedback of one of the twenty single units was made audible to the mute locked-in participant. The results indicate that (a) some of the non-audible single units were recruited during conditioning, (b) some were recruited differentially depending on the phase of the paradigm (listen, rest, or silent sing), and (c) single unit firing patterns were specific for different tone frequencies such that the tone could be recognized from the pattern of single unit firings. These data are important when conditioning single unit firings in brain-computer interfacing tasks because they provide evidence that increased numbers of previously unrelated single units can be incorporated into the task. This incorporation expands the bandwidth of the recorded single unit population and thus enhances the brain-computer interface. This is the first report of conditioning of single unit firings in a human participant with a brain to computer implant.

Lie to my face: An electromyography approach to the study of deceptive behavior

BACKGROUND

Deception is present in all walks of life, from social interactions to matters of homeland security. Nevertheless, reliable indicators of deceptive behavior in real-life scenarios remain elusive.

METHODS

By integrating electrophysiological and communicative approaches, we demonstrate a new and objective detection approach to identify participant-specific indicators of deceptive behavior in an interactive scenario of a two-person deception task. We recorded participants' facial muscle activity using novel dry screen-printed electrode arrays and applied machine-learning algorithms to identify lies based on brief facial responses.

RESULTS

With an average accuracy of 73%, we identified two groups of participants: Those who revealed their lies by activating their cheek muscles and those who activated their eyebrows. We found that the participants lied more often with time, with some switching their telltale muscle groups. Moreover, while the automated classifier, reported here, outperformed untrained human detectors, their performance was correlated, suggesting reliance on shared features.

CONCLUSIONS

Our findings demonstrate the feasibility of using wearable electrode arrays in detecting human lies in a social setting and set the stage for future research on individual differences in deception expression.

Machine learning algorithm for decoding multiple subthalamic spike trains for speech brain-machine interfaces

Objective. The goal of this study is to decode the electrical activity of single neurons in the human subthalamic nucleus (STN) to infer the speech features that a person articulated, heard or imagined. We also aim to evaluate the amount of subthalamic neurons required for high accuracy decoding suitable for real-life speech brain-machine interfaces (BMI).Approach. We intraoperatively recorded single-neuron activity in the STN of 21 neurosurgical patients with Parkinson's disease undergoing implantation of deep brain stimulator while patients produced, perceived or imagined the five monophthongal vowel sounds. Our decoder is based on machine learning algorithms that dynamically learn specific features of the speech-related firing patterns.Main results. In an extensive comparison of algorithms, our sparse decoder ('SpaDe'), based on sparse decomposition of the high dimensional neuronal feature space, outperformed the other algorithms in all three conditions: production, perception and imagery. For speech production, our algorithm, Spade, predicted all vowels correctly (accuracy: 100%; chance level: 20%). For perception accuracy was 96%, and for imagery: 88%. The accuracy of Spade showed a linear behavior in the amount of neurons for the perception data, and even faster for production or imagery.Significance. Our study demonstrates that the information encoded by single neurons in the STN about the production, perception and imagery of speech is suitable for high-accuracy decoding. It is therefore an important step towards BMIs for restoration of speech faculties that bears an enormous potential to alleviate the suffering of completely paralyzed ('locked-in') patients and allow them to communicate again with their environment. Moreover, our research indicates how many subthalamic neurons may be necessary to achieve each level of decoding accuracy, which is of supreme importance for a neurosurgeon planning the implantation of a speech BMI.

Impaired Timing of Speech-Related Neurons in the Subthalamic Nucleus of Parkinson Disease Patients Suffering Speech Disorders

BACKGROUND

Our previous study found degradation to subthalamic neuronal encoding of speech features in Parkinson disease (PD) patients suffering from speech disorders.

OBJECTIVE

To find how timing of speech-related neuronal firing changes in PD patients with speech disorders compared to PD patients without speech disorders.

METHODS

During the implantation of deep brain stimulator (DBS), we recorded the activity of single neurons in the subthalamic nucleus (STN) of 18 neurosurgical patients with PD while they articulated, listened to, or imagined articulation of 5 vowel sounds, each following a beep. We compared subthalamic activity of PD patients with (n = 10) vs without speech disorders.

RESULTS

In this comparison, patients with speech disorders had longer reaction times and shorter lengths of articulation. Their speech-related neuronal activity preceding speech onset (planning) was delayed relative to the beep, but the time between this activity and the emission of speech sound was similar. Notwithstanding, speech-related neuronal activity following the onset of speech (feedback) was delayed when computed relative to the onset. Only in these patients was the time lag of planning neurons significantly correlated with the reaction time. Neuronal activity in patients with speech disorders was delayed during imagined articulation of vowel sounds but earlier during speech perception.

CONCLUSION

Our findings indicate that longer reaction times in patients with speech disorders are due to STN or earlier activity of the speech control network. This is a first step in locating the source(s) of PD delays within this network and is therefore of utmost importance for future treatment of speech disorders.

Single-cell activity in human STG during perception of phonemes is organized according to manner of articulation

One of the central tasks of the human auditory system is to extract sound features from incoming acoustic signals that are most critical for speech perception. Specifically, phonological features and phonemes are the building blocks for more complex linguistic entities, such as syllables, words and sentences. Previous ECoG and EEG studies showed that various regions in the superior temporal gyrus (STG) exhibit selective responses to specific phonological features. However, electrical activity recorded by ECoG or EEG grids reflects average responses of large neuronal populations and is therefore limited in providing insights into activity patterns of single neurons. Here, we recorded spiking activity from 45 units in the STG from six neurosurgical patients who performed a listening task with phoneme stimuli. Fourteen units showed significant responsiveness to the stimuli. Using a Naïve-Bayes model, we find that single-cell responses to phonemes are governed by manner-of-articulation features and are organized according to sonority with two main clusters for sonorants and obstruents. We further find that 'neural similarity' (i.e. the similarity of evoked spiking activity between pairs of phonemes) is comparable to the 'perceptual similarity' (i.e. to what extent two phonemes are judged as sounding similar) based on perceptual confusion, assessed behaviorally in healthy subjects. Thus, phonemes that were perceptually similar also had similar neural responses. Taken together, our findings indicate that manner-of-articulation is the dominant organization dimension of phoneme representations at the single-cell level, suggesting a remarkable consistency across levels of analyses, from the single neuron level to that of large neuronal populations and behavior.

A state-based probabilistic method for decoding hand position during movement from ECoG signals in non-human primate

OBJECTIVE

In this study, we proposed a state-based probabilistic method for decoding hand positions during unilateral and bilateral movements using the ECoG signals recorded from the brain of Rhesus monkey.

APPROACH

A customized electrode array was implanted subdurally in the right hemisphere of the brain covering from the primary motor cortex to the frontal cortex. Three different experimental paradigms were considered: ipsilateral, contralateral, and bilateral movements. During unilateral movement, the monkey was trained to get food with one hand, while during bilateral movement, the monkey used its left and right hands alternately to get food. To estimate the hand positions, a state-based probabilistic method was introduced which was based on the conditional probability of the hand movement state (i.e. idle, right hand movement, and left hand movement) and the conditional expectation of the hand position for each state. Moreover, a hybrid feature extraction method based on linear discriminant analysis and partial least squares (PLS) was introduced.

MAIN RESULTS

The proposed method could successfully decode the hand positions during ipsilateral, contralateral, and bilateral movements and significantly improved the decoding performance compared to the conventional Kalman and PLS regression methods [Formula: see text]. The proposed hybrid feature extraction method was found to outperform both the PLS and PCA methods [Formula: see text]. Investigating the kinematic information of each frequency band shows that more informative frequency bands were [Formula: see text] (15-30 Hz) and [Formula: see text](50-100 Hz) for ipsilateral and [Formula: see text] and [Formula: see text] (100-200 Hz) for contralateral movements. It is observed that ipsilateral movement was decoded better than contralateral movement for [Formula: see text] (5-15 Hz) and [Formula: see text] bands, while contralateral movements was decoded better for [Formula: see text] (30-200 Hz) and hfECoG (200-400 Hz) bands.

SIGNIFICANCE

Accurate decoding the bilateral movement using the ECoG recorded from one brain hemisphere is an important issue toward real-life applications of the brain-machine interface technologies.

Degradation of Neuronal Encoding of Speech in the Subthalamic Nucleus in Parkinson's Disease

BACKGROUND

Most of the patients with Parkinson's disease suffer from speech disorders characterized mainly by dysarthria and hypophonia.

OBJECTIVE

To understand the deterioration of speech in the course of Parkinson's disease.

METHODS

We intraoperatively recorded single neuron activity in the subthalamic nucleus of 18 neurosurgical patients with Parkinson's disease undergoing implantation of deep brain stimulator while patients articulated 5 vowel sounds.

RESULTS

Here, we report that single subthalamic neurons encode individual vowel phonemes and employ 1 of 2 encoding schemes: broad or sharp tuning. Broadly tuned units respond to all examined phonemes, each with a different firing rate, whereas sharply tuned ones are specific to 1 to 2 phonemes. We then show that in comparison with patients without speech deficits, the spiking activity in patients with speech disorders was lower during speech production, overt or imagined, but not during perception. However, patients with speech disorders employed a larger percentage of the neurons for the aforementioned tasks. Whereas the lower firing rates affect mainly sharply tuned units, the extra units used a broad tuning encoding scheme.

CONCLUSION

Our findings suggest mechanisms of neuronal degradation due to Parkinsonian speech disorders and their possible compensation. As impairment in sharply tuned units may be compensated by broadly tuned ones, the proposed compensation model appears to be suboptimal, lending support to the persistence of speech disorders in the course of the disease.

Decoding Speech With Integrated Hybrid Signals Recorded From the Human Ventral Motor Cortex

Restoration of speech communication for locked-in patients by means of brain computer interfaces (BCIs) is currently an important area of active research. Among the neural signals obtained from intracranial recordings, single/multi-unit activity (SUA/MUA), local field potential (LFP), and electrocorticography (ECoG) are good candidates for an input signal for BCIs. However, the question of which signal or which combination of the three signal modalities is best suited for decoding speech production remains unverified. In order to record SUA, LFP, and ECoG simultaneously from a highly localized area of human ventral sensorimotor cortex (vSMC), we fabricated an electrode the size of which was 7 by 13 mm containing sparsely arranged microneedle and conventional macro contacts. We determined which signal modality is the most capable of decoding speech production, and tested if the combination of these signals could improve the decoding accuracy of spoken phonemes. Feature vectors were constructed from spike frequency obtained from SUAs and event-related spectral perturbation derived from ECoG and LFP signals, then input to the decoder. The results showed that the decoding accuracy for five spoken vowels was highest when features from multiple signals were combined and optimized for each subject, and reached 59% when averaged across all six subjects. This result suggests that multi-scale signals convey complementary information for speech articulation. The current study demonstrated that simultaneous recording of multi-scale neuronal activities could raise decoding accuracy even though the recording area is limited to a small portion of cortex, which is advantageous for future implementation of speech-assisting BCIs.

A piecewise probabilistic regression model to decode hand movement trajectories from epidural and subdural ECoG signals

OBJECTIVE

The primary concern of this study is to develop a probabilistic regression method that would improve the decoding of the hand movement trajectories from epidural ECoG as well as from subdural ECoG signals.

APPROACH

The model is characterized by the conditional expectation of the hand position given the ECoG signals. The conditional expectation of the hand position is then modeled by a linear combination of the conditional probability density functions defined for each segment of the movement. Moreover, a spatial linear filter is proposed for reducing the dimension of the feature space. The spatial linear filter is applied to each frequency band of the ECoG signals and extract the features with highest decoding performance.

MAIN RESULTS

For evaluating the proposed method, a dataset including 28 ECoG recordings from four adult Japanese macaques is used. The results show that the proposed decoding method outperforms the results with respect to the state of the art methods using this dataset. The relative kinematic information of each frequency band is also investigated using mutual information and decoding performance. The decoding performance shows that the best performance was obtained for high gamma bands from 50 to 200 Hz as well as high frequency ECoG band from 200 to 400 Hz for subdural recordings. However, the decoding performance was decreased for these frequency bands using epidural recordings. The mutual information shows that, on average, the high gamma band from 50 to 200 Hz and high frequency ECoG band from 200 to 400 Hz contain significantly more information than the average of the rest of the frequency bands [Formula: see text] for both subdural and epidural recordings. The results of high resolution time-frequency analysis show that ERD/ERS patterns in all frequency bands could reveal the dynamics of the ECoG responses during the movement. The onset and offset of the movement can be clearly identified by the ERD/ERS patterns.

SIGNIFICANCE

Reliable decoding the kinematic information from the brain signals paves the way for robust control of external devices.

Stimulus Phase Locking of Cortical Oscillations for Rhythmic Tone Sequences in Rats

Humans can rapidly detect regular patterns (i.e., within few cycles) without any special attention to the acoustic environment. This suggests that human sensory systems are equipped with a powerful mechanism for automatically predicting forthcoming stimuli to detect regularity. It has recently been hypothesized that the neural basis of sensory predictions exists for not only what happens (predictive coding) but also when a particular stimulus occurs (predictive timing). Here, we hypothesize that the phases of neural oscillations are critical in predictive timing, and these oscillations are modulated in a band-specific manner when acoustic patterns become predictable, i.e., regular. A high-density microelectrode array (10 × 10 within 4 × 4 mm²) was used to characterize spatial patterns of band-specific oscillations when a random-tone sequence was switched to a regular-tone sequence. Increasing the regularity of the tone sequence enhanced phase locking in a band-specific manner, notwithstanding the type of the regular sound pattern. Gamma-band phase locking increased immediately after the transition from random to regular sequences, while beta-band phase locking gradually evolved with time after the transition. The amplitude of the tone-evoked response, in contrast, increased with frequency separation with respect to the prior tone, suggesting that the evoked-response amplitude encodes sequence information on a local scale, i.e., the local order of tones. The phase locking modulation spread widely over the auditory cortex, while the amplitude modulation was confined around the activation foci. Thus, our data suggest that oscillatory phase plays a more important role than amplitude in the neuronal detection of tone sequence regularity, which is closely related to predictive timing. Furthermore, band-specific contributions may support recent theories that gamma oscillations encode bottom-up prediction errors, whereas beta oscillations are involved in top-down prediction.

Intracranial Electrophysiology of Auditory Selective Attention Associated with Speech Classification Tasks

Auditory selective attention paradigms are powerful tools for elucidating the various stages of speech processing. This study examined electrocorticographic activation during target detection tasks within and beyond auditory cortex. Subjects were nine neurosurgical patients undergoing chronic invasive monitoring for treatment of medically refractory epilepsy. Four subjects had left hemisphere electrode coverage, four had right coverage and one had bilateral coverage. Stimuli were 300 ms complex tones or monosyllabic words, each spoken by a different male or female talker. Subjects were instructed to press a button whenever they heard a target corresponding to a specific stimulus category (e.g., tones, animals, numbers). High gamma (70–150 Hz) activity was simultaneously recorded from Heschl’s gyrus (HG), superior, middle temporal and supramarginal gyri (STG, MTG, SMG), as well as prefrontal cortex (PFC). Data analysis focused on: (1) task effects (non-target words in tone detection vs. semantic categorization task); and (2) target effects (words as target vs. non-target during semantic classification). Responses within posteromedial HG (auditory core cortex) were minimally modulated by task and target. Non-core auditory cortex (anterolateral HG and lateral STG) exhibited sensitivity to task, with a smaller proportion of sites showing target effects. Auditory-related areas (MTG and SMG) and PFC showed both target and, to a lesser extent, task effects, that occurred later than those in the auditory cortex. Significant task and target effects were more prominent in the left hemisphere than in the right. Findings demonstrate a hierarchical organization of speech processing during auditory selective attention.

Vowel Imagery Decoding toward Silent Speech BCI Using Extreme Learning Machine with Electroencephalogram

The purpose of this study is to classify EEG data on imagined speech in a single trial. We recorded EEG data while five subjects imagined different vowels, /a/, /e/, /i/, /o/, and /u/. We divided each single trial dataset into thirty segments and extracted features (mean, variance, standard deviation, and skewness) from all segments. To reduce the dimension of the feature vector, we applied a feature selection algorithm based on the sparse regression model. These features were classified using a support vector machine with a radial basis function kernel, an extreme learning machine, and two variants of an extreme learning machine with different kernels. Because each single trial consisted of thirty segments, our algorithm decided the label of the single trial by selecting the most frequent output among the outputs of the thirty segments. As a result, we observed that the extreme learning machine and its variants achieved better classification rates than the support vector machine with a radial basis function kernel and linear discrimination analysis. Thus, our results suggested that EEG responses to imagined speech could be successfully classified in a single trial using an extreme learning machine with a radial basis function and linear kernel. This study with classification of imagined speech might contribute to the development of silent speech BCI systems.