"Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech

Generative embedding for model-based classification of fMRI data

Decoding models, such as those underlying multivariate classification algorithms, have been increasingly used to infer cognitive or clinical brain states from measures of brain activity obtained by functional magnetic resonance imaging (fMRI). The practicality of current classifiers, however, is restricted by two major challenges. First, due to the high data dimensionality and low sample size, algorithms struggle to separate informative from uninformative features, resulting in poor generalization performance. Second, popular discriminative methods such as support vector machines (SVMs) rarely afford mechanistic interpretability. In this paper, we address these issues by proposing a novel generative-embedding approach that incorporates neurobiologically interpretable generative models into discriminative classifiers. Our approach extends previous work on trial-by-trial classification for electrophysiological recordings to subject-by-subject classification for fMRI and offers two key advantages over conventional methods: it may provide more accurate predictions by exploiting discriminative information encoded in ‘hidden’ physiological quantities such as synaptic connection strengths; and it affords mechanistic interpretability of clinical classifications. Here, we introduce generative embedding for fMRI using a combination of dynamic causal models (DCMs) and SVMs. We propose a general procedure of DCM-based generative embedding for subject-wise classification, provide a concrete implementation, and suggest good-practice guidelines for unbiased application of generative embedding in the context of fMRI. We illustrate the utility of our approach by a clinical example in which we classify moderately aphasic patients and healthy controls using a DCM of thalamo-temporal regions during speech processing. Generative embedding achieves a near-perfect balanced classification accuracy of 98% and significantly outperforms conventional activation-based and correlation-based methods. This example demonstrates how disease states can be detected with very high accuracy and, at the same time, be interpreted mechanistically in terms of abnormalities in connectivity. We envisage that future applications of generative embedding may provide crucial advances in dissecting spectrum disorders into physiologically more well-defined subgroups.

Neurological and psychiatric spectrum disorders are typically defined in terms of particular symptom sets, despite increasing evidence that the same symptom may be caused by very different pathologies. Pathophysiological classification and effective treatment of such disorders will increasingly require a mechanistic understanding of inter-individual differences and clinical tools for making accurate diagnostic inference in individual patients. Previous classification studies have shown that functional magnetic resonance imaging (fMRI) can be used to differentiate between healthy controls and neurological or psychiatric patients. However, these studies are typically based on descriptive patterns and indirect measures of neural activity, and they rarely afford mechanistic insights into the underlying condition. In this paper, we address this challenge by proposing a classification approach that rests on a model of brain function and exploits the rich discriminative information encoded in directed interregional connection strengths. Based on an fMRI dataset acquired from moderately aphasic patients and healthy controls, we illustrate that our approach enables more accurate classification and deeper mechanistic insights about disease processes than conventional classification methods.

Model-based feature construction for multivariate decoding

Conventional decoding methods in neuroscience aim to predict discrete brain states from multivariate correlates of neural activity. This approach faces two important challenges. First, a small number of examples are typically represented by a much larger number of features, making it hard to select the few informative features that allow for accurate predictions. Second, accuracy estimates and information maps often remain descriptive and can be hard to interpret. In this paper, we propose a model-based decoding approach that addresses both challenges from a new angle. Our method involves (i) inverting a dynamic causal model of neurophysiological data in a trial-by-trial fashion; (ii) training and testing a discriminative classifier on a strongly reduced feature space derived from trial-wise estimates of the model parameters; and (iii) reconstructing the separating hyperplane. Since the approach is model-based, it provides a principled dimensionality reduction of the feature space; in addition, if the model is neurobiologically plausible, decoding results may offer a mechanistically meaningful interpretation. The proposed method can be used in conjunction with a variety of modelling approaches and brain data, and supports decoding of either trial or subject labels. Moreover, it can supplement evidence-based approaches for model-based decoding and enable structural model selection in cases where Bayesian model selection cannot be applied. Here, we illustrate its application using dynamic causal modelling (DCM) of electrophysiological recordings in rodents. We demonstrate that the approach achieves significant above-chance performance and, at the same time, allows for a neurobiological interpretation of the results.

What's behind a face: person context coding in fusiform face area as revealed by multivoxel pattern analysis

The identification of a face comprises processing of both visual features and conceptual knowledge. Studies showing that the fusiform face area (FFA) is sensitive to face identity generally neglect this dissociation. The present study is the first that isolates conceptual face processing by using words presented in a person context instead of faces. The design consisted of 2 different conditions. In one condition, participants were presented with blocks of words related to each other at the categorical level (e.g., brands of cars, European cities). The second condition consisted of blocks of words linked to the personality features of a specific face. Both conditions were created from the same 8 × 8 word matrix, thereby controlling for visual input across conditions. Univariate statistical contrasts did not yield any significant differences between the 2 conditions in FFA. However, a machine learning classification algorithm was able to successfully learn the functional relationship between the 2 contexts and their underlying response patterns in FFA, suggesting that these activation patterns can code for different semantic contexts. These results suggest that the level of processing in FFA goes beyond facial features. This has strong implications for the debate about the role of FFA in face identification.

Pattern classification of working memory networks reveals differential effects of methylphenidate, atomoxetine, and placebo in healthy volunteers

Stimulant and non-stimulant drugs can reduce symptoms of attention deficit/hyperactivity disorder (ADHD). The stimulant drug methylphenidate (MPH) and the non-stimulant drug atomoxetine (ATX) are both widely used for ADHD treatment, but their differential effects on human brain function remain unclear. We combined event-related fMRI with multivariate pattern recognition to characterize the effects of MPH and ATX in healthy volunteers performing a rewarded working memory (WM) task. The effects of MPH and ATX on WM were strongly dependent on their behavioral context. During non-rewarded trials, only MPH could be discriminated from placebo (PLC), with MPH producing a similar activation pattern to reward. During rewarded trials both drugs produced the opposite effect to reward, that is, attenuating WM networks and enhancing task-related deactivations (TRDs) in regions consistent with the default mode network (DMN). The drugs could be directly discriminated during the delay component of rewarded trials: MPH produced greater activity in WM networks and ATX produced greater activity in the DMN. Our data provide evidence that: (1) MPH and ATX have prominent effects during rewarded WM in task-activated and -deactivated networks; (2) during the delay component of rewarded trials, MPH and ATX have opposing effects on activated and deactivated networks: MPH enhances TRDs more than ATX, whereas ATX attenuates WM networks more than MPH; and (3) MPH mimics reward during encoding. Thus, interactions between drug effects and motivational state are crucial in defining the effects of MPH and ATX.

You had me at "Hello": Rapid extraction of dialect information from spoken words

Research on the neuronal underpinnings of speaker identity recognition has identified voice-selective areas in the human brain with evolutionary homologues in non-human primates who have comparable areas for processing species-specific calls. Most studies have focused on estimating the extent and location of these areas. In contrast, relatively few experiments have investigated the time-course of speaker identity, and in particular, dialect processing and identification by electro- or neuromagnetic means. We show here that dialect extraction occurs speaker-independently, pre-attentively and categorically. We used Standard American English and African-American English exemplars of 'Hello' in a magnetoencephalographic (MEG) Mismatch Negativity (MMN) experiment. The MMN as an automatic change detection response of the brain reflected dialect differences that were not entirely reducible to acoustic differences between the pronunciations of 'Hello'. Source analyses of the M100, an auditory evoked response to the vowels suggested additional processing in voice-selective areas whenever a dialect change was detected. These findings are not only relevant for the cognitive neuroscience of language, but also for the social sciences concerned with dialect and race perception.

Auditory cortex encodes the perceptual interpretation of ambiguous sound

The confounding of physical stimulus characteristics and perceptual interpretations of stimuli poses a problem for most neuroscientific studies of perception. In the auditory domain, this pertains to the entanglement of acoustics and percept. Traditionally, most study designs have relied on cognitive subtraction logic, which demands the use of one or more comparisons between stimulus types. This does not allow for a differentiation between effects due to acoustic differences (i.e., sensation) and those due to conscious perception. To overcome this problem, we used functional magnetic resonance imaging (fMRI) in humans and pattern-recognition analysis to identify activation patterns that encode the perceptual interpretation of physically identical, ambiguous sounds. We show that it is possible to retrieve the perceptual interpretation of ambiguous phonemes-information that is fully subjective to the listener-from fMRI measurements of brain activity in auditory areas in the superior temporal cortex, most prominently on the posterior bank of the left Heschl's gyrus and sulcus and in the adjoining left planum temporale. These findings suggest that, beyond the basic acoustic analysis of sounds, constructive perceptual processes take place in these relatively early cortical auditory networks. This disagrees with hierarchical models of auditory processing, which generally conceive of these areas as sets of feature detectors, whose task is restricted to the analysis of physical characteristics and the structure of sounds.

Communication and the primate brain: insights from neuroimaging studies in humans, chimpanzees and macaques

Considerable knowledge is available on the neural substrates for speech and language from brain-imaging studies in humans, but until recently there was a lack of data for comparison from other animal species on the evolutionarily conserved brain regions that process species-specific communication signals. To obtain new insights into the relationship of the substrates for communication in primates, we compared the results from several neuroimaging studies in humans with those that have recently been obtained from macaque monkeys and chimpanzees. The recent work in humans challenges the longstanding notion of highly localized speech areas. As a result, the brain regions that have been identified in humans for speech and nonlinguistic voice processing show a striking general correspondence to how the brains of other primates analyze species-specific vocalizations or information in the voice, such as voice identity. The comparative neuroimaging work has begun to clarify evolutionary relationships in brain function, supporting the notion that the brain regions that process communication signals in the human brain arose from a precursor network of regions that is present in nonhuman primates and is used for processing species-specific vocalizations. We conclude by considering how the stage now seems to be set for comparative neurobiology to characterize the ancestral state of the network that evolved in humans to support language.

Intracranial study of speech-elicited activity on the human posterolateral superior temporal gyrus

To clarify speech-elicited response patterns within auditory-responsive cortex of the posterolateral superior temporal (PLST) gyrus, time-frequency analyses of event-related band power in the high gamma frequency range (75-175 Hz) were performed on the electrocorticograms recorded from high-density subdural grid electrodes in 8 patients undergoing evaluation for medically intractable epilepsy. Stimuli were 6 stop consonant-vowel (CV) syllables that varied in their consonant place of articulation (POA) and voice onset time (VOT). Initial augmentation was maximal over several centimeters of PLST, lasted about 400 ms, and was often followed by suppression and a local outward expansion of activation. Maximal gamma power overlapped either the Nα or Pβ deflections of the average evoked potential (AEP). Correlations were observed between the relative magnitudes of gamma band responses elicited by unvoiced stop CV syllables (/pa/, /ka/, /ta/) and their corresponding voiced stop CV syllables (/ba/, /ga/, /da/), as well as by the VOT of the stimuli. VOT was also represented in the temporal patterns of the AEP. These findings, obtained in the passive awake state, indicate that PLST discriminates acoustic features associated with POA and VOT and serve as a benchmark upon which task-related speech activity can be compared.

Distributed processing and cortical specialization for speech and environmental sounds in human temporal cortex

Using functional MRI, we investigated whether auditory processing of both speech and meaningful non-linguistic environmental sounds in superior and middle temporal cortex relies on a complex and spatially distributed neural system. We found that evidence for spatially distributed processing of speech and environmental sounds in a substantial extent of temporal cortices. Most importantly, regions previously reported as selective for speech over environmental sounds also contained distributed information. The results indicate that temporal cortices supporting complex auditory processing, including regions previously described as speech-selective, are in fact highly heterogeneous.

Distributed representation of tone frequency in highly decodable spatio-temporal activity in the auditory cortex

Although the place code of tone frequency, or tonotopic map, has been widely accepted in the auditory cortex, tone-evoked activation becomes less frequency-specific at moderate or high sound pressure levels. This implies that sound frequency is not represented by a simple place code but that the information is distributed spatio-temporally irrespective of the focal activation. In this study, using a decoding-based analysis, we investigated multi-unit activities in the auditory cortices of anesthetized rats to elucidate how a tone frequency is represented in the spatio-temporal neural pattern. We attempted sequential dimensionality reduction (SDR), a specific implementation of recursive feature elimination (RFE) with support vector machine (SVM), to identify the optimal spatio-temporal window patterns for decoding test frequency. SDR selected approximately a quarter of the windows, and SDR-identified window patterns led to significantly better decoding than spatial patterns, in which temporal structures were eliminated, or high-spike-rate patterns, in which windows with high spike rates were selectively extracted. Thus, the test frequency is also encoded in temporal as well as spatial structures of neural activities and low-spike-rate windows. Yet, SDR recruited more high-spike-rate windows than low-spike-rate windows, resulting in a highly dispersive pattern that probably offers an advantage of discrimination ability. Further investigation of SVM weights suggested that low-spike-rate windows play significant roles in fine frequency differentiation. These findings support the hypothesis that the auditory cortex adopts a distributed code in tone frequency representation, in which high- and low-spike-rate activities play mutually complementary roles.

Segregation of vowels and consonants in human auditory cortex: evidence for distributed hierarchical organization

The speech signal consists of a continuous stream of consonants and vowels, which must be de- and encoded in human auditory cortex to ensure the robust recognition and categorization of speech sounds. We used small-voxel functional magnetic resonance imaging to study information encoded in local brain activation patterns elicited by consonant-vowel syllables, and by a control set of noise bursts. First, activation of anterior–lateral superior temporal cortex was seen when controlling for unspecific acoustic processing (syllables versus band-passed noises, in a “classic” subtraction-based design). Second, a classifier algorithm, which was trained and tested iteratively on data from all subjects to discriminate local brain activation patterns, yielded separations of cortical patches discriminative of vowel category versus patches discriminative of stop-consonant category across the entire superior temporal cortex, yet with regional differences in average classification accuracy. Overlap (voxels correctly classifying both speech sound categories) was surprisingly sparse. Third, lending further plausibility to the results, classification of speech–noise differences was generally superior to speech–speech classifications, with the no\ exception of a left anterior region, where speech–speech classification accuracies were significantly better. These data demonstrate that acoustic–phonetic features are encoded in complex yet sparsely overlapping local patterns of neural activity distributed hierarchically across different regions of the auditory cortex. The redundancy apparent in these multiple patterns may partly explain the robustness of phonemic representations.

Visualization of nonlinear kernel models in neuroimaging by sensitivity maps

There is significant current interest in decoding mental states from neuroimages. In this context kernel methods, e.g., support vector machines (SVM) are frequently adopted to learn statistical relations between patterns of brain activation and experimental conditions. In this paper we focus on visualization of such nonlinear kernel models. Specifically, we investigate the sensitivity map as a technique for generation of global summary maps of kernel classification models. We illustrate the performance of the sensitivity map on functional magnetic resonance (fMRI) data based on visual stimuli. We show that the performance of linear models is reduced for certain scan labelings/categorizations in this data set, while the nonlinear models provide more flexibility. We show that the sensitivity map can be used to visualize nonlinear versions of kernel logistic regression, the kernel Fisher discriminant, and the SVM, and conclude that the sensitivity map is a versatile and computationally efficient tool for visualization of nonlinear kernel models in neuroimaging.

Nonstimulated early visual areas carry information about surrounding context

Even within the early sensory areas, the majority of the input to any given cortical neuron comes from other cortical neurons. To extend our knowledge of the contextual information that is transmitted by such lateral and feedback connections, we investigated how visually nonstimulated regions in primary visual cortex (V1) and visual area V2 are influenced by the surrounding context. We used functional magnetic resonance imaging (fMRI) and pattern-classification methods to show that the cortical representation of a nonstimulated quarter-field carries information that can discriminate the surrounding visual context. We show further that the activity patterns in these regions are significantly related to those observed with feed-forward stimulation and that these effects are driven primarily by V1. These results thus demonstrate that visual context strongly influences early visual areas even in the absence of differential feed-forward thalamic stimulation.

Neuroanatomical differences in toddler boys with fragile x syndrome and idiopathic autism

CONTEXT

Autism is an etiologically heterogeneous neurodevelopmental disorder for which there is no known unifying etiology or pathogenesis. Many conditions of atypical development can lead to autism, including fragile X syndrome (FXS), which is presently the most common known single-gene cause of autism.

OBJECTIVE

To examine whole-brain morphometric patterns that discriminate young boys with FXS from those with idiopathic autism (iAUT) as well as control participants.

DESIGN

Cross-sectional, in vivo neuroimaging study.

SETTING

Academic medical centers.

PATIENTS

Young boys (n = 165; aged 1.57-4.15 years) diagnosed as having FXS or iAUT as well as typically developing and idiopathic developmentally delayed controls.

MAIN OUTCOME MEASURES

Univariate voxel-based morphometric analyses, voxel-based morphometric multivariate pattern classification (linear support vector machine), and clustering analyses (self-organizing map).

RESULTS

We found that frontal and temporal gray and white matter regions often implicated in social cognition, including the medial prefrontal cortex, orbitofrontal cortex, superior temporal region, temporal pole, amygdala, insula, and dorsal cingulum, were aberrant in FXS and iAUT as compared with controls. However, these differences were in opposite directions for FXS and iAUT relative to controls; in general, greater volume was seen in iAUT compared with controls, who in turn had greater volume than FXS. Multivariate analysis showed that the overall pattern of brain structure in iAUT generally resembled that of the controls more than FXS, both with and without AUT.

CONCLUSIONS

Our findings demonstrate that FXS and iAUT are associated with distinct neuroanatomical patterns, further underscoring the neurobiological heterogeneity of iAUT.

Prediction of individual brain maturity using fMRI

Group functional connectivity magnetic resonance imaging (fcMRI) studies have documented reliable changes in human functional brain maturity over development. Here we show that support vector machine-based multivariate pattern analysis extracts sufficient information from fcMRI data to make accurate predictions about individuals' brain maturity across development. The use of only 5 minutes of resting-state fcMRI data from 238 scans of typically developing volunteers (ages 7 to 30 years) allowed prediction of individual brain maturity as a functional connectivity maturation index. The resultant functional maturation curve accounted for 55% of the sample variance and followed a nonlinear asymptotic growth curve shape. The greatest relative contribution to predicting individual brain maturity was made by the weakening of short-range functional connections between the adult brain's major functional networks.

Supramodal representations of perceived emotions in the human brain

Basic emotional states (such as anger, fear, and joy) can be similarly conveyed by the face, the body, and the voice. Are there human brain regions that represent these emotional mental states regardless of the sensory cues from which they are perceived? To address this question, in the present study participants evaluated the intensity of emotions perceived from face movements, body movements, or vocal intonations, while their brain activity was measured with functional magnetic resonance imaging (fMRI). Using multivoxel pattern analysis, we compared the similarity of response patterns across modalities to test for brain regions in which emotion-specific patterns in one modality (e.g., faces) could predict emotion-specific patterns in another modality (e.g., bodies). A whole-brain searchlight analysis revealed modality-independent but emotion category-specific activity patterns in medial prefrontal cortex (MPFC) and left superior temporal sulcus (STS). Multivoxel patterns in these regions contained information about the category of the perceived emotions (anger, disgust, fear, happiness, sadness) across all modality comparisons (face-body, face-voice, body-voice), and independently of the perceived intensity of the emotions. No systematic emotion-related differences were observed in the overall amplitude of activation in MPFC or STS. These results reveal supramodal representations of emotions in high-level brain areas previously implicated in affective processing, mental state attribution, and theory-of-mind. We suggest that MPFC and STS represent perceived emotions at an abstract, modality-independent level, and thus play a key role in the understanding and categorization of others' emotional mental states.

Encoding and decoding in fMRI

Over the past decade fMRI researchers have developed increasingly sensitive techniques for analyzing the information represented in BOLD activity. The most popular of these techniques is linear classification, a simple technique for decoding information about experimental stimuli or tasks from patterns of activity across an array of voxels. A more recent development is the voxel-based encoding model, which describes the information about the stimulus or task that is represented in the activity of single voxels. Encoding and decoding are complementary operations: encoding uses stimuli to predict activity while decoding uses activity to predict information about the stimuli. However, in practice these two operations are often confused, and their respective strengths and weaknesses have not been made clear. Here we use the concept of a linearizing feature space to clarify the relationship between encoding and decoding. We show that encoding and decoding operations can both be used to investigate some of the most common questions about how information is represented in the brain. However, focusing on encoding models offers two important advantages over decoding. First, an encoding model can in principle provide a complete functional description of a region of interest, while a decoding model can provide only a partial description. Second, while it is straightforward to derive an optimal decoding model from an encoding model it is much more difficult to derive an encoding model from a decoding model. We propose a systematic modeling approach that begins by estimating an encoding model for every voxel in a scan and ends by using the estimated encoding models to perform decoding.

A Monte Carlo method for locally multivariate brain mapping

Locally multivariate approaches to functional brain mapping offer a highly appealing complement to conventional statistics, but require restrictive region-of-interest hypotheses, or, in exhaustive search forms (such as the "searchlight" algorithm; Kriegeskorte et al., 2006), are excessively computer intensive. We therefore propose a non-restrictive, comparatively fast yet highly sensitive method based on Monte Carlo approximation principles where locally multivariate maps are computed by averaging across voxelwise condition-discriminative information obtained from repeated stochastic sampling of fixed-size search volumes. On simulated data containing discriminative regions of varying size and contrast-to-noise ratio (CNR), the Monte Carlo method reduced the required computer resources by as much as 75% compared to the searchlight with no reduction in mapping performance. Notably, the Monte Carlo mapping approach not only outperformed the general linear method (GLM), but also produced higher discriminative voxel detection scores than the searchlight irrespective of classifier (linear or nonlinear support vector machine), discriminative region size or CNR. The improved performance was explained by the information-average procedure, and the Monte Carlo approach yielded mapping sensitivities of a few percent lower than an information-average exhaustive search. Finally, we demonstrate the utility of the algorithm on whole-brain, multi-subject functional magnetic resonance imaging (fMRI) data from a tactile study, revealing that the central representation of gentle touch is spatially distributed in somatosensory, insular and visual regions.

Invariance of firing rate and field potential dynamics to stimulus modulation rate in human auditory cortex

The effect of stimulus modulation rate on the underlying neural activity in human auditory cortex is not clear. Human studies (using both invasive and noninvasive techniques) have demonstrated that at the population level, auditory cortex follows stimulus envelope. Here we examined the effect of stimulus modulation rate by using a rare opportunity to record both spiking activity and local field potentials (LFP) in auditory cortex of patients during repeated presentations of an audio-visual movie clip presented at normal, double, and quadruple speeds. Mean firing rate during evoked activity remained the same across speeds and the temporal response profile of firing rate modulations at increased stimulus speeds was a linearly scaled version of the response during slower speeds. Additionally, stimulus induced power modulation of local field potentials in the high gamma band (64-128 Hz) exhibited similar temporal scaling as the neuronal firing rate modulations. Our data confirm and extend previous studies in humans and anesthetized animals, supporting a model in which both firing rate, and high-gamma LFP power modulations in auditory cortex follow the temporal envelope of the stimulus across different modulation rates.

Auditory Sensitivity to Formant Ratios:Toward an Account of Vowel Normalization

A long-standing question in speech perception research is how do listeners extract linguistic content from a highly variable acoustic input. In the domain of vowel perception, formant ratios, or the calculation of relative bark differences between vowel formants, have been a sporadically proposed solution. We propose a novel formant ratio algorithm in which the first (F1) and second (F2) formants are compared against the third formant (F3). Results from two magnetoencephelographic (MEG) experiments are presented that suggest auditory cortex is sensitive to formant ratios. Our findings also demonstrate that the perceptual system shows heightened sensitivity to formant ratios for tokens located in more crowded regions of the vowel space. Additionally, we present statistical evidence that this algorithm eliminates speaker-dependent variation based on age and gender from vowel productions. We conclude that these results present an impetus to reconsider formant ratios as a legitimate mechanistic component in the solution to the problem of speaker normalization.

Localization of sublexical speech perception components

Models of speech perception are in general agreement with respect to the major cortical regions involved, but lack precision with regard to localization and lateralization of processing units. To refine these models we conducted two Activation Likelihood Estimation (ALE) meta-analyses of the neuroimaging literature on sublexical speech perception. Based on foci reported in 23 fMRI experiments, we identified significant activation likelihoods in left and right superior temporal cortex and the left posterior middle frontal gyrus. Sub-analyses examining phonetic and phonological processes revealed only left mid-posterior superior temporal sulcus activation likelihood. A lateralization analysis demonstrated temporal lobe left lateralization in terms of magnitude, extent, and consistency of activity. Experiments requiring explicit attention to phonology drove this lateralization. An ALE analysis of eight fMRI studies on categorical phoneme perception revealed significant activation likelihood in the left supramarginal gyrus and angular gyrus. These results are consistent with a speech processing network in which the bilateral superior temporal cortices perform acoustic analysis of speech and non-speech auditory stimuli, the left mid-posterior superior temporal sulcus performs phonetic and phonological analysis, and the left inferior parietal lobule is involved in detection of differences between phoneme categories. These results modify current speech perception models in three ways: (1) specifying the most likely locations of dorsal stream processing units, (2) clarifying that phonetic and phonological superior temporal sulcus processing is left lateralized and localized to the mid-posterior portion, and (3) suggesting that both the supramarginal gyrus and angular gyrus may be involved in phoneme discrimination.

Cortical representation of natural complex sounds: effects of acoustic features and auditory object category

How the brain processes complex sounds, like voices or musical instrument sounds, is currently not well understood. The features comprising the acoustic profiles of such sounds are thought to be represented by neurons responding to increasing degrees of complexity throughout auditory cortex, with complete auditory "objects" encoded by neurons (or small networks of neurons) in anterior superior temporal regions. Although specialized voice and speech-sound regions have been proposed, it is unclear how other types of complex natural sounds are processed within this object-processing pathway. Using functional magnetic resonance imaging, we sought to demonstrate spatially distinct patterns of category-selective activity in human auditory cortex, independent of semantic content and low-level acoustic features. Category-selective responses were identified in anterior superior temporal regions, consisting of clusters selective for musical instrument sounds and for human speech. An additional subregion was identified that was particularly selective for the acoustic-phonetic content of speech. In contrast, regions along the superior temporal plane closer to primary auditory cortex were not selective for stimulus category, responding instead to specific acoustic features embedded in natural sounds, such as spectral structure and temporal modulation. Our results support a hierarchical organization of the anteroventral auditory-processing stream, with the most anterior regions representing the complete acoustic signature of auditory objects.

Predicting visual stimuli on the basis of activity in auditory cortices

Using multivariate pattern analysis of functional magnetic resonance imaging data, we found that the subjective experience of sound, in the absence of auditory stimulation, was associated with content-specific activity in early auditory cortices in humans. As subjects viewed sound-implying, but silent, visual stimuli, activity in auditory cortex differentiated among sounds related to various animals, musical instruments and objects. These results support the idea that early sensory cortex activity reflects perceptual experience, rather than sensory stimulation alone.

fMRI-Guided transcranial magnetic stimulation reveals that the superior temporal sulcus is a cortical locus of the McGurk effect

A compelling example of auditory-visual multisensory integration is the McGurk effect, in which an auditory syllable is perceived very differently depending on whether it is accompanied by a visual movie of a speaker pronouncing the same syllable or a different, incongruent syllable. Anatomical and physiological studies in human and nonhuman primates have suggested that the superior temporal sulcus (STS) is involved in auditory-visual integration for both speech and nonspeech stimuli. We hypothesized that the STS plays a critical role in the creation of the McGurk percept. Because the location of multisensory integration in the STS varies from subject to subject, the location of auditory-visual speech processing in the STS was first identified in each subject with fMRI. Then, activity in this region of the STS was disrupted with single-pulse transcranial magnetic stimulation (TMS) as subjects rated their percept of McGurk and non-McGurk stimuli. Across three experiments, TMS of the STS significantly reduced the likelihood of the McGurk percept but did not interfere with perception of non-McGurk stimuli. TMS of the STS was effective at disrupting the McGurk effect only in a narrow temporal window from 100 ms before auditory syllable onset to 100 ms after onset, and TMS of a control location did not influence perception of McGurk or control stimuli. These results demonstrate that the STS plays a critical role in the McGurk effect and auditory-visual integration of speech.

Sparse logistic regression for whole-brain classification of fMRI data

Multivariate pattern recognition methods are increasingly being used to identify multiregional brain activity patterns that collectively discriminate one cognitive condition or experimental group from another, using fMRI data. The performance of these methods is often limited because the number of regions considered in the analysis of fMRI data is large compared to the number of observations (trials or participants). Existing methods that aim to tackle this dimensionality problem are less than optimal because they either over-fit the data or are computationally intractable. Here, we describe a novel method based on logistic regression using a combination of L1 and L2 norm regularization that more accurately estimates discriminative brain regions across multiple conditions or groups. The L1 norm, computed using a fast estimation procedure, ensures a fast, sparse and generalizable solution; the L2 norm ensures that correlated brain regions are included in the resulting solution, a critical aspect of fMRI data analysis often overlooked by existing methods. We first evaluate the performance of our method on simulated data and then examine its effectiveness in discriminating between well-matched music and speech stimuli. We also compared our procedures with other methods which use either L1-norm regularization alone or support vector machine-based feature elimination. On simulated data, our methods performed significantly better than existing methods across a wide range of contrast-to-noise ratios and feature prevalence rates. On experimental fMRI data, our methods were more effective in selectively isolating a distributed fronto-temporal network that distinguished between brain regions known to be involved in speech and music processing. These findings suggest that our method is not only computationally efficient, but it also achieves the twin objectives of identifying relevant discriminative brain regions and accurately classifying fMRI data.

Domain general change detection accounts for "dishabituation" effects in temporal-parietal regions in functional magnetic resonance imaging studies of speech perception

Functional magnetic resonance imaging (fMRI) studies of speech sound categorization often compare conditions in which a stimulus is presented repeatedly to conditions in which multiple stimuli are presented. This approach has established that a set of superior temporal and inferior parietal regions respond more strongly to conditions containing stimulus change. Here, we examine whether this contrast is driven by habituation to a repeating condition or by selective responding to change. Experiment 1 directly tests this by comparing the observed response to long trains of stimuli against a constructed hemodynamic response modeling the hypothesis that no habituation occurs. The results are consistent with the view that enhanced response to conditions involving phonemic variability reflect change detection. In a second experiment, the specificity of these responses to linguistically relevant stimulus variability was studied by including a condition in which the talker, rather than phonemic category, was variable from stimulus to stimulus. In this context, strong change detection responses were observed to changes in talker, but not to changes in phoneme category. The results prompt a reconsideration of two assumptions common to fMRI studies of speech sound categorization: they suggest that temporoparietal responses in passive paradigms such as those used here are better characterized as reflecting change detection than habituation, and that their apparent selectivity to speech sound categories may reflect a more general preference for variability in highly salient or behaviorally relevant stimulus dimensions.

Linking brain-wide multivoxel activation patterns to behaviour: Examples from language and math

A key goal of cognitive neuroscience is to find simple and direct connections between brain and behaviour. However, fMRI analysis typically involves choices between many possible options, with each choice potentially biasing any brain-behaviour correlations that emerge. Standard methods of fMRI analysis assess each voxel individually, but then face the problem of selection bias when combining those voxels into a region-of-interest, or ROI. Multivariate pattern-based fMRI analysis methods use classifiers to analyse multiple voxels together, but can also introduce selection bias via data-reduction steps as feature selection of voxels, pre-selecting activated regions, or principal components analysis. We show here that strong brain-behaviour links can be revealed without any voxel selection or data reduction, using just plain linear regression as a classifier applied to the whole brain at once, i.e. treating each entire brain volume as a single multi-voxel pattern. The brain-behaviour correlations emerged despite the fact that the classifier was not provided with any information at all about subjects' behaviour, but instead was given only the neural data and its condition-labels. Surprisingly, more powerful classifiers such as a linear SVM and regularised logistic regression produce very similar results. We discuss some possible reasons why the very simple brain-wide linear regression model is able to find correlations with behaviour that are as strong as those obtained on the one hand from a specific ROI and on the other hand from more complex classifiers. In a manner which is unencumbered by arbitrary choices, our approach offers a method for investigating connections between brain and behaviour which is simple, rigorous and direct.

Relating Population-Code Representations between Man, Monkey, and Computational Models

Perceptual and cognitive content is thought to be represented in the brain by patterns of activity across populations of neurons. In order to test whether a computational model can explain a given population code and whether corresponding codes in man and monkey convey the same information, we need to quantitatively relate population-code representations. Here I give a brief introduction to representational similarity analysis, a particular approach to this problem. A population code is characterized by a representational dissimilarity matrix (RDM), which contains a dissimilarity for each pair of activity patterns elicited by a given stimulus set. The RDM encapsulates which distinctions the representation emphasizes and which it deemphasizes. By analyzing correlations between RDMs we can test models and compare different species. Moreover, we can study how representations are transformed across stages of processing and how they relate to behavioral measures of object similarity. We use an example from object vision to illustrate the method's potential to bridge major divides that have hampered progress in systems neuroscience.

Multisensory functional magnetic resonance imaging: a future perspective

Advances in functional magnetic resonance imaging (fMRI) technology and analytic tools provide a powerful approach to unravel how the human brain combines the different sensory systems. In this perspective, we outline promising future directions of fMRI to make optimal use of its strengths in multisensory research, and to meet its weaker sides by combining it with other imaging modalities and computational modeling.

Phonological repetition-suppression in bilateral superior temporal sulci

Evidence has accumulated that posterior superior temporal sulcus (STS) is critically involved in phonological processing during speech perception, although there are conflicting accounts regarding the degree of lateralization. The current fMRI experiment aimed to identify phonological processing during speech perception through repetition-suppression effects. Repetition-suppression occurs when brain activity decreases from repetitive presentation of stimulus characteristics, in regions of cortex that process those characteristics. We manipulated the degree of phonological repetition among words in short lists to obtain systematic decreases in brain response, indicative of phonological processing. The fMRI experiment presented seventeen participants with recorded wordlists, of low, medium, or high phonological repetition, defined by how many phonemes were shared among words. Bilaterally, middle STS demonstrated activity differences consistent with our prediction of repetition-suppression, as responses decreased systematically with each increase in phonological repetition. Phonological repetition-suppression in bilateral STS converges with neuroimaging evidence for phonological processing, and word deafness resulting from bilateral superior temporal lesions.

The brain-computer interface cycle

Brain-computer interfaces (BCIs) have attracted much attention recently, triggered by new scientific progress in understanding brain function and by impressive applications. The aim of this review is to give an overview of the various steps in the BCI cycle, i.e., the loop from the measurement of brain activity, classification of data, feedback to the subject and the effect of feedback on brain activity. In this article we will review the critical steps of the BCI cycle, the present issues and state-of-the-art results. Moreover, we will develop a vision on how recently obtained results may contribute to new insights in neurocognition and, in particular, in the neural representation of perceived stimuli, intended actions and emotions. Now is the right time to explore what can be gained by embracing real-time, online BCI and by adding it to the set of experimental tools already available to the cognitive neuroscientist. We close by pointing out some unresolved issues and present our view on how BCI could become an important new tool for probing human cognition.

Evolution of brain-computer interfaces: going beyond classic motor physiology

The notion that a computer can decode brain signals to infer the intentions of a human and then enact those intentions directly through a machine is becoming a realistic technical possibility. These types of devices are known as brain-computer interfaces (BCIs). The evolution of these neuroprosthetic technologies could have significant implications for patients with motor disabilities by enhancing their ability to interact and communicate with their environment. The cortical physiology most investigated and used for device control has been brain signals from the primary motor cortex. To date, this classic motor physiology has been an effective substrate for demonstrating the potential efficacy of BCI-based control. However, emerging research now stands to further enhance our understanding of the cortical physiology underpinning human intent and provide further signals for more complex brain-derived control. In this review, the authors report the current status of BCIs and detail the emerging research trends that stand to augment clinical applications in the future.

Expertise with artificial nonspeech sounds recruits speech-sensitive cortical regions

Regions of the human temporal lobe show greater activation for speech than for other sounds. These differences may reflect intrinsically specialized domain-specific adaptations for processing speech, or they may be driven by the significant expertise we have in listening to the speech signal. To test the expertise hypothesis, we used a video-game-based paradigm that tacitly trained listeners to categorize acoustically complex, artificial nonlinguistic sounds. Before and after training, we used functional MRI to measure how expertise with these sounds modulated temporal lobe activation. Participants' ability to explicitly categorize the nonspeech sounds predicted the change in pretraining to posttraining activation in speech-sensitive regions of the left posterior superior temporal sulcus, suggesting that emergent auditory expertise may help drive this functional regionalization. Thus, seemingly domain-specific patterns of neural activation in higher cortical regions may be driven in part by experience-based restructuring of high-dimensional perceptual space.

Sound categories are represented as distributed patterns in the human auditory cortex

The ability to recognize sounds allows humans and animals to efficiently detect behaviorally relevant events, even in the absence of visual information. Sound recognition in the human brain has been assumed to proceed through several functionally specialized areas, culminating in cortical modules where category-specific processing is carried out. In the present high-resolution fMRI experiment, we challenged this model by using well-controlled natural auditory stimuli and by employing an advanced analysis strategy based on an iterative machine-learning algorithm that allows modeling of spatially distributed, as well as localized, response patterns. Sounds of cats, female singers, acoustic guitars, and tones were controlled for their time-varying spectral characteristics and presented to subjects at three different pitch levels. Sound category information--not detectable with conventional contrast-based methods analysis--could be detected with multivoxel pattern analyses and attributed to spatially distributed areas over the supratemporal cortices. A more localized pattern was observed for processing of pitch laterally to primary auditory areas. Our findings indicate that distributed neuronal populations within the human auditory cortices, including areas conventionally associated with lower-level auditory processing, entail categorical representations of sounds beyond their physical properties.

Dynamic and task-dependent encoding of speech and voice by phase reorganization of cortical oscillations

Speech and vocal sounds are at the core of human communication. Cortical processing of these sounds critically depends on behavioral demands. However, the neurocomputational mechanisms enabling this adaptive processing remain elusive. Here we examine the task-dependent reorganization of electroencephalographic responses to natural speech sounds (vowels /a/, /i/, /u/) spoken by three speakers (two female, one male) while listeners perform a one-back task on either vowel or speaker identity. We show that dynamic changes of sound-evoked responses and phase patterns of cortical oscillations in the alpha band (8-12 Hz) closely reflect the abstraction and analysis of the sounds along the task-relevant dimension. Vowel categorization leads to a significant temporal realignment of responses to the same vowel, e.g., /a/, independent of who pronounced this vowel, whereas speaker categorization leads to a significant temporal realignment of responses to the same speaker, e.g., speaker 1, independent of which vowel she/he pronounced. This transient and goal-dependent realignment of neuronal responses to physically different external events provides a robust cortical coding mechanism for forming and processing abstract representations of auditory (speech) input.