Intracranial study of speech-elicited activity on the human posterolateral superior temporal gyrus

Does Electrophysiological Maturation Shape Language Acquisition?

Infants master temporal patterns of their native language at a developmental trajectory from slow to fast: Shortly after birth, they recognize the slow acoustic modulations specific to their native language before tuning into faster language-specific patterns between 6 and 12 months of age. We propose here that this trajectory is constrained by neuronal maturation-in particular, the gradual emergence of high-frequency neural oscillations in the infant electroencephalogram. Infants' initial focus on slow prosodic modulations is consistent with the prenatal availability of slow electrophysiological activity (i.e., theta- and delta-band oscillations). Our proposal is consistent with the temporal patterns of infant-directed speech, which initially amplifies slow modulations, approaching the faster modulation range of adult-directed speech only as infants' language has advanced sufficiently. Moreover, our proposal agrees with evidence from premature infants showing maturational age is a stronger predictor of language development than ex utero exposure to speech, indicating that premature infants cannot exploit their earlier availability of speech because of electrophysiological constraints. In sum, we provide a new perspective on language acquisition emphasizing neuronal development as a critical driving force of infants' language development.

Sensory processing of native and non-native phonotactic patterns in the alpha and beta frequency bands

The phonotactic patterns of one's native language are established within cortical network processing during development. Sensory processing of native language phonotactic patterns established in memory may be modulated by top-down signals within the alpha and beta frequency bands. To explore sensory processing of phonotactic patterns in the alpha and beta frequency bands, electroencephalograms (EEGs) were recorded from native Polish and native English-speaking adults as they listened to spoken nonwords within same and different nonword pairs. The nonwords contained three phonological sequence onsets that occur in the Polish and English languages (/pət/, /st/, /sət/) and one onset sequence /pt/, which occurs in Polish but not in English onsets. Source localization modeling was used to transform 64-channel EEGs into brain source-level channels. Spectral power values in the low frequencies (2-29 Hz) were analyzed in response to the first nonword in nonword pairs within the context of counterbalanced listening-task conditions, which were presented on separate testing days. For the with-task listening condition, participants performed a behavioral task to the second nonword in the pairs. For the without-task condition participants were only instructed to listen to the stimuli. Thus, in the with-task condition, the first nonword served as a cue for the second nonword, the target stimulus. The results revealed decreased spectral power in the beta frequency band for the with-task condition compared to the without-task condition in response to native language phonotactic patterns. In contrast, the task-related suppression effects in response to the non-native phonotactic pattern /pt/ for the English listeners extended into the alpha frequency band. These effects were localized to source channels in left auditory cortex, the left anterior temporal cortex and the occipital pole. This exploratory study revealed a pattern of results that, if replicated, suggests that native language speech perception is supported by modulations in the alpha and beta frequency bands.

Lateralization and Time-Course of Cortical Phonological Representations during Syllable Production

Spoken language contains information at a broad range of timescales, from phonetic distinctions on the order of milliseconds to semantic contexts which shift over seconds to minutes. It is not well understood how the brain's speech production systems combine features at these timescales into a coherent vocal output. We investigated the spatial and temporal representations in cerebral cortex of three phonological units with different durations: consonants, vowels, and syllables. Electrocorticography (ECoG) recordings were obtained from five participants while speaking single syllables. We developed a novel clustering and Kalman filter-based trend analysis procedure to sort electrodes into temporal response profiles. A linear discriminant classifier was used to determine how strongly each electrode's response encoded phonological features. We found distinct time-courses of encoding phonological units depending on their duration: consonants were represented more during speech preparation, vowels were represented evenly throughout trials, and syllables during production. Locations of strongly speech-encoding electrodes (the top 30% of electrodes) likewise depended on phonological element duration, with consonant-encoding electrodes left-lateralized, vowel-encoding hemispherically balanced, and syllable-encoding right-lateralized. The lateralization of speech-encoding electrodes depended on onset time, with electrodes active before or after speech production favoring left hemisphere and those active during speech favoring the right. Single-electrode speech classification revealed cortical areas with preferential encoding of particular phonemic elements, including consonant encoding in the left precentral and postcentral gyri and syllable encoding in the right middle frontal gyrus. Our findings support neurolinguistic theories of left hemisphere specialization for processing short-timescale linguistic units and right hemisphere processing of longer-duration units.

Gamma Activation and Alpha Suppression within Human Auditory Cortex during a Speech Classification Task

The dynamics of information flow within the auditory cortical hierarchy associated with speech processing and the emergence of hemispheric specialization remain incompletely understood. To study these questions with high spatiotemporal resolution, intracranial recordings in 29 human neurosurgical patients of both sexes were obtained while subjects performed a semantic classification task. Neural activity was recorded from posteromedial portion of Heschl's gyrus (HGPM) and anterolateral portion of Heschl's gyrus (HGAL), planum temporale (PT), planum polare, insula, and superior temporal gyrus (STG). Responses to monosyllabic words exhibited early gamma power increases and a later suppression of alpha power, envisioned to represent feedforward activity and decreased feedback signaling, respectively. Gamma activation and alpha suppression had distinct magnitude and latency profiles. HGPM and PT had the strongest gamma responses with shortest onset latencies, indicating that they are the earliest auditory cortical processing stages. The origin of attenuated top-down influences in auditory cortex, as indexed by alpha suppression, was in STG and HGAL. Gamma responses and alpha suppression were typically larger to nontarget words than tones. Alpha suppression was uniformly greater to target versus nontarget stimuli. Hemispheric bias for words versus tones and for target versus nontarget words, when present, was left lateralized. Better task performance was associated with increased gamma activity in the left PT and greater alpha suppression in HGPM and HGAL bilaterally. The prominence of alpha suppression during semantic classification and its accessibility for noninvasive electrophysiologic studies suggests that this measure is a promising index of auditory cortical speech processing.SIGNIFICANCE STATEMENT Understanding the dynamics of cortical speech processing requires the use of active tasks. This is the first comprehensive intracranial electroencephalography study to examine cortical activity within the superior temporal plane, lateral superior temporal gyrus, and the insula during a semantic classification task. Distinct gamma activation and alpha suppression profiles clarify the functional organization of feedforward and feedback processing within the auditory cortical hierarchy. Asymmetries in cortical speech processing emerge at early processing stages. Relationships between cortical activity and task performance are interpreted in the context of current models of speech processing. Results lay the groundwork for iEEG studies using connectivity measures of the bidirectional information flow within the auditory processing hierarchy.

Acoustic-level and language-specific processing of native and non-native phonological sequence onsets in the low gamma and theta-frequency bands

Acoustic structures associated with native-language phonological sequences are enhanced within auditory pathways for perception, although the underlying mechanisms are not well understood. To elucidate processes that facilitate perception, time-frequency (T-F) analyses of EEGs obtained from native speakers of English and Polish were conducted. Participants listened to same and different nonword pairs within counterbalanced attend and passive conditions. Nonwords contained the onsets /pt/, /pət/, /st/, and /sət/ that occur in both the Polish and English languages with the exception of /pt/, which never occurs in the English language in word onset. Measures of spectral power and inter-trial phase locking (ITPL) in the low gamma (LG) and theta-frequency bands were analyzed from two bilateral, auditory source-level channels, created through source localization modeling. Results revealed significantly larger spectral power in LG for the English listeners to the unfamiliar /pt/ onsets from the right hemisphere at early cortical stages, during the passive condition. Further, ITPL values revealed distinctive responses in high and low-theta to acoustic characteristics of the onsets, which were modulated by language exposure. These findings, language-specific processing in LG and acoustic-level and language-specific processing in theta, support the view that multi scale temporal processing in the LG and theta-frequency bands facilitates speech perception.

Speech Computations of the Human Superior Temporal Gyrus

Human speech perception results from neural computations that transform external acoustic speech signals into internal representations of words. The superior temporal gyrus (STG) contains the nonprimary auditory cortex and is a critical locus for phonological processing. Here, we describe how speech sound representation in the STG relies on fundamentally nonlinear and dynamical processes, such as categorization, normalization, contextual restoration, and the extraction of temporal structure. A spatial mosaic of local cortical sites on the STG exhibits complex auditory encoding for distinct acoustic-phonetic and prosodic features. We propose that as a population ensemble, these distributed patterns of neural activity give rise to abstract, higher-order phonemic and syllabic representations that support speech perception. This review presents a multi-scale, recurrent model of phonological processing in the STG, highlighting the critical interface between auditory and language systems. Expected final online publication date for the Annual Review of Psychology, Volume 73 is January 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Learning nonnative speech sounds changes local encoding in the adult human cortex

Adults can learn to identify nonnative speech sounds with training, albeit with substantial variability in learning behavior. Increases in behavioral accuracy are associated with increased separability for sound representations in cortical speech areas. However, it remains unclear whether individual auditory neural populations all show the same types of changes with learning, or whether there are heterogeneous encoding patterns. Here, we used high-resolution direct neural recordings to examine local population response patterns, while native English listeners learned to recognize unfamiliar vocal pitch patterns in Mandarin Chinese tones. We found a distributed set of neural populations in bilateral superior temporal gyrus and ventrolateral frontal cortex, where the encoding of Mandarin tones changed throughout training as a function of trial-by-trial accuracy ("learning effect"), including both increases and decreases in the separability of tones. These populations were distinct from populations that showed changes as a function of exposure to the stimuli regardless of trial-by-trial accuracy. These learning effects were driven in part by more variable neural responses to repeated presentations of acoustically identical stimuli. Finally, learning effects could be predicted from speech-evoked activity even before training, suggesting that intrinsic properties of these populations make them amenable to behavior-related changes. Together, these results demonstrate that nonnative speech sound learning involves a wide array of changes in neural representations across a distributed set of brain regions.

Functional characterization of human Heschl's gyrus in response to natural speech

Heschl's gyrus (HG) is a brain area that includes the primary auditory cortex in humans. Due to the limitations in obtaining direct neural measurements from this region during naturalistic speech listening, the functional organization and the role of HG in speech perception remain uncertain. Here, we used intracranial EEG to directly record neural activity in HG in eight neurosurgical patients as they listened to continuous speech stories. We studied the spatial distribution of acoustic tuning and the organization of linguistic feature encoding. We found a main gradient of change from posteromedial to anterolateral parts of HG. We also observed a decrease in frequency and temporal modulation tuning and an increase in phonemic representation, speaker normalization, speech sensitivity, and response latency. We did not observe a difference between the two brain hemispheres. These findings reveal a functional role for HG in processing and transforming simple to complex acoustic features and inform neurophysiological models of speech processing in the human auditory cortex.

Categorical Encoding of Vowels in Primary Auditory Cortex

Speech perception involves mapping from a continuous and variable acoustic speech signal to discrete, linguistically meaningful units. However, it is unclear where in the auditory processing stream speech sound representations cease to be veridical (faithfully encoding precise acoustic properties) and become categorical (encoding sounds as linguistic categories). In this study, we used functional magnetic resonance imaging and multivariate pattern analysis to determine whether tonotopic primary auditory cortex (PAC), defined as tonotopic voxels falling within Heschl's gyrus, represents one class of speech sounds-vowels-veridically or categorically. For each of 15 participants, 4 individualized synthetic vowel stimuli were generated such that the vowels were equidistant in acoustic space, yet straddled a categorical boundary (with the first 2 vowels perceived as [i] and the last 2 perceived as [i]). Each participant's 4 vowels were then presented in a block design with an irrelevant but attention-demanding level change detection task. We found that in PAC bilaterally, neural discrimination between pairs of vowels that crossed the categorical boundary was more accurate than neural discrimination between equivalently spaced vowel pairs that fell within a category. These findings suggest that PAC does not represent vowel sounds veridically, but that encoding of vowels is shaped by linguistically relevant phonemic categories.

Neural Correlates of Vocal Auditory Feedback Processing: Unique Insights from Electrocorticography Recordings in a Human Cochlear Implant User

There is considerable interest in understanding cortical processing and the function of top-down and bottom-up human neural circuits that control speech production. Research efforts to investigate these circuits are aided by analysis of spectro-temporal response characteristics of neural activity recorded by electrocorticography (ECoG). Further, cortical processing may be altered in the case of hearing-impaired cochlear implant (CI) users, as electric excitation of the auditory nerve creates a markedly different neural code for speech compared with that of the functionally intact hearing system. Studies of cortical activity in CI users typically record scalp potentials and are hampered by stimulus artifact contamination and by spatiotemporal filtering imposed by the skull. We present a unique case of a CI user who required direct recordings from the cortical surface using subdural electrodes implanted for epilepsy assessment. Using experimental conditions where the subject vocalized in the presence (CIs ON) or absence (CIs OFF) of auditory feedback, or listened to playback of self-vocalizations without production, we observed ECoG activity primarily in γ (32–70 Hz) and high γ (70–150 Hz) bands at focal regions on the lateral surface of the superior temporal gyrus (STG). High γ band responses differed in their amplitudes across conditions and cortical sites, possibly reflecting different rates of stimulus presentation and differing levels of neural adaptation. STG γ responses to playback and vocalization with auditory feedback were not different from responses to vocalization without feedback, indicating this activity reflects not only auditory, but also attentional, efference-copy, and sensorimotor processing during speech production.

Chasing language through the brain: Successive parallel networks

OBJECTIVE

To describe the spatio-temporal dynamics and interactions during linguistic and memory tasks.

METHODS

Event-related electrocorticographic (ECoG) spectral patterns obtained during cognitive tasks from 26 epilepsy patients (aged: 9-60 y) were analyzed in order to examine the spatio-temporal patterns of activation of cortical language areas. ECoGs (1024 Hz/channel) were recorded from 1567 subdural electrodes and 510 depth electrodes chronically implanted over or within the frontal, parietal, occipital and/or temporal lobes as part of their surgical work-up for intractable seizures. Six language/memory tasks were performed, which required responding verbally to auditory or visual word stimuli. Detailed analysis of electrode locations allowed combining results across patients.

RESULTS

Transient increases in induced ECoG gamma power (70-100 Hz) were observed in response to hearing words (central superior temporal gyrus), reading text and naming pictures (occipital and fusiform cortex) and speaking (pre-central, post-central and sub-central cortex).

CONCLUSIONS

Between these activations there was widespread spatial divergence followed by convergence of gamma activity that reliably identified cortical areas associated with task-specific processes.

SIGNIFICANCE

The combined dataset supports the concept of functionally-specific locally parallel language networks that are widely distributed, partially interacting in succession to serve the cognitive and behavioral demands of the tasks.

Electrophysiology of the Human Superior Temporal Sulcus during Speech Processing

The superior temporal sulcus (STS) is a crucial hub for speech perception and can be studied with high spatiotemporal resolution using electrodes targeting mesial temporal structures in epilepsy patients. Goals of the current study were to clarify functional distinctions between the upper (STSU) and the lower (STSL) bank, hemispheric asymmetries, and activity during self-initiated speech. Electrophysiologic properties were characterized using semantic categorization and dialog-based tasks. Gamma-band activity and alpha-band suppression were used as complementary measures of STS activation. Gamma responses to auditory stimuli were weaker in STSL compared with STSU and had longer onset latencies. Activity in anterior STS was larger during speaking than listening; the opposite pattern was observed more posteriorly. Opposite hemispheric asymmetries were found for alpha suppression in STSU and STSL. Alpha suppression in the STS emerged earlier than in core auditory cortex, suggesting feedback signaling within the auditory cortical hierarchy. STSL was the only region where gamma responses to words presented in the semantic categorization tasks were larger in subjects with superior task performance. More pronounced alpha suppression was associated with better task performance in Heschl's gyrus, superior temporal gyrus, and STS. Functional differences between STSU and STSL warrant their separate assessment in future studies.

Transformation of a temporal speech cue to a spatial neural code in human auditory cortex

In speech, listeners extract continuously-varying spectrotemporal cues from the acoustic signal to perceive discrete phonetic categories. Spectral cues are spatially encoded in the amplitude of responses in phonetically-tuned neural populations in auditory cortex. It remains unknown whether similar neurophysiological mechanisms encode temporal cues like voice-onset time (VOT), which distinguishes sounds like /b/ and/p/. We used direct brain recordings in humans to investigate the neural encoding of temporal speech cues with a VOT continuum from /ba/ to /pa/. We found that distinct neural populations respond preferentially to VOTs from one phonetic category, and are also sensitive to sub-phonetic VOT differences within a population's preferred category. In a simple neural network model, simulated populations tuned to detect either temporal gaps or coincidences between spectral cues captured encoding patterns observed in real neural data. These results demonstrate that a spatial/amplitude neural code underlies the cortical representation of both spectral and temporal speech cues.

Early Interactive Acoustic Experience with Non-speech Generalizes to Speech and Confers a Syllabic Processing Advantage at 9 Months

During early development, the infant brain is highly plastic and sensory experiences modulate emerging cortical maps, enhancing processing efficiency as infants set up key linguistic precursors. Early interactive acoustic experience (IAE) with spectrotemporally-modulated non-speech has been shown to facilitate optimal acoustic processing and generalizes to novel non-speech sounds at 7-months-of-age. Here we demonstrate that effects of non-speech IAE endure well beyond the immediate training period and robustly generalize to speech processing. Infants who received non-speech IAE differed at 9-months-of-age from both naïve controls and those with only passive acoustic exposure, demonstrating broad modulation of oscillatory dynamics. For the standard syllable, increased high-gamma (>70 Hz) power within auditory cortices indicates that IAE fosters native speech processing, facilitating establishment of phonemic representations. The higher left beta power seen may reflect increased linking of sensory information and corresponding articulatory patterns, while bilateral decreases in theta power suggest more mature automatized speech processing, as less neuronal resources were allocated to process syllabic information. For the deviant syllable, left-lateralized gamma (<70 Hz) enhancement suggests IAE promotes phonemic-related discrimination abilities. Theta power increases in right auditory cortex, known for favoring slow-rate decoding, implies IAE facilitates the more demanding processing of the sporadic deviant syllable.

Neural Entrainment and Attentional Selection in the Listening Brain

The streams of sounds we typically attend to abound in acoustic regularities. Neural entrainment is seen as an important mechanism that the listening brain exploits to attune to these regularities and to enhance the representation of attended sounds. We delineate the neurophysiology underlying this mechanism and review entrainment alongside its more pragmatic signature, often called 'speech tracking'. The latter has become a popular analytical approach to trace the reflection of acoustic and linguistic information at different levels of granularity, from neurophysiology to neuroimaging. As we discuss, the concept of entrainment offers both a putative neurophysiological mechanism for selective listening and a versatile window onto the neural basis of hearing and speech comprehension.

The Encoding of Speech Sounds in the Superior Temporal Gyrus

The human superior temporal gyrus (STG) is critical for extracting meaningful linguistic features from speech input. Local neural populations are tuned to acoustic-phonetic features of all consonants and vowels and to dynamic cues for intonational pitch. These populations are embedded throughout broader functional zones that are sensitive to amplitude-based temporal cues. Beyond speech features, STG representations are strongly modulated by learned knowledge and perceptual goals. Currently, a major challenge is to understand how these features are integrated across space and time in the brain during natural speech comprehension. We present a theory that temporally recurrent connections within STG generate context-dependent phonological representations, spanning longer temporal sequences relevant for coherent percepts of syllables, words, and phrases.

Speaker-normalized sound representations in the human auditory cortex

The acoustic dimensions that distinguish speech sounds (like the vowel differences in "boot" and "boat") also differentiate speakers' voices. Therefore, listeners must normalize across speakers without losing linguistic information. Past behavioral work suggests an important role for auditory contrast enhancement in normalization: preceding context affects listeners' perception of subsequent speech sounds. Here, using intracranial electrocorticography in humans, we investigate whether and how such context effects arise in auditory cortex. Participants identified speech sounds that were preceded by phrases from two different speakers whose voices differed along the same acoustic dimension as target words (the lowest resonance of the vocal tract). In every participant, target vowels evoke a speaker-dependent neural response that is consistent with the listener's perception, and which follows from a contrast enhancement model. Auditory cortex processing thus displays a critical feature of normalization, allowing listeners to extract meaningful content from the voices of diverse speakers.

The peri-Sylvian cortical network underlying single word repetition revealed by electrocortical stimulation and direct neural recordings

Verbal repetition requires the coordination of auditory, memory, linguistic, and motor systems. To date, the basic dynamics of neural information processing in this deceptively simple behavior are largely unknown. Here, we examined the neural processes underlying verbal repetition using focal interruption (electrocortical stimulation) in 58 patients undergoing awake craniotomies, and neurophysiological recordings (electrocorticography) in 8 patients while they performed a single word repetition task. Electrocortical stimulation revealed that sub-components of the left peri-Sylvian network involved in single word repetition could be differentially interrupted, producing transient perceptual deficits, paraphasic errors, or speech arrest. Electrocorticography revealed the detailed spatio-temporal dynamics of cortical activation, involving a highly-ordered, but overlapping temporal progression of cortical high gamma (75-150Hz) activity throughout the peri-Sylvian cortex. We observed functionally distinct serial and parallel cortical processing corresponding to successive stages of general auditory processing (posterior superior temporal gyrus), speech-specific auditory processing (middle and posterior superior temporal gyrus), working memory (inferior frontal cortex), and motor articulation (sensorimotor cortex). Together, these methods reveal the dynamics of coordinated activity across peri-Sylvian cortex during verbal repetition.

Minimally-verbal children with autism show deficits in theta and gamma oscillations during processing of semantically-related visual information

To acquire language, children must build phonemic representations of their native language, learn to associate auditory words to visual objects and assemble a lexicon. It is not clear however, whether the limited linguistic ability seen in minimally-verbal (MV) children with Autism Spectrum Disorder (ASD) relates to deficits in cortical representation of an object and/or in linking an object to its semantic information. This EEG-based study investigated neural mechanisms underlying visual processing of common objects in MV-ASD and control children. Ten MV-ASD children, 4- to 7- years-old and 15 age/gender-matched controls, were presented with a picture-word matching paradigm. Time-frequency analyses were conducted at the sources generating the event-related responses at both early and late visual processing. Permutation testing identified spectral power and phase coherence clusters that significantly differed between the groups. As compared to controls, MV-ASD children exhibited smaller amplitudes and longer source latencies; decreased gamma and theta power with less theta phase coherence in occipital regions, and reduced frontal gamma power. Our results confirm that visual processing is altered in MV-ASD children and suggest that some of the linguistic differences observed in these children arise from impaired object/label cortical representations and reduced allocation of attention, which would impact lexical acquisition.

Age-related differences in Voice-Onset-Time in Polish language users: An ERP study

Using the Mismatch Negativity (MMN) paradigm we investigated for the first time cortical responses to consonant - vowel (CV) syllables differing in Voice-Onset-Time (VOT) for Polish, a member of the Slavic group of languages. The study aimed at testing age-related effects on different ERP responses in young (20-30 years of age) and elderly (60-68 years) native Polish speakers. Participants were presented with a sequence of voiced and voiceless stop CV syllables /to/ and /do/ with different VOT values (-100 ms, -70 ms, -30 ms, -20 ms, +20 ms, +50 ms). We analyzed MMN and P1, N1, N1', P2, N2 components. Our results showed an age-related decline in voicing perception in all tested ERP components. This decline could be explained in relation to a general slowing in neural processing with advancing age and may be associated with difficulties in temporal- and spectral-information processing in elderly people. Our findings revealed also that specific features of Slavic languages influence ERP morphology in a different way than reported in the literature for aspirating languages.

Spontaneous Neural Activity in the Superior Temporal Gyrus Recapitulates Tuning for Speech Features

Background: Numerous studies have demonstrated that individuals exhibit structured neural activity in many brain regions during rest that is also observed during different tasks, however it is still not clear whether and how resting state activity patterns may relate to underlying tuning for specific stimuli. In the posterior superior temporal gyrus (STG), distinct neural activity patterns are observed during the perception of specific linguistic speech features. We hypothesized that spontaneous resting-state neural dynamics of the STG would be structured to reflect its role in speech perception, exhibiting an organization along speech features as seen during speech perception.

Methods: Human cortical local field potentials were recorded from the superior temporal gyrus (STG) in 8 patients undergoing surgical treatment of epilepsy. Signals were recorded during speech perception and rest. Patterns of neural activity (high gamma power: 70–150 Hz) during rest, extracted with spatiotemporal principal component analysis, were compared to spatiotemporal neural responses to speech features during perception. Hierarchical clustering was applied to look for patterns in rest that corresponded to speech feature tuning.

Results: Significant correlations were found between neural responses to speech features (sentence onsets, consonants, and vowels) and the spontaneous neural activity in the STG. Across subjects, these correlations clustered into five groups, demonstrating tuning for speech features—most robustly for acoustic onsets. These correlations were not seen in other brain areas, or during motor and spectrally-rotated speech control tasks.

Conclusions: In this study, we present evidence that the RS structure of STG activity robustly recapitulates its stimulus-evoked response to acoustic onsets. Further, secondary patterns in RS activity appear to correlate with stimulus-evoked responses to speech features. The role of these spontaneous spatiotemporal activity patterns remains to be elucidated.

Toward a Database of Intracranial Electrophysiology during Natural Language Presentation

Intracranial electrophysiology (iEEG) studies using cognitive tasks contribute to the understanding of the neural basis of language. However, though iEEG is recorded continuously during clinical treatment, due to patient considerations task time is limited. To increase the usefulness of iEEG recordings for language study, we provided patients with a tablet pre-loaded with media filled with natural language, wirelessly synchronized to clinical iEEG. This iEEG data collected and time-locked to natural language presentation is particularly applicable for studying the neural basis of combining words into larger contexts. We validate this approach with pilot analyses involving words heard during a movie, tagging syntactic properties and verb contextual probabilities. Event-related averages of high-frequency power (70-170Hz) identified bilateral perisylvian electrodes with differential responses to syntactic class and a linear regression identified activity associated with contextual probabilities, demonstrating the usefulness of aligning media to iEEG. We imagine future multi-site collaborations building an 'intracranial neurolinguistic corpus'.

Across-subjects classification of stimulus modality from human MEG high frequency activity

Single-trial analyses have the potential to uncover meaningful brain dynamics that are obscured when averaging across trials. However, low signal-to-noise ratio (SNR) can impede the use of single-trial analyses and decoding methods. In this study, we investigate the applicability of a single-trial approach to decode stimulus modality from magnetoencephalographic (MEG) high frequency activity. In order to classify the auditory versus visual presentation of words, we combine beamformer source reconstruction with the random forest classification method. To enable group level inference, the classification is embedded in an across-subjects framework. We show that single-trial gamma SNR allows for good classification performance (accuracy across subjects: 66.44%). This implies that the characteristics of high frequency activity have a high consistency across trials and subjects. The random forest classifier assigned informational value to activity in both auditory and visual cortex with high spatial specificity. Across time, gamma power was most informative during stimulus presentation. Among all frequency bands, the 75 Hz to 95 Hz band was the most informative frequency band in visual as well as in auditory areas. Especially in visual areas, a broad range of gamma frequencies (55 Hz to 125 Hz) contributed to the successful classification. Thus, we demonstrate the feasibility of single-trial approaches for decoding the stimulus modality across subjects from high frequency activity and describe the discriminative gamma activity in time, frequency, and space.

Averaging brain activity across trials is a powerful way to increase signal-to-noise ratio in MEG data. This approach, however, potentially obscures meaningful brain dynamics that unfold on the single-trial level. Single-trial analyses have been successfully applied to time domain or low frequency oscillatory activity; its application to MEG high frequency activity is hindered by the low amplitude of these signals. In the present study, we show that stimulus modality (visual versus auditory presentation of words) can successfully be decoded from single-trial MEG high frequency activity by combining source reconstruction with a random forest classification algorithm. This approach reveals patterns of activity above 75 Hz in both visual and auditory cortex, highlighting the importance of high frequency activity for the processing of domain-specific stimuli. Thereby, our results extend prior findings by revealing high-frequency activity in auditory cortex related to auditory word stimuli in MEG data. The adopted across-subjects framework furthermore suggests a high inter-individual consistency in the high frequency activity patterns.

Multi-Regional Adaptation in Human Auditory Association Cortex

In auditory cortex, neural responses decrease with stimulus repetition, known as adaptation. Adaptation is thought to facilitate detection of novel sounds and improve perception in noisy environments. Although it is well established that adaptation occurs in primary auditory cortex, it is not known whether adaptation also occurs in higher auditory areas involved in processing complex sounds, such as speech. Resolving this issue is important for understanding the neural bases of adaptation and to avoid potential post-operative deficits after temporal lobe surgery for treatment of focal epilepsy. Intracranial electrocorticographic recordings were acquired simultaneously from electrodes implanted in primary and association auditory areas of the right (non-dominant) temporal lobe in a patient with complex partial seizures originating from the inferior parietal lobe. Simple and complex sounds were presented in a passive oddball paradigm. We measured changes in single-trial high-gamma power (70–150 Hz) and in regional and inter-regional network-level activity indexed by cross-frequency coupling. Repetitive tones elicited the greatest adaptation and corresponding increases in cross-frequency coupling in primary auditory cortex. Conversely, auditory association cortex showed stronger adaptation for complex sounds, including speech. This first report of multi-regional adaptation in human auditory cortex highlights the role of the non-dominant temporal lobe in suppressing neural responses to repetitive background sounds (noise). These results underscore the clinical utility of functional mapping to avoid potential post-operative deficits including increased listening difficulties in noisy, real-world environments.

Auditory processing in the human cortex: An intracranial electrophysiology perspective

Objective

Direct electrophysiological recordings in epilepsy patients offer an opportunity to study human auditory cortical processing with unprecedented spatiotemporal resolution. This review highlights recent intracranial studies of human auditory cortex and focuses on its basic response properties as well as modulation of cortical activity during the performance of active behavioral tasks.

Data Sources: Literature review.

Review Methods: A review of the literature was conducted to summarize the functional organization of human auditory and auditory‐related cortex as revealed using intracranial recordings.

Results

The tonotopically organized core auditory cortex within the posteromedial portion of Heschl's gyrus represents spectrotemporal features of sounds with high temporal precision and short response latencies. At this level of processing, high gamma (70–150 Hz) activity is minimally modulated by task demands. Non‐core cortex on the lateral surface of the superior temporal gyrus also maintains representation of stimulus acoustic features and, for speech, subserves transformation of acoustic inputs into phonemic representations. High gamma responses in this region are modulated by task requirements. Prefrontal cortex exhibits complex response patterns, related to stimulus intelligibility and task relevance. At this level of auditory processing, activity is strongly modulated by task requirements and reflects behavioral performance.

Conclusions

Direct recordings from the human brain reveal hierarchical organization of sound processing within auditory and auditory‐related cortex.

Level of Evidence

Level V

Chronic ambulatory electrocorticography from human speech cortex

Direct intracranial recording of human brain activity is an important approach for deciphering neural mechanisms of cognition. Such recordings, usually made in patients with epilepsy undergoing inpatient monitoring for seizure localization, are limited in duration and depend on patients' tolerance for the challenges associated with recovering from brain surgery. Thus, typical intracranial recordings, similar to most non-invasive approaches in humans, provide snapshots of brain activity in acute, highly constrained settings, limiting opportunities to understand long timescale and natural, real-world phenomena. A new device for treating some forms of drug-resistant epilepsy, the NeuroPace RNS® System, includes a cranially-implanted neurostimulator and intracranial electrodes that continuously monitor brain activity and respond to incipient seizures with electrical counterstimulation. The RNS System can record epileptic brain activity over years, but whether it can record meaningful, behavior-related physiological responses has not been demonstrated. Here, in a human subject with electrodes implanted over high-level speech-auditory cortex (Wernicke's area; posterior superior temporal gyrus), we report that cortical evoked responses to spoken sentences are robust, selective to phonetic features, and stable over nearly 1.5 years. In a second subject with RNS System electrodes implanted over frontal cortex (Broca's area, posterior inferior frontal gyrus), we found that word production during a naming task reliably evokes cortical responses preceding speech onset. The spatiotemporal resolution, high signal-to-noise, and wireless nature of this system's intracranial recordings make it a powerful new approach to investigate the neural correlates of human cognition over long timescales in natural ambulatory settings.

Dynamic Encoding of Acoustic Features in Neural Responses to Continuous Speech

Humans are unique in their ability to communicate using spoken language. However, it remains unclear how the speech signal is transformed and represented in the brain at different stages of the auditory pathway. In this study, we characterized electroencephalography responses to continuous speech by obtaining the time-locked responses to phoneme instances (phoneme-related potential). We showed that responses to different phoneme categories are organized by phonetic features. We found that each instance of a phoneme in continuous speech produces multiple distinguishable neural responses occurring as early as 50 ms and as late as 400 ms after the phoneme onset. Comparing the patterns of phoneme similarity in the neural responses and the acoustic signals confirms a repetitive appearance of acoustic distinctions of phonemes in the neural data. Analysis of the phonetic and speaker information in neural activations revealed that different time intervals jointly encode the acoustic similarity of both phonetic and speaker categories. These findings provide evidence for a dynamic neural transformation of low-level speech features as they propagate along the auditory pathway, and form an empirical framework to study the representational changes in learning, attention, and speech disorders.SIGNIFICANCE STATEMENT We characterized the properties of evoked neural responses to phoneme instances in continuous speech. We show that each instance of a phoneme in continuous speech produces several observable neural responses at different times occurring as early as 50 ms and as late as 400 ms after the phoneme onset. Each temporal event explicitly encodes the acoustic similarity of phonemes, and linguistic and nonlinguistic information are best represented at different time intervals. Finally, we show a joint encoding of phonetic and speaker information, where the neural representation of speakers is dependent on phoneme category. These findings provide compelling new evidence for dynamic processing of speech sounds in the auditory pathway.

Perceptual restoration of masked speech in human cortex

Humans are adept at understanding speech despite the fact that our natural listening environment is often filled with interference. An example of this capacity is phoneme restoration, in which part of a word is completely replaced by noise, yet listeners report hearing the whole word. The neurological basis for this unconscious fill-in phenomenon is unknown, despite being a fundamental characteristic of human hearing. Here, using direct cortical recordings in humans, we demonstrate that missing speech is restored at the acoustic-phonetic level in bilateral auditory cortex, in real-time. This restoration is preceded by specific neural activity patterns in a separate language area, left frontal cortex, which predicts the word that participants later report hearing. These results demonstrate that during speech perception, missing acoustic content is synthesized online from the integration of incoming sensory cues and the internal neural dynamics that bias word-level expectation and prediction.

We can often ‘fill in' missing or occluded sounds from a speech signal—an effect known as phoneme restoration. Leonard et al. found a real-time restoration of the missing sounds in the superior temporal auditory cortex in humans. Interestingly, neural activity in frontal regions prior to the stimulus can predict the word that the participant would later hear.

Intracerebral evidence of rhythm transform in the human auditory cortex

Musical entrainment is shared by all human cultures and the perception of a periodic beat is a cornerstone of this entrainment behavior. Here, we investigated whether beat perception might have its roots in the earliest stages of auditory cortical processing. Local field potentials were recorded from 8 patients implanted with depth-electrodes in Heschl's gyrus and the planum temporale (55 recording sites in total), usually considered as human primary and secondary auditory cortices. Using a frequency-tagging approach, we show that both low-frequency (<30 Hz) and high-frequency (>30 Hz) neural activities in these structures faithfully track auditory rhythms through frequency-locking to the rhythm envelope. A selective gain in amplitude of the response frequency-locked to the beat frequency was observed for the low-frequency activities but not for the high-frequency activities, and was sharper in the planum temporale, especially for the more challenging syncopated rhythm. Hence, this gain process is not systematic in all activities produced in these areas and depends on the complexity of the rhythmic input. Moreover, this gain was disrupted when the rhythm was presented at fast speed, revealing low-pass response properties which could account for the propensity to perceive a beat only within the musical tempo range. Together, these observations show that, even though part of these neural transforms of rhythms could already take place in subcortical auditory processes, the earliest auditory cortical processes shape the neural representation of rhythmic inputs in favor of the emergence of a periodic beat.

Oscillatory Dynamics Underlying Perceptual Narrowing of Native Phoneme Mapping from 6 to 12 Months of Age

During the first months of life, human infants process phonemic elements from all languages similarly. However, by 12 months of age, as language-specific phonemic maps are established, infants respond preferentially to their native language. This process, known as perceptual narrowing, supports neural representation and thus efficient processing of the distinctive phonemes within the sound environment. Although oscillatory mechanisms underlying processing of native and non-native phonemic contrasts were recently delineated in 6-month-old infants, the maturational trajectory of these mechanisms remained unclear. A group of typically developing infants born into monolingual English families, were followed from 6 to 12 months and presented with English and Spanish syllable contrasts varying in voice-onset time. Brain responses were recorded with high-density electroencephalogram, and sources of event-related potential generators identified at right and left auditory cortices at 6 and 12 months and also at frontal cortex at 6 months. Time-frequency analyses conducted at source level found variations in both θ and γ ranges across age. Compared with 6-month-olds, 12-month-olds' responses to native phonemes showed smaller and faster phase synchronization and less spectral power in the θ range, and increases in left phase synchrony as well as induced high-γ activity in both frontal and left auditory sources. These results demonstrate that infants become more automatized and efficient in processing their native language as they approach 12 months of age via the interplay between θ and γ oscillations. We suggest that, while θ oscillations support syllable processing, γ oscillations underlie phonemic perceptual narrowing, progressively favoring mapping of native over non-native language across the first year of life.

SIGNIFICANCE STATEMENT

During early language acquisition, typically developing infants gradually construct phonemic maps of their native language in auditory cortex. It is well known that, by 12 months of age, human infants move from universal discrimination of most linguistic phonemic contrasts to phonemic expertise in their native language. This perceptual narrowing occurs at the expense of the ability to process non-native phonemes. However, the neural mechanisms underlying this process are still poorly understood. Here we demonstrate that perceptual narrowing is, at least in part, accomplished by decreasing power and phase coherence in the θ range while increasing activity in high-γ in left auditory cortex. Understanding the normative neural mechanisms that support early language acquisition is crucial to understanding and perhaps ameliorating developmental language disorders.

Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity

OBJECTIVE

The superior temporal gyrus (STG) and neighboring brain regions play a key role in human language processing. Previous studies have attempted to reconstruct speech information from brain activity in the STG, but few of them incorporate the probabilistic framework and engineering methodology used in modern speech recognition systems. In this work, we describe the initial efforts toward the design of a neural speech recognition (NSR) system that performs continuous phoneme recognition on English stimuli with arbitrary vocabulary sizes using the high gamma band power of local field potentials in the STG and neighboring cortical areas obtained via electrocorticography.

APPROACH

The system implements a Viterbi decoder that incorporates phoneme likelihood estimates from a linear discriminant analysis model and transition probabilities from an n-gram phonemic language model. Grid searches were used in an attempt to determine optimal parameterizations of the feature vectors and Viterbi decoder.

MAIN RESULTS

The performance of the system was significantly improved by using spatiotemporal representations of the neural activity (as opposed to purely spatial representations) and by including language modeling and Viterbi decoding in the NSR system.

SIGNIFICANCE

These results emphasize the importance of modeling the temporal dynamics of neural responses when analyzing their variations with respect to varying stimuli and demonstrate that speech recognition techniques can be successfully leveraged when decoding speech from neural signals. Guided by the results detailed in this work, further development of the NSR system could have applications in the fields of automatic speech recognition and neural prosthetics.

Electrocorticographic Activation within Human Auditory Cortex during Dialog-Based Language and Cognitive Testing

Current models of cortical speech and language processing include multiple regions within the temporal lobe of both hemispheres. Human communication, by necessity, involves complex interactions between regions subserving speech and language processing with those involved in more general cognitive functions. To assess these interactions, we utilized an ecologically salient conversation-based approach. This approach mandates that we first clarify activity patterns at the earliest stages of cortical speech processing. Therefore, we examined high gamma (70–150 Hz) responses within the electrocorticogram (ECoG) recorded simultaneously from Heschl’s gyrus (HG) and lateral surface of the superior temporal gyrus (STG). Subjects were neurosurgical patients undergoing evaluation for treatment of medically intractable epilepsy. They performed an expanded version of the Mini-mental state examination (MMSE), which included additional spelling, naming, and memory-based tasks. ECoG was recorded from HG and the STG using multicontact depth and subdural electrode arrays, respectively. Differences in high gamma activity during listening to the interviewer and the subject’s self-generated verbal responses were quantified for each recording site and across sites within HG and STG. The expanded MMSE produced widespread activation in auditory cortex of both hemispheres. No significant difference was found between activity during listening to the interviewer’s questions and the subject’s answers in posteromedial HG (auditory core cortex). A different pattern was observed throughout anterolateral HG and posterior and middle portions of lateral STG (non-core auditory cortical areas), where activity was significantly greater during listening compared to speaking. No systematic task-specific differences in the degree of suppression during speaking relative to listening were found in posterior and middle STG. Individual sites could, however, exhibit task-related variability in the degree of suppression during speaking compared to listening. The current study demonstrates that ECoG recordings can be acquired in time-efficient dialog-based paradigms, permitting examination of language and cognition in an ecologically salient manner. The results obtained from auditory cortex serve as a foundation for future studies addressing patterns of activity beyond auditory cortex that subserve human communication.

Cortical subnetwork dynamics during human language tasks

Language tasks require the coordinated activation of multiple subnetworks-groups of related cortical interactions involved in specific components of task processing. Although electrocorticography (ECoG) has sufficient temporal and spatial resolution to capture the dynamics of event-related interactions between cortical sites, it is difficult to decompose these complex spatiotemporal patterns into functionally discrete subnetworks without explicit knowledge of each subnetwork's timing. We hypothesized that subnetworks corresponding to distinct components of task-related processing could be identified as groups of interactions with co-varying strengths. In this study, five subjects implanted with ECoG grids over language areas performed word repetition and picture naming. We estimated the interaction strength between each pair of electrodes during each task using a time-varying dynamic Bayesian network (tvDBN) model constructed from the power of high gamma (70-110Hz) activity, a surrogate for population firing rates. We then reduced the dimensionality of this model using principal component analysis (PCA) to identify groups of interactions with co-varying strengths, which we term functional network components (FNCs). This data-driven technique estimates both the weight of each interaction's contribution to a particular subnetwork, and the temporal profile of each subnetwork's activation during the task. We found FNCs with temporal and anatomical features consistent with articulatory preparation in both tasks, and with auditory and visual processing in the word repetition and picture naming tasks, respectively. These FNCs were highly consistent between subjects with similar electrode placement, and were robust enough to be characterized in single trials. Furthermore, the interaction patterns uncovered by FNC analysis correlated well with recent literature suggesting important functional-anatomical distinctions between processing external and self-produced speech. Our results demonstrate that subnetwork decomposition of event-related cortical interactions is a powerful paradigm for interpreting the rich dynamics of large-scale, distributed cortical networks during human cognitive tasks.

The auditory representation of speech sounds in human motor cortex

In humans, listening to speech evokes neural responses in the motor cortex. This has been controversially interpreted as evidence that speech sounds are processed as articulatory gestures. However, it is unclear what information is actually encoded by such neural activity. We used high-density direct human cortical recordings while participants spoke and listened to speech sounds. Motor cortex neural patterns during listening were substantially different than during articulation of the same sounds. During listening, we observed neural activity in the superior and inferior regions of ventral motor cortex. During speaking, responses were distributed throughout somatotopic representations of speech articulators in motor cortex. The structure of responses in motor cortex during listening was organized along acoustic features similar to auditory cortex, rather than along articulatory features as during speaking. Motor cortex does not contain articulatory representations of perceived actions in speech, but rather, represents auditory vocal information.

Functional Organization of the Ventral Auditory Pathway

The fundamental problem in audition is determining the mechanisms required by the brain to transform an unlabelled mixture of auditory stimuli into coherent perceptual representations. This process is called auditory-scene analysis. The perceptual representations that result from auditory-scene analysis are formed through a complex interaction of perceptual grouping, attention, categorization and decision-making. Despite a great deal of scientific energy devoted to understanding these aspects of hearing, we still do not understand (1) how sound perception arises from neural activity and (2) the causal relationship between neural activity and sound perception. Here, we review the role of the "ventral" auditory pathway in sound perception. We hypothesize that, in the early parts of the auditory cortex, neural activity reflects the auditory properties of a stimulus. However, in latter parts of the auditory cortex, neurons encode the sensory evidence that forms an auditory decision and are causally involved in the decision process. Finally, in the prefrontal cortex, which receives input from the auditory cortex, neural activity reflects the actual perceptual decision. Together, these studies indicate that the ventral pathway contains hierarchical circuits that are specialized for auditory perception and scene analysis.

Neurobiological evidence for voicing underspecification in English

In long-term memory, the phoneme units that make up words are coded for the distinctive features and feature values that are necessary to distinguish between words in the mental lexicon. Underspecification theory says that the phonemes that have unmarked feature values are even more abstract in that the feature is omitted from the representation altogether. This makes phoneme representations in words more sparse than the fully specified phonetic representations of the same words. Eulitz and Lahiri (2004) demonstrated that this theory predicts certain asymmetries in the Mismatch Negativity (MMN) response to phoneme contrasts. We expand on this research by demonstrating underspecification-driven asymmetry in the brain response to laryngeal feature contrasts in English (i.e. what makes /d/ and /t/ different). We add a new test by showing that the asymmetry disappears if the MMN paradigm is modified to encourage the formation of phonetic memory traces instead of phonemic memory traces. This result adds further neurobiological evidence that long-term phonological representations are more sparsely represented than phonetic representations.

Can you hear me yet? An intracranial investigation of speech and non-speech audiovisual interactions in human cortex

In everyday conversation, viewing a talker's face can provide information about the timing and content of an upcoming speech signal, resulting in improved intelligibility. Using electrocorticography, we tested whether human auditory cortex in Heschl's gyrus (HG) and on superior temporal gyrus (STG) and motor cortex on precentral gyrus (PreC) were responsive to visual/gestural information prior to the onset of sound and whether early stages of auditory processing were sensitive to the visual content (speech syllable versus non-speech motion). Event-related band power (ERBP) in the high gamma band was content-specific prior to acoustic onset on STG and PreC, and ERBP in the beta band differed in all three areas. Following sound onset, we found with no evidence for content-specificity in HG, evidence for visual specificity in PreC, and specificity for both modalities in STG. These results support models of audio-visual processing in which sensory information is integrated in non-primary cortical areas.

Using speech and electrocorticography to map human auditory cortex

Much less is known about the organization of the human auditory cortex compared to non-human primate auditory cortices. In an effort to further investigate the response properties of human auditory cortex, we present preliminary findings from human subjects implanted with depth electrodes in Heschl's gyrus (HG) as part of their neurosurgical treatment of epilepsy. Each subject had electrocorticography (ECoG) responses taken from medial and lateral HG in response to both speech and non-speech stimuli, including during speech production. Responses were somewhat variable across subjects, but posteromedial HG demonstrated frequency following responses to the stimuli in all subjects to some degree. Results and implications are discussed.

Sound identification in human auditory cortex: Differential contribution of local field potentials and high gamma power as revealed by direct intracranial recordings

High gamma power has become the principal means of assessing auditory cortical activation in human intracranial studies, albeit at the expense of low frequency local field potentials (LFPs). It is unclear whether limiting analyses to high gamma impedes ability of clarifying auditory cortical organization. We compared the two measures obtained from posterolateral superior temporal gyrus (PLST) and evaluated their relative utility in sound categorization. Subjects were neurosurgical patients undergoing invasive monitoring for medically refractory epilepsy. Stimuli (consonant-vowel syllables varying in voicing and place of articulation and control tones) elicited robust evoked potentials and high gamma activity on PLST. LFPs had greater across-subject variability, yet yielded higher classification accuracy, relative to high gamma power. Classification was enhanced by including temporal detail of LFPs and combining LFP and high gamma. We conclude that future studies should consider utilizing both LFP and high gamma when investigating the functional organization of human auditory cortex.

Dynamic encoding of speech sequence probability in human temporal cortex

Sensory processing involves identification of stimulus features, but also integration with the surrounding sensory and cognitive context. Previous work in animals and humans has shown fine-scale sensitivity to context in the form of learned knowledge about the statistics of the sensory environment, including relative probabilities of discrete units in a stream of sequential auditory input. These statistics are a defining characteristic of one of the most important sequential signals humans encounter: speech. For speech, extensive exposure to a language tunes listeners to the statistics of sound sequences. To address how speech sequence statistics are neurally encoded, we used high-resolution direct cortical recordings from human lateral superior temporal cortex as subjects listened to words and nonwords with varying transition probabilities between sound segments. In addition to their sensitivity to acoustic features (including contextual features, such as coarticulation), we found that neural responses dynamically encoded the language-level probability of both preceding and upcoming speech sounds. Transition probability first negatively modulated neural responses, followed by positive modulation of neural responses, consistent with coordinated predictive and retrospective recognition processes, respectively. Furthermore, transition probability encoding was different for real English words compared with nonwords, providing evidence for online interactions with high-order linguistic knowledge. These results demonstrate that sensory processing of deeply learned stimuli involves integrating physical stimulus features with their contextual sequential structure. Despite not being consciously aware of phoneme sequence statistics, listeners use this information to process spoken input and to link low-level acoustic representations with linguistic information about word identity and meaning.

Direct physiologic evidence of a heteromodal convergence region for proper naming in human left anterior temporal lobe

Retrieving the names of friends, loved ones, and famous people is a fundamental human ability. This ability depends on the left anterior temporal lobe (ATL), where lesions can be associated with impaired naming of people regardless of modality (e.g., picture or voice). This finding has led to the idea that the left ATL is a modality-independent convergence region for proper naming. Hypotheses for how proper-name dispositions are organized within the left ATL include both a single modality-independent (heteromodal) convergence region and spatially discrete modality-dependent (unimodal) regions. Here we show direct electrophysiologic evidence that the left ATL is heteromodal for proper-name retrieval. Using intracranial recordings placed directly on the surface of the left ATL in human subjects, we demonstrate nearly identical responses to picture and voice stimuli of famous U.S. politicians during a naming task. Our results demonstrate convergent and robust large-scale neurophysiologic responses to picture and voice naming in the human left ATL. This finding supports the idea of heteromodal (i.e., transmodal) dispositions for proper naming in the left ATL.

Distributed neural representations of phonological features during speech perception

A fundamental goal of the human auditory system is to map complex acoustic signals onto stable internal representations of the basic sound patterns of speech. Phonemes and the distinctive features that they comprise constitute the basic building blocks from which higher-level linguistic representations, such as words and sentences, are formed. Although the neural structures underlying phonemic representations have been well studied, there is considerable debate regarding frontal-motor cortical contributions to speech as well as the extent of lateralization of phonological representations within auditory cortex. Here we used functional magnetic resonance imaging (fMRI) and multivoxel pattern analysis to investigate the distributed patterns of activation that are associated with the categorical and perceptual similarity structure of 16 consonant exemplars in the English language used in Miller and Nicely's (1955) classic study of acoustic confusability. Participants performed an incidental task while listening to phonemes in the MRI scanner. Neural activity in bilateral anterior superior temporal gyrus and supratemporal plane was correlated with the first two components derived from a multidimensional scaling analysis of a behaviorally derived confusability matrix. We further showed that neural representations corresponding to the categorical features of voicing, manner of articulation, and place of articulation were widely distributed throughout bilateral primary, secondary, and association areas of the superior temporal cortex, but not motor cortex. Although classification of phonological features was generally bilateral, we found that multivariate pattern information was moderately stronger in the left compared with the right hemisphere for place but not for voicing or manner of articulation.

Neural pathways for visual speech perception

This paper examines the questions, what levels of speech can be perceived visually, and how is visual speech represented by the brain? Review of the literature leads to the conclusions that every level of psycholinguistic speech structure (i.e., phonetic features, phonemes, syllables, words, and prosody) can be perceived visually, although individuals differ in their abilities to do so; and that there are visual modality-specific representations of speech qua speech in higher-level vision brain areas. That is, the visual system represents the modal patterns of visual speech. The suggestion that the auditory speech pathway receives and represents visual speech is examined in light of neuroimaging evidence on the auditory speech pathways. We outline the generally agreed-upon organization of the visual ventral and dorsal pathways and examine several types of visual processing that might be related to speech through those pathways, specifically, face and body, orthography, and sign language processing. In this context, we examine the visual speech processing literature, which reveals widespread diverse patterns of activity in posterior temporal cortices in response to visual speech stimuli. We outline a model of the visual and auditory speech pathways and make several suggestions: (1) The visual perception of speech relies on visual pathway representations of speech qua speech. (2) A proposed site of these representations, the temporal visual speech area (TVSA) has been demonstrated in posterior temporal cortex, ventral and posterior to multisensory posterior superior temporal sulcus (pSTS). (3) Given that visual speech has dynamic and configural features, its representations in feedforward visual pathways are expected to integrate these features, possibly in TVSA.

Adaptation of high-gamma responses in human auditory association cortex

This study investigates adaptation of high-frequency cortical responses [>60 Hz; high-gamma (HG)] to simple and complex sounds in human nonprimary auditory cortex. We used intracranial electrocorticographic recordings to measure event-related changes in HG power as a function of stimulus probability. Tone and speech stimuli were presented in a series of traditional oddball and control paradigms. We hypothesized that HG power attenuates with stimulus repetition over multiple concurrent time scales in auditory association cortex. Time-frequency analyses were performed to identify auditory-responsive sites. Single-trial analyses and quantitative modeling were then used to measure trial-to-trial changes in HG power for high (frequent), low (infrequent), and equal (control) stimulus probabilities. Results show strong reduction of HG responses to frequently repeated tones and speech, with no differences in responses to infrequent and equal-probability stimuli. Adaptation of the HG frequent response, and not stimulus-acoustic differences or deviance-detection enhancement effects, accounted for the differential responses observed for frequent and infrequent sounds. Adaptation of HG responses showed a rapid onset (less than two trials) with slower adaptation between consecutive, repeated trials (2-10 s) and across trials in a stimulus block (∼7 min). The auditory-evoked N100 response also showed repetition-related adaptation, consistent with previous human scalp and animal single-unit recordings. These findings indicate that HG responses are highly sensitive to the regularities of simple and complex auditory events and show adaptation on multiple concurrent time scales in human auditory association cortex.

Differential activation of human core, non-core and auditory-related cortex during speech categorization tasks as revealed by intracranial recordings

Speech perception requires that sounds be transformed into speech-related objects with lexical and semantic meaning. It is unclear at what level in the auditory pathways this transformation emerges. Primary auditory cortex has been implicated in both representation of acoustic sound attributes and sound objects. While non-primary auditory cortex located on the posterolateral superior temporal gyrus (PLST) is clearly involved in acoustic-to-phonetic pre-lexical representations, it is unclear what role this region plays in auditory object formation. Additional data support the importance of prefrontal cortex in the formation of auditory objects, while other data would implicate this region in auditory object selection. To help clarify the respective roles of auditory and auditory-related cortex in the formation and selection of auditory objects, we examined high gamma activity simultaneously recorded directly from Heschl's gyrus (HG), PLST and prefrontal cortex, while subjects performed auditory semantic detection tasks. Subjects were patients undergoing evaluation for treatment of medically intractable epilepsy. We found that activity in posteromedial HG and early activity on PLST was robust to sound stimuli regardless of their context, and minimally modulated by tasks. Later activity on PLST could be strongly modulated by semantic context, but not by behavioral performance. Activity within prefrontal cortex also was related to semantic context, and did co-vary with behavior. We propose that activity in posteromedial HG and early activity on PLST primarily reflect the representation of spectrotemporal sound attributes. Later activity on PLST represents a pre-lexical processing stage and is an intermediate step in the formation of word objects. Activity in prefrontal cortex appears directly involved in word object selection. The roles of other auditory and auditory-related cortical areas in the formation of word objects remain to be explored.

Functional organization of human auditory cortex: investigation of response latencies through direct recordings

The model for functional organization of human auditory cortex is in part based on findings in non-human primates, where the auditory cortex is hierarchically delineated into core, belt and parabelt fields. This model envisions that core cortex directly projects to belt, but not to parabelt, whereas belt regions are a major source of direct input for auditory parabelt. In humans, the posteromedial portion of Heschl's gyrus (HG) represents core auditory cortex, whereas the anterolateral portion of HG and the posterolateral superior temporal gyrus (PLST) are generally interpreted as belt and parabelt, respectively. In this scheme, response latencies can be hypothesized to progress in serial fashion from posteromedial to anterolateral HG to PLST. We examined this hypothesis by comparing response latencies to multiple stimuli, measured across these regions using simultaneous intracranial recordings in neurosurgical patients. Stimuli were 100 Hz click trains and the speech syllable /da/. Response latencies were determined by examining event-related band power in the high gamma frequency range. The earliest responses in auditory cortex occurred in posteromedial HG. Responses elicited from sites in anterolateral HG were neither earlier in latency from sites on PLST, nor more robust. Anterolateral HG and PLST exhibited some preference for speech syllable stimuli compared to click trains. These findings are not supportive of a strict serial model envisioning principal flow of information along HG to PLST. In contrast, data suggest that a portion of PLST may represent a relatively early stage in the auditory cortical hierarchy.

Mapping phonemic processing zones along human perisylvian cortex: an electro-corticographic investigation

The auditory system is organized such that progressively more complex features are represented across successive cortical hierarchical stages. Just when and where the processing of phonemes, fundamental elements of the speech signal, is achieved in this hierarchy remains a matter of vigorous debate. Non-invasive measures of phonemic representation have been somewhat equivocal. While some studies point to a primary role for middle/anterior regions of the superior temporal gyrus (STG), others implicate the posterior STG. Differences in stimulation, task and inter-individual anatomical/functional variability may account for these discrepant findings. Here, we sought to clarify this issue by mapping phonemic representation across left perisylvian cortex, taking advantage of the excellent sampling density afforded by intracranial recordings in humans. We asked whether one or both major divisions of the STG were sensitive to phonemic transitions. The high signal-to-noise characteristics of direct intracranial recordings allowed for analysis at the individual participant level, circumventing issues of inter-individual anatomic and functional variability that may have obscured previous findings at the group level of analysis. The mismatch negativity (MMN), an electrophysiological response elicited by changes in repetitive streams of stimulation, served as our primary dependent measure. Oddball configurations of pairs of phonemes, spectro-temporally matched non-phonemes, and simple tones were presented. The loci of the MMN clearly differed as a function of stimulus type. Phoneme representation was most robust over middle/anterior STG/STS, but was also observed over posterior STG/SMG. These data point to multiple phonemic processing zones along perisylvian cortex, both anterior and posterior to primary auditory cortex. This finding is considered within the context of a dual stream model of auditory processing in which functionally distinct ventral and dorsal auditory processing pathways may be engaged by speech stimuli.

Neural mechanisms of auditory categorization: from across brain areas to within local microcircuits

Categorization enables listeners to efficiently encode and respond to auditory stimuli. Behavioral evidence for auditory categorization has been well documented across a broad range of human and non-human animal species. Moreover, neural correlates of auditory categorization have been documented in a variety of different brain regions in the ventral auditory pathway, which is thought to underlie auditory-object processing and auditory perception. Here, we review and discuss how neural representations of auditory categories are transformed across different scales of neural organization in the ventral auditory pathway: from across different brain areas to within local microcircuits. We propose different neural transformations across different scales of neural organization in auditory categorization. Along the ascending auditory system in the ventral pathway, there is a progression in the encoding of categories from simple acoustic categories to categories for abstract information. On the other hand, in local microcircuits, different classes of neurons differentially compute categorical information.