Rhythmic (hierarchical) versus serial structure in speech and other behavior

Listening Effort for Speech in Noise Perception Using Pupil Dilation: A Comparison Among Percussionists, Non-Percussionists, and Non-Musicians

BACKGROUND AND OBJECTIVES

Most studies in literature attribute the benefits of musical training on speech in noise (SIN) perception to "experience-based" plasticity, which assists in the activation of speech-processing networks. However, whether musicianship provides an advantage for the listening effort (LE) required to comprehend speech in degraded environments has received less attention. The current study aimed to understand the influence of Indian classical music training on SIN perception and its related LE across percussionists, non-percussionists, and non-musicians.

SUBJECTS AND METHODS

A quasi-experiment was conducted on 16 percussionists, 17 non-percussionists, and 26 non-musicians aged 18-35 years with normal hearing. In phase 1, musical abilities were assessed using Mini-Profile of Music Perception Skills (Mini-PROMS). Phase 2 examined SIN using Tamil Phonemically-Balanced Words and Tamil Matrix Sentence Test at +5 dB, 0 dB, and -5 dB SNR and LE using pupillometry, measuring pupil dilations with an eye-tracker.

RESULTS

Fractional Logit and Linear Regression models demonstrated that percussionists outperformed non-percussionists in Tuning and Speed subsets of Mini-PROMS. Percussionists outperformed non-percussionists and non-musicians in SIN and LE at -5 dB SNR for words and at 0 dB and -5 dB SNR for sentences.

CONCLUSIONS

Percussionists have the greatest advantage in decoding SIN with reduced LE followed by non-percussionists and non-musicians, demonstrating musician-advantage in most challenging listening conditions.

Scale in Language

A central concern of the cognitive science of language since its origins has been the concept of the linguistic system. Recent approaches to the system concept in language point to the exceedingly complex relations that hold between many kinds of interdependent systems, but it can be difficult to know how to proceed when "everything is connected." This paper offers a framework for tackling that challenge by identifying *scale* as a conceptual mooring for the interdisciplinary study of language systems. The paper begins by defining the scale concept-simply, the possibility for a measure to be larger or smaller in different instances of a system, such as a phonemic inventory, a word's frequency value in a corpus, or a speaker population. We review sites of scale difference in and across linguistic subsystems, drawing on findings from linguistic typology, grammatical description, morphosyntactic theory, psycholinguistics, computational corpus work, and social network demography. We consider possible explanations for scaling differences and constraints in language. We then turn to the question of *dependencies between* sites of scale difference in language, reviewing four sample domains of scale dependency: in phonological systems, across levels of grammatical structure (Menzerath's Law), in corpora (Zipf's Law and related issues), and in speaker population size. Finally, we consider the implications of the review, including the utility of a scale framework for generating new questions and inspiring methodological innovations and interdisciplinary collaborations in cognitive-scientific research on language.

Differential sensitivity to speech rhythms in young and older adults

Sensitivity to the temporal properties of auditory patterns tends to be poorer in older listeners, and this has been hypothesized to be one factor contributing to their poorer speech understanding. This study examined sensitivity to speech rhythms in young and older normal-hearing subjects, using a task designed to measure the effect of speech rhythmic context on the detection of changes in the timing of word onsets in spoken sentences. A temporal-shift detection paradigm was used in which listeners were presented with an intact sentence followed by two versions of the sentence in which a portion of speech was replaced with a silent gap: one with correct gap timing (the same duration as the missing speech) and one with altered gap timing (shorter or longer than the duration of the missing speech), resulting in an early or late resumption of the sentence after the gap. The sentences were presented with either an intact rhythm or an altered rhythm preceding the silent gap. Listeners judged which sentence had the altered gap timing, and thresholds for the detection of deviations from the correct timing were calculated separately for shortened and lengthened gaps. Both young and older listeners demonstrated lower thresholds in the intact rhythm condition than in the altered rhythm conditions. However, shortened gaps led to lower thresholds than lengthened gaps for the young listeners, while older listeners were not sensitive to the direction of the change in timing. These results show that both young and older listeners rely on speech rhythms to generate temporal expectancies for upcoming speech events. However, the absence of lower thresholds for shortened gaps among the older listeners indicates a change in speech-timing expectancies with age. A further examination of individual differences within the older group revealed that those with better rhythm-discrimination abilities (from a separate study) tended to show the same heightened sensitivity to early events observed with the young listeners.

Children's Neural Sensitivity to Prosodic Features of Natural Speech and Its Significance to Speech Development in Cochlear Implanted Children

Catchy utterances, such as proverbs, verses, and nursery rhymes (i.e., "No pain, no gain" in English), contain strong-prosodic (SP) features and are child-friendly in repeating and memorizing; yet the way those prosodic features encoded by neural activity and their influence on speech development in children are still largely unknown. Using functional near-infrared spectroscopy (fNIRS), this study investigated the cortical responses to the perception of natural speech sentences with strong/weak-prosodic (SP/WP) features and evaluated the speech communication ability in 21 pre-lingually deaf children with cochlear implantation (CI) and 25 normal hearing (NH) children. A comprehensive evaluation of speech communication ability was conducted on all the participants to explore the potential correlations between neural activities and children's speech development. The SP information evoked right-lateralized cortical responses across a broad brain network in NH children and facilitated the early integration of linguistic information, highlighting children's neural sensitivity to natural SP sentences. In contrast, children with CI showed significantly weaker cortical activation and characteristic deficits in speech perception with SP features, suggesting hearing loss at the early age of life, causing significantly impaired sensitivity to prosodic features of sentences. Importantly, the level of neural sensitivity to SP sentences was significantly related to the speech behaviors of all children participants. These findings demonstrate the significance of speech prosodic features in children's speech development.

Localizing category-related information in speech with multi-scale analyses

Measurements of the physical outputs of speech-vocal tract geometry and acoustic energy-are high-dimensional, but linguistic theories posit a low-dimensional set of categories such as phonemes and phrase types. How can it be determined when and where in high-dimensional articulatory and acoustic signals there is information related to theoretical categories? For a variety of reasons, it is problematic to directly quantify mutual information between hypothesized categories and signals. To address this issue, a multi-scale analysis method is proposed for localizing category-related information in an ensemble of speech signals using machine learning algorithms. By analyzing how classification accuracy on unseen data varies as the temporal extent of training input is systematically restricted, inferences can be drawn regarding the temporal distribution of category-related information. The method can also be used to investigate redundancy between subsets of signal dimensions. Two types of theoretical categories are examined in this paper: phonemic/gestural categories and syntactic relative clause categories. Moreover, two different machine learning algorithms were examined: linear discriminant analysis and neural networks with long short-term memory units. Both algorithms detected category-related information earlier and later in signals than would be expected given standard theoretical assumptions about when linguistic categories should influence speech. The neural network algorithm was able to identify category-related information to a greater extent than the discriminant analyses.

Linguistic syncopation: Meter-syntax alignment affects sentence comprehension and sensorimotor synchronization

The hierarchical organization of speech rhythm into meter putatively confers cognitive affordances for perception, memory, and motor coordination. Meter also aligns with phrasal structure in systematic ways. In this paper, we show that this alignment affects the robustness of syntactic comprehension and discuss possible underlying mechanisms. In two experiments, we manipulated meter-syntax alignment while sentences with relative clause structures were either read as text (experiment 1, n = 40) or listened to as speech (experiment 2, n = 40). In experiment 2, we also measured the stability with which participants could tap in time with the metrical accents in the sentences they were comprehending. In addition to making more mistakes, sensorimotor synchronization was disrupted when syntactic cues clashed with the metrical context. We suggest that this reflects a tight coordination of top-down linguistic knowledge with the sensorimotor system to optimize comprehension.

The three second time window in poems and language processing in general: Complementarity of discrete timing and temporal continuity

The existence of discrete time windows has triggered the search for permanence and continuity for artists (including poets) in multiple cultures throughout history. In this article, we argue that there exists a 3-s window in the temporal structure of poems as well as in the aesthetic appreciation of poetry by reviewing previous literature on the temporal aspects of poems. This 3-s window can also be considered to be a general temporal machinery underlying human behavior, including language production and perception in general. The reafference principle has provided us a unique frame for understanding cognitive processes. However, "time" was absent in the original two-stage reafference principle. Therefore, we propose a three-stage cycling model of language perception, taking into account time and time windows. We also inspect the possible neural implementations of the three stages: the generation, maintenance, and comparison of predictions (as well as the integration of predictions into the representational context). These three stages are embedded in a temporal window of ~3 s and are repeated in a cycling mode, resulting in the representational context being continuously updated. Thus, it is possible that "semantics" could be carried forward across different time windows, being a "glue" linking the discrete time windows and thus achieving the subjective feeling of temporal continuity. Candidates of such "semantic glue" could include semantic and syntactic structures as well as identity and emotion.

Rhythm in speech and animal vocalizations: a cross-species perspective

Why does human speech have rhythm? As we cannot travel back in time to witness how speech developed its rhythmic properties and why humans have the cognitive skills to process them, we rely on alternative methods to find out. One powerful tool is the comparative approach: studying the presence or absence of cognitive/behavioral traits in other species to determine which traits are shared between species and which are recent human inventions. Vocalizations of many species exhibit temporal structure, but little is known about how these rhythmic structures evolved, are perceived and produced, their biological and developmental bases, and communicative functions. We review the literature on rhythm in speech and animal vocalizations as a first step toward understanding similarities and differences across species. We extend this review to quantitative techniques that are useful for computing rhythmic structure in acoustic sequences and hence facilitate cross-species research. We report links between vocal perception and motor coordination and the differentiation of rhythm based on hierarchical temporal structure. While still far from a complete cross-species perspective of speech rhythm, our review puts some pieces of the puzzle together.

Motor Reproduction of Time Interval Depends on Internal Temporal Cues in the Brain: Sensorimotor Imagery in Rhythm

How the human brain perceives time intervals is a fascinating topic that has been explored in many fields of study. This study examined how time intervals are replicated in three conditions: with no internalized cue (PT), with an internalized cue without a beat (AS), and with an internalized cue with a beat (RS). In PT, participants accurately reproduced the time intervals up to approximately 3 s. Over 3 s, however, the reproduction errors became increasingly negative. In RS, longer presentations of over 5.6 s and 13 beats induced accurate time intervals in reproductions. This suggests longer exposure to beat presentation leads to stable internalization and efficiency in the sensorimotor processing of perception and reproduction. In AS, up to approximately 3 s, the results were similar to those of RS whereas over 3 s, the results shifted and became similar to those of PT. The time intervals between the first two stimuli indicate that the strategies of time-interval reproduction in AS may shift from RS to PT. Neural basis underlying the reproduction of time intervals without a beat may depend on length of time interval between adjacent stimuli in sequences.

Hierarchical temporal structure in music, speech and animal vocalizations: jazz is like a conversation, humpbacks sing like hermit thrushes

Humans talk, sing and play music. Some species of birds and whales sing long and complex songs. All these behaviours and sounds exhibit hierarchical structure-syllables and notes are positioned within words and musical phrases, words and motives in sentences and musical phrases, and so on. We developed a new method to measure and compare hierarchical temporal structures in speech, song and music. The method identifies temporal events as peaks in the sound amplitude envelope, and quantifies event clustering across a range of timescales using Allan factor (AF) variance. AF variances were analysed and compared for over 200 different recordings from more than 16 different categories of signals, including recordings of speech in different contexts and languages, musical compositions and performances from different genres. Non-human vocalizations from two bird species and two types of marine mammals were also analysed for comparison. The resulting patterns of AF variance across timescales were distinct to each of four natural categories of complex sound: speech, popular music, classical music and complex animal vocalizations. Comparisons within and across categories indicated that nested clustering in longer timescales was more prominent when prosodic variation was greater, and when sounds came from interactions among individuals, including interactions between speakers, musicians, and even killer whales. Nested clustering also was more prominent for music compared with speech, and reflected beat structure for popular music and self-similarity across timescales for classical music. In summary, hierarchical temporal structures reflect the behavioural and social processes underlying complex vocalizations and musical performances.

The Temporal Prediction of Stress in Speech and Its Relation to Musical Beat Perception

While rhythmic expectancies are thought to be at the base of beat perception in music, the extent to which stress patterns in speech are similarly represented and predicted during on-line language comprehension is debated. The temporal prediction of stress may be advantageous to speech processing, as stress patterns aid segmentation and mark new information in utterances. However, while linguistic stress patterns may be organized into hierarchical metrical structures similarly to musical meter, they do not typically present the same degree of periodicity. We review the theoretical background for the idea that stress patterns are predicted and address the following questions: First, what is the evidence that listeners can predict the temporal location of stress based on preceding rhythm? If they can, is it thanks to neural entrainment mechanisms similar to those utilized for musical beat perception? And lastly, what linguistic factors other than rhythm may account for the prediction of stress in natural speech? We conclude that while expectancies based on the periodic presentation of stresses are at play in some of the current literature, other processes are likely to affect the prediction of stress in more naturalistic, less isochronous speech. Specifically, aspects of prosody other than amplitude changes (e.g., intonation) as well as lexical, syntactic and information structural constraints on the realization of stress may all contribute to the probabilistic expectation of stress in speech.

Learning of hierarchical serial patterns emerges in infancy

Recursive, hierarchically organized serial patterns provide the underlying structure in many cognitive and motor domains including speech, language, music, social interaction, and motor action. We investigated whether learning of hierarchical patterns emerges in infancy by habituating 204 infants to different hierarchical serial patterns and then testing for discrimination and generalization of such patterns. Results indicated that 8- to 10-month-old and 12- to 14-month-old infants exhibited sensitivity to the difference between hierarchical and non-hierarchical structure but that 4- to 6-month-old infants did not. These findings demonstrate that the ability to perceive, learn, and generalize recursive, hierarchical, pattern rules emerges in infancy and add to growing evidence that general-purpose pattern learning mechanisms emerge during the first year of life.

The Communication of Musical Metre in Piano Performance

Six pianists of varying levels of experience were required to give repeated performances of a note sequence presented for sight-performance under two conditions. The two conditions differed only in the location of metrical stresses signified by the notation. In condition (b) the bar-lines and beams were shifted one note to the right in comparison to condition (a). Details of note duration and intensity were recorded by computer. This allowed the discovery of significant differences in expressive treatment between the two conditions. There were significant agreements between subjects concerning the position, nature and direction of expressive variation. The more experienced players, however, made greater use of expressive variation than did the less experienced players. In a second experiment listeners were asked to identify the metre of each performance from the first experiment. They achieved greatest success at this with the most experienced player, least success with the least experienced player. A detailed examination of between-performer agreements and disagreements, and their effects on listeners, allowed the isolation of a proposed set of generally effective procedures for signalling stress. These procedures imply the existence of an internal representation guiding performance which identifies the major metrical subdivisions of a bar in the absence of notational symbols specifically marking these subdivisions.

Recognizing Melodies: A Dynamic Interpretation

Two experiments explore hypotheses about rhythm and contour in recognition of simple pitch strings (melodies). Target melodies that differed with respect to pitch relationships (interval and contour pitch differences) and rhythm, were presented to ordinary listeners who were told to learn the melodies (Phase I). In a subsequent recognition test (Phase II), listeners had to recognize these same target melodies although they were transposed to a different musical key. In recognition, target melodies appeared in the original rhythm or in new rhythms that simulated some pause properties of the original rhythm. Target melodies were interspersed with decoy melodies that either preserved the pitch contour of targets or did not; all appeared in the original rhythm and in new rhythms. Results indicated that a new rhythmic context lowered recognizability of target melodies, and that decoys were most confusing when they possessed the same “dynamic shape” (contour-plus-rhythm) as targets (Experiment 1). Also, target recognition improved with Phase I familiarity (Experiment 2), although rhythmic shifts remained detrimental across levels of target familiarity. Confusions based on “dynamic shape” accounted for a relatively high proportion of errors where familiarity with targets is low. Findings were interpreted in terms of a theory of context-sensitive dynamic attending in which remembering is assumed to involve recapitulation of the original rhythmical activities involved in attending to melodies.

Rhythmic facilitation of sensory processing: A critical review

Here we review the role of brain oscillations in sensory processing. We examine the idea that neural entrainment of intrinsic oscillations underlies the processing of rhythmic stimuli in the context of simple isochronous rhythms as well as in music and speech. This has been a topic of growing interest over recent years; however, many issues remain highly controversial: how do fluctuations of intrinsic neural oscillations-both spontaneous and entrained to external stimuli-affect perception, and does this occur automatically or can it be actively controlled by top-down factors? Some of the controversy in the literature stems from confounding use of terminology. Moreover, it is not straightforward how theories and findings regarding isochronous rhythms generalize to more complex, naturalistic stimuli, such as speech and music. Here we aim to clarify terminology, and distinguish between different phenomena that are often lumped together as reflecting "neural entrainment" but may actually vary in their mechanistic underpinnings. Furthermore, we discuss specific caveats and confounds related to making inferences about oscillatory mechanisms from human electrophysiological data.

Incorporation of feedback during beat synchronization is an index of neural maturation and reading skills

Speech communication involves integration and coordination of sensory perception and motor production, requiring precise temporal coupling. Beat synchronization, the coordination of movement with a pacing sound, can be used as an index of this sensorimotor timing. We assessed adolescents' synchronization and capacity to correct asynchronies when given online visual feedback. Variability of synchronization while receiving feedback predicted phonological memory and reading sub-skills, as well as maturation of cortical auditory processing; less variable synchronization during the presence of feedback tracked with maturation of cortical processing of sound onsets and resting gamma activity. We suggest the ability to incorporate feedback during synchronization is an index of intentional, multimodal timing-based integration in the maturing adolescent brain. Precision of temporal coding across modalities is important for speech processing and literacy skills that rely on dynamic interactions with sound. Synchronization employing feedback may prove useful as a remedial strategy for individuals who struggle with timing-based language learning impairments.

On the Acquisition of Reading Fluency

The acquisition of reading fluency crucially involves the beginning reader's tacit recognition that s/he must learn to compensate for the absence of graphic signals corresponding to certain prosodie cues by making better use of the morphological and syntactic cues that are preserved. It is argued that the success of the method of repeated readings and similar reading instruction techniques results from the fact that these methods facilitate discovery of the appropriate syntactic phrasing in the written signal. It is suggested that the crucial step comes with the beginning reader's recognition that parsing strategies other than those which rely on prosody or its somewhat haphazard graphic analogues are required in order to read with sense.

Visual Rhythms: Report on a Method for Facilitating the Teaching of Reading

Sentences were prepared such that they appeared on a TV screen with each syllable timed, syllable-by-syllable, as though it were spoken. This was accomplished by synchronizing the onset timing of each syllable as it appeared on the screen with the onset timing of the same syllable as it was heard through the speaker. The result was a sentence which “grew” left-to-right across the screen and which could be presented either visually by itself or in combination with its auditory counterpart. Primary-school children were exposed in training sessions to these sentences in “visual rhythm.” Their pre-to-post-experimental change in reading fluency was compared with that of other children in a control condition whose training was the same except that the sentences they saw were not rhythmic, but static. Positive results encouraged the discussion and rationale for further applications of the method.

Structural Analysis of Post-Tonal Therapeutic Improvisatory Music

The analytical investigation examining tonal therapeutic improvisatory music (Lee, C. A., 1989) would seem to indicate that music therapy should be exploring its own investigative form of analysis. The adoption of a Schenkerian construct of analysis accommodates music therapy perhaps more clearly than the often complex statistics and pitch classes that, for the most part, concern the analysis of atonal music. The tonal infrastructure can normally be described in direct relation to its harmonic foundation. With atonality, however, such reference points are not easily available, except in serialism or similar theoretical forms of music. It is the intention of this article to investigate the diverse constructs of post-tonal analysis and consider their validity in exploring atonal issues of therapeutic improvisation. Initial focus will be on formal analysis of the improvisation, interspersed with a more informal subjective framework of enquiry endeavouring to evaluate relationships between both areas.

The passage of improvisation selected is taken from a different session with the same client, thus enabling various parallels to be drawn.

Hierarchical Disability in the Syntax of Aphasic Children

The writings by a group of aphasic children in a standardised situation were compared with those of a group of deaf children. In spite of writing sentences of comparable length, the aphasic children produced a more restricted range of sentence types than did the deaf children. This was particularly noticeable in their relative failure to use structures requiring embedding. It is hypothesised that these aphasic children have difficulties with hierarchically ordered material. This lack of hierarchical planning ability has been noted in other tasks.

A multimodal spectral approach to characterize rhythm in natural speech

Human utterances demonstrate temporal patterning, also referred to as rhythm. While simple oromotor behaviors (e.g., chewing) feature a salient periodical structure, conversational speech displays a time-varying quasi-rhythmic pattern. Quantification of periodicity in speech is challenging. Unimodal spectral approaches have highlighted rhythmic aspects of speech. However, speech is a complex multimodal phenomenon that arises from the interplay of articulatory, respiratory, and vocal systems. The present study addressed the question of whether a multimodal spectral approach, in the form of coherence analysis between electromyographic (EMG) and acoustic signals, would allow one to characterize rhythm in natural speech more efficiently than a unimodal analysis. The main experimental task consisted of speech production at three speaking rates; a simple oromotor task served as control. The EMG-acoustic coherence emerged as a sensitive means of tracking speech rhythm, whereas spectral analysis of either EMG or acoustic amplitude envelope alone was less informative. Coherence metrics seem to distinguish and highlight rhythmic structure in natural speech.

The role of rhythm in perceiving speech in noise: a comparison of percussionists, vocalists and non-musicians

The natural rhythms of speech help a listener follow what is being said, especially in noisy conditions. There is increasing evidence for links between rhythm abilities and language skills; however, the role of rhythm-related expertise in perceiving speech in noise is unknown. The present study assesses musical competence (rhythmic and melodic discrimination), speech-in-noise perception and auditory working memory in young adult percussionists, vocalists and non-musicians. Outcomes reveal that better ability to discriminate rhythms is associated with better sentence-in-noise (but not words-in-noise) perception across all participants. These outcomes suggest that sensitivity to rhythm helps a listener understand unfolding speech patterns in degraded listening conditions, and that observations of a "musician advantage" for speech-in-noise perception may be mediated in part by superior rhythm skills.

Evidence for a rhythm perception deficit in children who stutter

Stuttering is a neurodevelopmental disorder that affects the timing and rhythmic flow of speech production. When speech is synchronized with an external rhythmic pacing signal (e.g., a metronome), even severe stuttering can be markedly alleviated, suggesting that people who stutter may have difficulty generating an internal rhythm to pace their speech. To investigate this possibility, children who stutter and typically-developing children (n=17 per group, aged 6-11 years) were compared in terms of their auditory rhythm discrimination abilities of simple and complex rhythms. Children who stutter showed worse rhythm discrimination than typically-developing children. These findings provide the first evidence of impaired rhythm perception in children who stutter, supporting the conclusion that developmental stuttering may be associated with a deficit in rhythm processing.

Multiple factors are involved in the dysarthria associated with Parkinson's disease: a review with implications for clinical practice and research

PURPOSE

Motor speech abnormalities are highly common and debilitating in individuals with idiopathic Parkinson's disease (IPD). These abnormalities, collectively termed hypokinetic dysarthria (HKD), have been traditionally attributed to hypokinesia and bradykinesia secondary to muscle rigidity and dopamine deficits. However, the role of rigidity and dopamine in the development of HKD is far from clear. The purpose of the present study was to offer an alternative view of the factors underlying HKD.

METHOD

The authors conducted an extensive, but not exhaustive, review of the literature to examine the evidence for the traditional view versus the alternative view.

RESULTS

The review suggests that HKD is a highly complex and variable phenomenon including multiple factors, such as scaling and maintaining movement amplitude and effort; preplanning and initiation of movements; internal cueing; sensory and temporal processing; automaticity; emotive vocalization; and attention to action (vocal vigilance). Although not part of the dysarthria, nonmotor factors, such as depression, aging, and cognitive-linguistic abnormalities, are likely to contribute to the overall speech symptomatology associated with IPD.

CONCLUSION

These findings have important implications for clinical practice and research.

Behavioral and multimodal neuroimaging evidence for a deficit in brain timing networks in stuttering: a hypothesis and theory

The fluent production of speech requires accurately timed movements. In this article, we propose that a deficit in brain timing networks is one of the core neurophysiological deficits in stuttering. We first discuss the experimental evidence supporting the involvement of the basal ganglia and supplementary motor area (SMA) in stuttering and the involvement of the cerebellum as a possible mechanism for compensating for the neural deficits that underlie stuttering. Next, we outline the involvement of the right inferior frontal gyrus (IFG) as another putative compensatory locus in stuttering and suggest a role for this structure in an expanded core timing-network. Subsequently, we review behavioral studies of timing in people who stutter and examine their behavioral performance as compared to people who do not stutter. Finally, we highlight challenges to existing research and provide avenues for future research with specific hypotheses.

Toward a computational framework for cognitive biology: unifying approaches from cognitive neuroscience and comparative cognition

Progress in understanding cognition requires a quantitative, theoretical framework, grounded in the other natural sciences and able to bridge between implementational, algorithmic and computational levels of explanation. I review recent results in neuroscience and cognitive biology that, when combined, provide key components of such an improved conceptual framework for contemporary cognitive science. Starting at the neuronal level, I first discuss the contemporary realization that single neurons are powerful tree-shaped computers, which implies a reorientation of computational models of learning and plasticity to a lower, cellular, level. I then turn to predictive systems theory (predictive coding and prediction-based learning) which provides a powerful formal framework for understanding brain function at a more global level. Although most formal models concerning predictive coding are framed in associationist terms, I argue that modern data necessitate a reinterpretation of such models in cognitive terms: as model-based predictive systems. Finally, I review the role of the theory of computation and formal language theory in the recent explosion of comparative biological research attempting to isolate and explore how different species differ in their cognitive capacities. Experiments to date strongly suggest that there is an important difference between humans and most other species, best characterized cognitively as a propensity by our species to infer tree structures from sequential data. Computationally, this capacity entails generative capacities above the regular (finite-state) level; implementationally, it requires some neural equivalent of a push-down stack. I dub this unusual human propensity "dendrophilia", and make a number of concrete suggestions about how such a system may be implemented in the human brain, about how and why it evolved, and what this implies for models of language acquisition. I conclude that, although much remains to be done, a neurally-grounded framework for theoretical cognitive science is within reach that can move beyond polarized debates and provide a more adequate theoretical future for cognitive biology.

Impaired extraction of speech rhythm from temporal modulation patterns in speech in developmental dyslexia

Dyslexia is associated with impaired neural representation of the sound structure of words (phonology). The “phonological deficit” in dyslexia may arise in part from impaired speech rhythm perception, thought to depend on neural oscillatory phase-locking to slow amplitude modulation (AM) patterns in the speech envelope. Speech contains AM patterns at multiple temporal rates, and these different AM rates are associated with phonological units of different grain sizes, e.g., related to stress, syllables or phonemes. Here, we assess the ability of adults with dyslexia to use speech AMs to identify rhythm patterns (RPs). We study 3 important temporal rates: “Stress” (~2 Hz), “Syllable” (~4 Hz) and “Sub-beat” (reduced syllables, ~14 Hz). 21 dyslexics and 21 controls listened to nursery rhyme sentences that had been tone-vocoded using either single AM rates from the speech envelope (Stress only, Syllable only, Sub-beat only) or pairs of AM rates (Stress + Syllable, Syllable + Sub-beat). They were asked to use the acoustic rhythm of the stimulus to identity the original nursery rhyme sentence. The data showed that dyslexics were significantly poorer at detecting rhythm compared to controls when they had to utilize multi-rate temporal information from pairs of AMs (Stress + Syllable or Syllable + Sub-beat). These data suggest that dyslexia is associated with a reduced ability to utilize AMs <20 Hz for rhythm recognition. This perceptual deficit in utilizing AM patterns in speech could be underpinned by less efficient neuronal phase alignment and cross-frequency neuronal oscillatory synchronization in dyslexia. Dyslexics' perceptual difficulties in capturing the full spectro-temporal complexity of speech over multiple timescales could contribute to the development of impaired phonological representations for words, the cognitive hallmark of dyslexia across languages.

Assessment of rhythmic entrainment at multiple timescales in dyslexia: evidence for disruption to syllable timing

Developmental dyslexia is associated with rhythmic difficulties, including impaired perception of beat patterns in music and prosodic stress patterns in speech. Spoken prosodic rhythm is cued by slow (<10 Hz) fluctuations in speech signal amplitude. Impaired neural oscillatory tracking of these slow amplitude modulation (AM) patterns is one plausible source of impaired rhythm tracking in dyslexia. Here, we characterise the temporal profile of the dyslexic rhythm deficit by examining rhythmic entrainment at multiple speech timescales. Adult dyslexic participants completed two experiments aimed at testing the perception and production of speech rhythm. In the perception task, participants tapped along to the beat of 4 metrically-regular nursery rhyme sentences. In the production task, participants produced the same 4 sentences in time to a metronome beat. Rhythmic entrainment was assessed using both traditional rhythmic indices and a novel AM-based measure, which utilised 3 dominant AM timescales in the speech signal each associated with a different phonological grain-sized unit (0.9-2.5 Hz, prosodic stress; 2.5-12 Hz, syllables; 12-40 Hz, phonemes). The AM-based measure revealed atypical rhythmic entrainment by dyslexic participants to syllable patterns in speech, in perception and production. In the perception task, both groups showed equally strong phase-locking to Syllable AM patterns, but dyslexic responses were entrained to a significantly earlier oscillatory phase angle than controls. In the production task, dyslexic utterances showed shorter syllable intervals, and differences in Syllable:Phoneme AM cross-frequency synchronisation. Our data support the view that rhythmic entrainment at slow (∼5 Hz, Syllable) rates is atypical in dyslexia, suggesting that neural mechanisms for syllable perception and production may also be atypical. These syllable timing deficits could contribute to the atypical development of phonological representations for spoken words, the central cognitive characteristic of developmental dyslexia across languages.

Neural underpinnings of music: the polyrhythmic brain

Musical rhythm, consisting of apparently abstract intervals of accented temporal events, has the remarkable ability to move our minds and bodies. Why do certain rhythms make us want to tap our feet, bop our heads or even get up and dance? And how does the brain process rhythmically complex rhythms during our experiences of music? In this chapter, we describe some common forms of rhythmic complexity in music and propose that the theory of predictive coding can explain how rhythm and rhythmic complexity are processed in the brain. We also consider how this theory may reveal why we feel so compelled by rhythmic tension in music. First, musical-theoretical and neuroscientific frameworks of rhythm are presented, in which rhythm perception is conceptualized as an interaction between what is heard ('rhythm') and the brain's anticipatory structuring of music ('the meter'). Second, three different examples of tension between rhythm and meter in music are described: syncopation, polyrhythm and groove. Third, we present the theory of predictive coding of music, which posits a hierarchical organization of brain responses reflecting fundamental, survival-related mechanisms associated with predicting future events. According to this theory, perception and learning is manifested through the brain's Bayesian minimization of the error between the input to the brain and the brain's prior expectations. Fourth, empirical studies of neural and behavioral effects of syncopation, polyrhythm and groove will be reported, and we propose how these studies can be seen as special cases of the predictive coding theory. Finally, we argue that musical rhythm exploits the brain's general principles of anticipation and propose that pleasure from musical rhythm may be a result of such anticipatory mechanisms.

Rhythmic cognition in humans and animals: distinguishing meter and pulse perception

This paper outlines a cognitive and comparative perspective on human rhythmic cognition that emphasizes a key distinction between pulse perception and meter perception. Pulse perception involves the extraction of a regular pulse or "tactus" from a stream of events. Meter perception involves grouping of events into hierarchical trees with differing levels of "strength", or perceptual prominence. I argue that metrically-structured rhythms are required to either perform or move appropriately to music (e.g., to dance). Rhythms, from this metrical perspective, constitute "trees in time." Rhythmic syntax represents a neglected form of musical syntax, and warrants more thorough neuroscientific investigation. The recent literature on animal entrainment clearly demonstrates the capacity to extract the pulse from rhythmic music, and to entrain periodic movements to this pulse, in several parrot species and a California sea lion, and a more limited ability to do so in one chimpanzee. However, the ability of these or other species to infer hierarchical rhythmic trees remains, for the most part, unexplored (with some apparent negative results from macaques). The results from this animal comparative research, combined with new methods to explore rhythmic cognition neurally, provide exciting new routes for understanding not just rhythmic cognition, but hierarchical cognition more generally, from a biological and neural perspective.

Development of ordinal sequence perception in infancy

Perception of the ordinal position of a sequence element is critical to many cognitive and motor functions. Here, the prediction that this ability is based on a domain-general perceptual mechanism and, thus, that it emerges prior to the emergence of language was tested. Infants were habituated with sequences of moving/sounding objects and then tested for the ability to perceive the invariant ordinal position of a single element (Experiment 1) or the invariant relative ordinal position of two adjacent elements (Experiment 2). Experiment 1 tested 4- and 6-month-old infants and showed that 4-month-old infants focused on conflicting low-level sequence statistics and, therefore, failed to detect the ordinal position information, but that 6-month-old infants ignored the statistics and detected the ordinal position information. Experiment 2 tested 6-, 8-, and 10-month-old infants and showed that only 10-month-old infants detected relative ordinal position information and that they could only accomplish this with the aid of concurrent statistical cues. Together, these results indicate that a domain-general ability to detect ordinal position information emerges during infancy and that its initial emergence is preceded and facilitated by the earlier emergence of the ability to detect statistical cues.

A dynamical model of hierarchical selection and coordination in speech planning

studies of the control of complex sequential movements have dissociated two aspects of movement planning: control over the sequential selection of movement plans, and control over the precise timing of movement execution. This distinction is particularly relevant in the production of speech: utterances contain sequentially ordered words and syllables, but articulatory movements are often executed in a non-sequential, overlapping manner with precisely coordinated relative timing. This study presents a hybrid dynamical model in which competitive activation controls selection of movement plans and coupled oscillatory systems govern coordination. The model departs from previous approaches by ascribing an important role to competitive selection of articulatory plans within a syllable. Numerical simulations show that the model reproduces a variety of speech production phenomena, such as effects of preparation and utterance composition on reaction time, and asymmetries in patterns of articulatory timing associated with onsets and codas. The model furthermore provides a unified understanding of a diverse group of phonetic and phonological phenomena which have not previously been related.

Rhythmic priming enhances the phonological processing of speech

While natural speech does not possess the same degree of temporal regularity found in music, there is recent evidence to suggest that temporal regularity enhances speech processing. The aim of this experiment was to examine whether speech processing would be enhanced by the prior presentation of a rhythmical prime. We recorded electrophysiological (EEG) and behavioural (reaction time) data while participants listened to nonsense words preceded by a simple rhythm. Results showed that speech processing was enhanced by the temporal expectations generated by the prime. Interestingly, beat and metrical structure of the prime had an effect on different ERP components elicited by the following word (N100, P300). These results indicate that using a musical-like rhythmic prime matched to the prosodic features of speech enhances phonological processing of spoken words and thus reveal a cross-domain effect of musical rhythm on the processing of speech rhythm.

A corticostriatal neural system enhances auditory perception through temporal context processing

The temporal context of an acoustic signal can greatly influence its perception. The present study investigated the neural correlates underlying perceptual facilitation by regular temporal contexts in humans. Participants listened to temporally regular (periodic) or temporally irregular (nonperiodic) sequences of tones while performing an intensity discrimination task. Participants performed significantly better on intensity discrimination during periodic than nonperiodic tone sequences. There was greater activation in the putamen for periodic than nonperiodic sequences. Conversely, there was greater activation in bilateral primary and secondary auditory cortices (planum polare and planum temporale) for nonperiodic than periodic sequences. Across individuals, greater putamen activation correlated with lesser auditory cortical activation in both right and left hemispheres. These findings suggest that temporal regularity is detected in the putamen, and that such detection facilitates temporal-lobe cortical processing associated with superior auditory perception. Thus, this study reveals a corticostriatal system associated with contextual facilitation for auditory perception through temporal regularity processing.

Multistability, cross-modal binding and the additivity of conjoined grouping principles

We present a sceptical view of multimodal multistability--drawing most of our examples from the relation between audition and vision. We begin by summarizing some of the principal ways in which audio-visual binding takes place. We review the evidence that unambiguous stimulation in one modality may affect the perception of a multistable stimulus in another modality. Cross-modal influences of one multistable stimulus on the multistability of another are different: they have occurred only in speech perception. We then argue that the strongest relation between perceptual organization in vision and perceptual organization in audition is likely to be by way of analogous Gestalt laws. We conclude with some general observations about multimodality.

The function of dopaminergic neural signal transmission in auditory pulse perception: evidence from dopaminergic treatment in Parkinson's patients

Auditory pulse perception, which is the perception of relatively salient and regularly appearing events in an acoustic sequence, is a necessary function in humans and has been suggested to rely on basal ganglia function. Our study investigated the effect dopamine depletion has on the auditory pulse perception in Parkinson's disease (PD). We examined PD patients and healthy seniors in this study, and all participants performed a pulse perception task and a motor control task. The pulse perception task consisted of a two alternative forced choice task in which subjects had to identify stimuli as metrical or non-metrical. We tested PD patients before and after the administration of l-3,4-dihydroxyphenylalanin (l-DOPA). The healthy control group performed the same tasks twice. PD patients that were dopamine depleted performed the pulse perception task equally well and as fast as did the healthy control group. However, after the administration of l-DOPA, PD patients performed the pulse perception task significantly faster than they did before the pharmacological intervention, which showed that pulse perception can be modulated by dopaminergic stimulation. These findings indicate that pulse perception relies on dopaminergic mechanisms but is not affected by dopamine depletion in the early stages of PD.

Effects of reading fluency stimulation with emphasis on prosody

PURPOSE

To characterize the reading ability of children with five years of education according to temporal standards, as well as to compare the reading fluency performance of these children before and after a reading program based on prosody standards.

METHODS

Thirty two fourth-grade children participated in this study. They performed reading aloud and picture description tasks in order to verify reading rate, speech rate, text comprehension and the adequacy of prosody variation. Afterwards, it was carried out a reading stimulation program with emphasis on prosody, composed of five 15-minute sessions of reading tasks. At the end of the program, children were re-assessed, in order verify their performance after stimulation.

RESULTS

Changes were observed in reading rate, number of words misread, and quality of prosody during the reading task.

CONCLUSIONS

The reading program promoted positive changes in reading fluency measures.

Rate and duration memory of naturalistic sounds

Research has shown that the tempi of familiar songs are remarkably well-remembered with a high degree of accuracy. The goal of the present research was to determine whether this ability generalizes to various types of ecological sounds and, if so, how rate information is encoded into the cognitive system. Across three experiments, participants were familiarized with a set of sounds and later asked to remember the rate of each. Experiment 1 revealed that the inherent rate of sounds is incidentally learned such that subsequent recognition is comparable across both prospective and retrospective paradigms. Experiment 2 confirmed these findings through an adjustment task and further demonstrated that memory for event rate remains highly accurate regardless if attending is initially directed toward a sound's rate and/or pitch qualities. Lastly, Experiment 3 assessed the ability to recognize event rate vs. duration when the two dimensions systematically co-vary with one another and revealed that rate variations influence duration recognition but not vice versa. These overall findings are discussed relative to a framework that emphasizes the role of event structure in time judgment behavior.