Sign language and pantomime production differentially engage frontal and parietal cortices

Auditory areas are recruited for naturalistic visual meaning in early deaf people

Congenital deafness enhances responses of auditory cortices to non-auditory tasks, yet the nature of the reorganization is not well understood. Here, naturalistic stimuli are used to induce neural synchrony across early deaf and hearing individuals. Participants watch a silent animated film in an intact version and three versions with gradually distorted meaning. Differences between groups are observed in higher-order auditory cortices in all stimuli, with no statistically significant effects in the primary auditory cortex. Comparison between levels of scrambling revealed a heterogeneity of function in secondary auditory areas. Both hemispheres show greater synchrony in the deaf than in the hearing participants for the intact movie and high-level variants. However, only the right hemisphere shows an increased inter-subject synchrony in the deaf people for the low-level movie variants. An event segmentation validates these results: the dynamics of the right secondary auditory cortex in the deaf people consist of shorter-length events with more transitions than the left. Our results reveal how deaf individuals use their auditory cortex to process visual meaning.

A normative model of brain responses to social scenarios reflects the maturity of children and adolescents' social-emotional abilities

The rapid brain maturation in childhood and adolescence accompanies the development of socio-emotional functioning. However, it is unclear how the maturation of the neural activity drives the development of socio-emotional functioning and individual differences. This study aimed to reflect the age dependence of inter-individual differences in brain responses to socio-emotional scenarios and to develop naturalistic imaging indicators to assess the maturity of socio-emotional ability at the individual level. Using three independent naturalistic imaging datasets containing healthy participants (n = 111, 21 and 122), we found and validated that age-modulated inter-individual concordance of brain responses to socio-emotional movies in specific brain regions. The similarity of an individual's brain response to the average response of older participants was defined as response typicality, which predicted an individual's emotion regulation strategies in adolescence and theory of mind (ToM) in childhood. Its predictive power was not superseded by age, sex, cognitive performance or executive function. We further showed that the movie's valence and arousal ratings grounded the response typicality. The findings highlight that forming typical brain response patterns may be a neural phenotype underlying the maturation of socio-emotional ability. The proposed response typicality represents a neuroimaging approach to measure individuals' maturity of cognitive reappraisal and ToM.

Grounded Cognition Entails Linguistic Relativity: Response to Commentators

In this paper, I respond to eight commentaries on my target article called "Grounded cognition entails linguistic relativity: A neglected implication of a major semantic theory." The commentaries span a broad range of disciplines and perspectives. I have organized my response around the following topics: (1) an introductory synopsis of my main argument; (2) grounded versus amodal theories of concepts; (3) language-specific versus language-independent concepts; (4) language, culture, and cognition; (5) language itself as a source of conceptual grounding; (6) abstract concepts, linguistic relativity, and contextual and individual variability; (7) word meanings as language-specific predictions; and (8) some final remarks about the importance of cross-linguistic diversity.

Visual Form and Event Semantics Predict Transitivity in Silent Gestures: Evidence for Compositionality

Silent gesture is not considered to be linguistic, on par with spoken and sign languages. It is claimed that silent gestures, unlike language, represent events holistically, without compositional structure. However, recent research has demonstrated that gesturers use consistent strategies when representing objects and events, and that there are behavioral and clinically relevant limits on what form a gesture may take to effect a particular meaning. This systematicity challenges a holistic interpretation of silent gesture, which predicts that there should be no stable form-meaning correspondence across event representations. Here, we demonstrate to the contrary that untrained gesturers systematically manipulate the form of their gestures when representing events with and without a theme (e.g., Someone popped the balloon vs. Someone walked), that is, transitive and intransitive events. We elicited silent gestures and annotated them for manual features active in coding transitivity distinctions in sign languages. We trained linear support vector machines to make item-by-item transitivity predictions based on these features. Prediction accuracy was good across the entire dataset, thus demonstrating that systematicity in silent gesture can be explained with recourse to subunits. We argue that handshape features are constructs co-opted from cognitive systems subserving manual action production and comprehension for communicative purposes, which may integrate into the linguistic system of emerging sign languages. We further suggest that nonsigners tend to map event participants to each hand, a strategy found across genetically and geographically distinct sign languages, suggesting the strategy's cognitive foundation.

Sign language aphasia

Signed languages are naturally occurring, fully formed linguistic systems that rely on the movement of the hands, arms, torso, and face within a sign space for production, and are perceived predominantly using visual perception. Despite stark differences in modality and linguistic structure, functional neural organization is strikingly similar to spoken language. Generally speaking, left frontal areas support sign production, and regions in the auditory cortex underlie sign comprehension-despite signers not relying on audition to process language. Given this, should a deaf or hearing signer suffer damage to the left cerebral hemisphere, language is vulnerable to impairment. Multiple cases of sign language aphasia have been documented following left hemisphere injury, and the general pattern of linguistic deficits mirrors those observed in spoken language. The right hemisphere likely plays a role in non-linguistic but critical visuospatial functions of sign language; therefore, individuals who are spared from damage to the left hemisphere but suffer injury to the right are at risk for a different set of communication deficits. In this chapter, we review the neurobiology of sign language and patterns of language deficits that follow brain injury in the deaf signing population.

New Perspectives on the Neurobiology of Sign Languages

The first 40 years of research on the neurobiology of sign languages (1960-2000) established that the same key left hemisphere brain regions support both signed and spoken languages, based primarily on evidence from signers with brain injury and at the end of the 20th century, based on evidence from emerging functional neuroimaging technologies (positron emission tomography and fMRI). Building on this earlier work, this review focuses on what we have learned about the neurobiology of sign languages in the last 15-20 years, what controversies remain unresolved, and directions for future research. Production and comprehension processes are addressed separately in order to capture whether and how output and input differences between sign and speech impact the neural substrates supporting language. In addition, the review includes aspects of language that are unique to sign languages, such as pervasive lexical iconicity, fingerspelling, linguistic facial expressions, and depictive classifier constructions. Summary sketches of the neural networks supporting sign language production and comprehension are provided with the hope that these will inspire future research as we begin to develop a more complete neurobiological model of sign language processing.

Pantomime of tool use: looking beyond apraxia

Pantomime has a long tradition in clinical neuropsychology of apraxia. It has been much more used by researchers and clinicians to assess tool-use disorders than real tool use. Nevertheless, it remains incompletely understood and has given rise to controversies, such as the involvement of the left inferior parietal lobe or the nature of the underlying cognitive processes. The present article offers a comprehensive framework, with the aim of specifying the neural and cognitive bases of pantomime. To do so, we conducted a series of meta-analyses of brain-lesion, neuroimaging and behavioural studies about pantomime and other related tasks (i.e. real tool use, imitation of meaningless postures and semantic knowledge). The first key finding is that the area PF (Area PF complex) within the left inferior parietal lobe is crucially involved in both pantomime and real tool use as well as in the kinematics component of pantomime. The second key finding is the absence of a well-defined neural substrate for the posture component of pantomime (both grip errors and body-part-as-tool responses). The third key finding is the role played by the intraparietal sulcus in both pantomime and imitation of meaningless postures. The fourth key finding is that the left angular gyrus seems to be critical in the production of motor actions directed towards the body. The fifth key finding is that performance on pantomime is strongly correlated with the severity of semantic deficits. Taken together, these findings invite us to offer a neurocognitive model of pantomime, which provides an integrated alternative to the two hypotheses that dominate the field: The gesture-engram hypothesis and the communicative hypothesis. More specifically, this model assumes that technical reasoning (notably the left area PF), the motor-control system (notably the intraparietal sulcus), body structural description (notably the left angular gyrus), semantic knowledge (notably the polar temporal lobes) and potentially theory of mind (notably the middle prefrontal cortex) work in concert to produce pantomime. The original features of this model open new avenues for understanding the neurocognitive bases of pantomime, emphasizing that pantomime is a communicative task that nevertheless originates in specific tool-use (not motor-related) cognitive processes. .

Disentangling Pantomime From Early Sign in a New Sign Language: Window Into Language Evolution Research

In this study, we aim to disentangle pantomime from early signs in a newly-born sign language: Sao Tome and Principe Sign Language. Our results show that within 2 years of their first contact with one another, a community of 100 participants interacting everyday was able to build a shared language. The growth of linguistic systematicity, which included a decrease in use of pantomime, reduction of the amplitude of signs and an increase in articulation economy, showcases a learning, and social interaction process that constitutes a continuum and not a cut-off system. The human cognitive system is endowed with mechanisms for symbolization that allow the process of arbitrariness to unfold and the expansion of linguistic complexity. Our study helps to clarify the role of pantomime in a new sign language and how this role might be linked with language itself, showing implications for language evolution research.

Sign Language: How the Brain Represents Phonology without Sound

A rare case of a deaf signer undergoing awake craniotomy has revealed that sensorimotor cortex is functionally organized for signing. Electrocorticography recordings indicated neural tuning to linguistically-relevant handshapes and body locations and distinct neural activity for linguistic versus transitional movements.

Shared and modality-specific brain regions that mediate auditory and visual word comprehension

Visual speech carried by lip movements is an integral part of communication. Yet, it remains unclear in how far visual and acoustic speech comprehension are mediated by the same brain regions. Using multivariate classification of full-brain MEG data, we first probed where the brain represents acoustically and visually conveyed word identities. We then tested where these sensory-driven representations are predictive of participants' trial-wise comprehension. The comprehension-relevant representations of auditory and visual speech converged only in anterior angular and inferior frontal regions and were spatially dissociated from those representations that best reflected the sensory-driven word identity. These results provide a neural explanation for the behavioural dissociation of acoustic and visual speech comprehension and suggest that cerebral representations encoding word identities may be more modality-specific than often upheld.

Gesture, sign, and language: The coming of age of sign language and gesture studies

How does sign language compare with gesture, on the one hand, and spoken language on the other? Sign was once viewed as nothing more than a system of pictorial gestures without linguistic structure. More recently, researchers have argued that sign is no different from spoken language, with all of the same linguistic structures. The pendulum is currently swinging back toward the view that sign is gestural, or at least has gestural components. The goal of this review is to elucidate the relationships among sign language, gesture, and spoken language. We do so by taking a close look not only at how sign has been studied over the past 50 years, but also at how the spontaneous gestures that accompany speech have been studied. We conclude that signers gesture just as speakers do. Both produce imagistic gestures along with more categorical signs or words. Because at present it is difficult to tell where sign stops and gesture begins, we suggest that sign should not be compared with speech alone but should be compared with speech-plus-gesture. Although it might be easier (and, in some cases, preferable) to blur the distinction between sign and gesture, we argue that distinguishing between sign (or speech) and gesture is essential to predict certain types of learning and allows us to understand the conditions under which gesture takes on properties of sign, and speech takes on properties of gesture. We end by calling for new technology that may help us better calibrate the borders between sign and gesture.

The neural circuits recruited for the production of signs and fingerspelled words

Signing differs from typical non-linguistic hand actions because movements are not visually guided, finger movements are complex (particularly for fingerspelling), and signs are not produced as holistic gestures. We used positron emission tomography to investigate the neural circuits involved in the production of American Sign Language (ASL). Different types of signs (one-handed (articulated in neutral space), two-handed (neutral space), and one-handed body-anchored signs) were elicited by asking deaf native signers to produce sign translations of English words. Participants also fingerspelled (one-handed) printed English words. For the baseline task, participants indicated whether a word contained a descending letter. Fingerspelling engaged ipsilateral motor cortex and cerebellar cortex in contrast to both one-handed signs and the descender baseline task, which may reflect greater timing demands and complexity of handshape sequences required for fingerspelling. Greater activation in the visual word form area was also observed for fingerspelled words compared to one-handed signs. Body-anchored signs engaged bilateral superior parietal cortex to a greater extent than the descender baseline task and neutral space signs, reflecting the motor control and proprioceptive monitoring required to direct the hand toward a specific location on the body. Less activation in parts of the motor circuit was observed for two-handed signs compared to one-handed signs, possibly because, for half of the signs, handshape and movement goals were spread across the two limbs. Finally, the conjunction analysis comparing each sign type with the descender baseline task revealed common activation in the supramarginal gyrus bilaterally, which we interpret as reflecting phonological retrieval and encoding processes.

The biology of linguistic expression impacts neural correlates for spatial language

Biological differences between signed and spoken languages may be most evident in the expression of spatial information. PET was used to investigate the neural substrates supporting the production of spatial language in American Sign Language as expressed by classifier constructions, in which handshape indicates object type and the location/motion of the hand iconically depicts the location/motion of a referent object. Deaf native signers performed a picture description task in which they overtly named objects or produced classifier constructions that varied in location, motion, or object type. In contrast to the expression of location and motion, the production of both lexical signs and object type classifier morphemes engaged left inferior frontal cortex and left inferior temporal cortex, supporting the hypothesis that unlike the location and motion components of a classifier construction, classifier handshapes are categorical morphemes that are retrieved via left hemisphere language regions. In addition, lexical signs engaged the anterior temporal lobes to a greater extent than classifier constructions, which we suggest reflects increased semantic processing required to name individual objects compared with simply indicating the type of object. Both location and motion classifier constructions engaged bilateral superior parietal cortex, with some evidence that the expression of static locations differentially engaged the left intraparietal sulcus. We argue that bilateral parietal activation reflects the biological underpinnings of sign language. To express spatial information, signers must transform visual-spatial representations into a body-centered reference frame and reach toward target locations within signing space.

Towards a new neurobiology of language

Theoretical advances in language research and the availability of increasingly high-resolution experimental techniques in the cognitive neurosciences are profoundly changing how we investigate and conceive of the neural basis of speech and language processing. Recent work closely aligns language research with issues at the core of systems neuroscience, ranging from neurophysiological and neuroanatomic characterizations to questions about neural coding. Here we highlight, across different aspects of language processing (perception, production, sign language, meaning construction), new insights and approaches to the neurobiology of language, aiming to describe promising new areas of investigation in which the neurosciences intersect with linguistic research more closely than before. This paper summarizes in brief some of the issues that constitute the background for talks presented in a symposium at the Annual Meeting of the Society for Neuroscience. It is not a comprehensive review of any of the issues that are discussed in the symposium.