Memory and Common Ground Processes in Language Use

The Limitations of Large Language Models for Understanding Human Language and Cognition

Researchers have recently argued that the capabilities of Large Language Models (LLMs) can provide new insights into longstanding debates about the role of learning and/or innateness in the development and evolution of human language. Here, we argue on two grounds that LLMs alone tell us very little about human language and cognition in terms of acquisition and evolution. First, any similarities between human language and the output of LLMs are purely functional. Borrowing the "four questions" framework from ethology, we argue that what LLMs do is superficially similar, but how they do it is not. In contrast to the rich multimodal data humans leverage in interactive language learning, LLMs rely on immersive exposure to vastly greater quantities of unimodal text data, with recent multimodal efforts built upon mappings between images and text. Second, turning to functional similarities between human language and LLM output, we show that human linguistic behavior is much broader. LLMs were designed to imitate the very specific behavior of human writing; while they do this impressively, the underlying mechanisms of these models limit their capacities for meaning and naturalistic interaction, and their potential for dealing with the diversity in human language. We conclude by emphasising that LLMs are not theories of language, but tools that may be used to study language, and that can only be effectively applied with specific hypotheses to motivate research.

Using an episodic specificity induction to improve children's future thinking

Episodic future thinking (EFT) is the ability to subjectively pre-experience a specific future event. Future-oriented cognition in young children positively predicts physical health and financial status later in life. Can EFT be improved in children, even temporarily? Developmental research emphasizes the importance of thinking about one's own near future to enhance EFT, whereas research in adults suggests benefits reside in constructing a richly detailed event. We bridged the two perspectives to examine whether a procedure, the "episodic specificity induction" (ESI), could be adapted to encourage an episodic mode of thinking in children, benefitting performance on a variety of subsequent EFT tasks. The present study implemented a child-friendly ESI in which children mentally simulated a future event and were probed for specific details about it. We randomly assigned 66 children aged 6 and 7 years to one of two conditions: (1) ESI, in which children imagined "having breakfast tomorrow" in detail, describing surroundings, people, and actions, or (2) a Control condition (i.e., no construction), in which children simply viewed and described a picture of another child having breakfast. Children then completed a series of future thinking tasks assessing prospective memory, recollection/imagination of events, delay of gratification, and planning. Our ESI was successful in promoting the construction of a detailed event, and subsequently increasing the number of details of recollected and imagined events on an outcome task as compared to a control condition. Nonetheless, the effect of ESI was smaller than expected - a finding that fits with recent work suggesting that such interventions may be too cognitively taxing for young children and/or that benefits may hinge on further development in episodic processes. We discuss possible modifications to the induction and implications for EFT amelioration in young children.

Beyond speaking: neurocognitive perspectives on language production in social interaction

The human faculty to speak has evolved, so has been argued, for communicating with others and for engaging in social interactions. Hence the human cognitive system should be equipped to address the demands that social interaction places on the language production system. These demands include the need to coordinate speaking with listening, the need to integrate own (verbal) actions with the interlocutor's actions, and the need to adapt language flexibly to the interlocutor and the social context. In order to meet these demands, core processes of language production are supported by cognitive processes that enable interpersonal coordination and social cognition. To fully understand the cognitive architecture and its neural implementation enabling humans to speak in social interaction, our understanding of how humans produce language needs to be connected to our understanding of how humans gain insights into other people's mental states and coordinate in social interaction. This article reviews theories and neurocognitive experiments that make this connection and can contribute to advancing our understanding of speaking in social interaction. This article is part of a discussion meeting issue 'Face2face: advancing the science of social interaction'.

Unraveling the functional attributes of the language connectome: crucial subnetworks, flexibility and variability

Language processing is a highly integrative function, intertwining linguistic operations (processing the language code intentionally used for communication) and extra-linguistic processes (e.g., attention monitoring, predictive inference, long-term memory). This synergetic cognitive architecture requires a distributed and specialized neural substrate. Brain systems have mainly been examined at rest. However, task-related functional connectivity provides additional and valuable information about how information is processed when various cognitive states are involved. We gathered thirteen language fMRI tasks in a unique database of one hundred and fifty neurotypical adults (InLang [Interactive networks of Language] database), providing the opportunity to assess language features across a wide range of linguistic processes. Using this database, we applied network theory as a computational tool to model the task-related functional connectome of language (LANG atlas). The organization of this data-driven neurocognitive atlas of language was examined at multiple levels, uncovering its major components (or crucial subnetworks), and its anatomical and functional correlates. In addition, we estimated its reconfiguration as a function of linguistic demand (flexibility) or several factors such as age or gender (variability). We observed that several discrete networks could be specifically shaped to promote key functional features of language: coding-decoding (Net1), control-executive (Net2), abstract-knowledge (Net3), and sensorimotor (Net4) functions. The architecture of these systems and the functional connectivity of the pivotal brain regions varied according to the nature of the linguistic process, gender, or age. By accounting for the multifaceted nature of language and modulating factors, this study can contribute to enriching and refining existing neurocognitive models of language. The LANG atlas can also be considered a reference for comparative or clinical studies involving various patients and conditions.

Evidence of Audience Design in Amnesia: Adaptation in Gesture but Not Speech

Speakers design communication for their audience, providing more information in both speech and gesture when their listener is naïve to the topic. We test whether the hippocampal declarative memory system contributes to multimodal audience design. The hippocampus, while traditionally linked to episodic and relational memory, has also been linked to the ability to imagine the mental states of others and use language flexibly. We examined the speech and gesture use of four patients with hippocampal amnesia when describing how to complete everyday tasks (e.g., how to tie a shoe) to an imagined child listener and an adult listener. Although patients with amnesia did not increase their total number of words and instructional steps for the child listener, they did produce representational gestures at significantly higher rates for the imagined child compared to the adult listener. They also gestured at similar frequencies to neurotypical peers, suggesting that hand gesture can be a meaningful communicative resource, even in the case of severe declarative memory impairment. We discuss the contributions of multiple memory systems to multimodal audience design and the potential of gesture to act as a window into the social cognitive processes of individuals with neurologic disorders.

Neural signatures associated with temporal compression in the verbal retelling of past events

When we retell our past experiences, we aim to reproduce some version of the original events; this reproduced version is often temporally compressed relative to the original. However, it is currently unclear how this compression manifests in brain activity. One possibility is that a compressed retrieved memory manifests as a neural pattern which is more dissimilar to the original, relative to a more detailed or vivid memory. However, we argue that measuring raw dissimilarity alone is insufficient, as it confuses a variety of interesting and uninteresting changes. To address this problem, we examine brain pattern changes that are consistent across people. We show that temporal compression in individuals’ retelling of past events predicts systematic encoding-to-recall transformations in several higher associative regions. These findings elucidate how neural representations are not simply reactivated, but can also be transformed due to temporal compression during a universal form of human memory expression: verbal retelling.

Brain patterns measured while participants first watched a movie in the fMRI scanner, then recalled the movie’s key narrative features, demonstrate that temporal compression in individuals’ retelling of past events predicts encoding-to-recall transformations.

Encoding Motion Events During Language Production: Effects of Audience Design and Conceptual Salience

We investigate the extent to which pragmatic versus conceptual factors can affect a speaker's decision to mention or omit different components of an event. In the two experiments, we demonstrate the special role of pragmatic factors related to audience design in speakers' decisions to mention conceptually "peripheral" event components, such as sources (i.e., starting points) in source-goal motion events (e.g., a baby crawling from a crib to a toybox). In particular, we found that pragmatic factors related to audience design could not only drive the decision to omit sources from mention, but could also motivate speakers to mention sources more often than needed. By contrast, speaker's decisions to talk about goals did not appear to be fundamentally driven by pragmatic factors in communication. We also manipulated the animacy of the figure in motion and found that participants in our studies treated both animate and inanimate source-goal motion events in the same way, both linguistically and in memory. We discuss the implications of our work for message generation across different communicative contexts and for future work on the topic of audience design.

What is Functional Communication? A Theoretical Framework for Real-World Communication Applied to Aphasia Rehabilitation

Aphasia is an impairment of language caused by acquired brain damage such as stroke or traumatic brain injury, that affects a person’s ability to communicate effectively. The aim of rehabilitation in aphasia is to improve everyday communication, improving an individual’s ability to function in their day-to-day life. For that reason, a thorough understanding of naturalistic communication and its underlying mechanisms is imperative. The field of aphasiology currently lacks an agreed, comprehensive, theoretically founded definition of communication. Instead, multiple disparate interpretations of functional communication are used. We argue that this makes it nearly impossible to validly and reliably assess a person’s communicative performance, to target this behaviour through therapy, and to measure improvements post-therapy. In this article we propose a structured, theoretical approach to defining the concept of functional communication. We argue for a view of communication as “situated language use”, borrowed from empirical psycholinguistic studies with non-brain damaged adults. This framework defines language use as: (1) interactive, (2) multimodal, and (3) contextual. Existing research on each component of the framework from non-brain damaged adults and people with aphasia is reviewed. The consequences of adopting this approach to assessment and therapy for aphasia rehabilitation are discussed. The aim of this article is to encourage a more systematic, comprehensive approach to the study and treatment of situated language use in aphasia.

Describing communication during a forensic investigation using the Pebbles on a Scale metaphor

During the investigation of a crime, evidence is collected, analyzed, interpreted, and discussed by various stakeholders. This article examines the communication that may occur between two of these stakeholders: detectives and forensic analysts, and how their interaction influences the interpretation of evidence as the investigation proceeds and the theory of the case evolves. Such communication can be understood as sets of actions that are inter-dependent: for example, a request for a specific analysis by a detective leads to analyses and conclusions that the analyst shares with the detective, which leads to an assessment of these conclusions relative to the theory of the case, which leads to further analysis requests, and so forth. We present the Pebbles on a Scale metaphor, which describes how communication and the understanding of evidence takes place between the detective and analysts, and the different ways in which they consider the information as a function of their roles in the investigation. Using a hypothetical case for illustration, we discuss communicative challenges, the evolving theory of the case, the language that is used by analysts to discuss "yes", "no" and "I don't know" conclusions, and how those conclusions are used by detectives during the progression of the investigation.

Missing links: The functional unification of language and memory (L∪M)

The field of neurocognition is currently undergoing a significant change of perspective. Traditional neurocognitive models evolved into an integrative and dynamic vision of cognitive functioning. Dynamic integration assumes an interaction between cognitive domains traditionally considered to be distinct. Language and declarative memory are regarded as separate functions supported by different neural systems. However, they also share anatomical structures (notably, the inferior frontal gyrus, the supplementary motor area, the superior and middle temporal gyrus, and the hippocampal complex) and cognitive processes (such as semantic and working memory) that merge to endorse our quintessential daily lives. We propose a new model, "L∪M" (i.e., Language/union/Memory), that considers these two functions interactively. We fractionated language and declarative memory into three fundamental dimensions or systems ("Receiver-Transmitter", "Controller-Manager" and "Transformer-Associative" Systems), that communicate reciprocally. We formalized their interactions at the brain level with a connectivity-based approach. This new taxonomy overcomes the modular view of cognitive functioning and reconciles functional specialization with plasticity in neurological disorders.

What are friends for? The impact of friendship on communicative efficiency and cortisol response during collaborative problem solving among younger and older women

Conversation is a skilled activity that depends on cognitive and social processes, both of which develop through adulthood. We examined the effects of age and partner familiarity on communicative efficiency and cortisol reactivity. Younger and older women interacted with familiar or unfamiliar partners in a dyadic collaborative conversation task (N = 8 in each group). Regardless of age, referential expressions among familiar and unfamiliar partners became more efficient over time, and cortisol concentrations were lower for speakers interacting with familiar partners. These findings suggest that communicative effectiveness is largely preserved with age, as is the stress-buffering effect of a familiar partner.

The limited role of hippocampal declarative memory in transient semantic activation during online language processing

Recent findings point to a role for hippocampus in the moment-by-moment processing of language, including the use and generation of semantic features in certain contexts. What role the hippocampus might play in the processing of semantic relations in spoken language comprehension, however, is unknown. Here we test patients with bilateral hippocampal damage and dense amnesia in order to examine the necessity of hippocampus for lexico-semantic mapping processes in spoken language understanding. In two visual-world eye-tracking experiments, we monitor eye movements to images that are semantically related to spoken words and sentences. We find no impairment in amnesia, relative to matched healthy comparison participants. These findings suggest, at least for close semantic links and simple language comprehension tasks, a lack of necessity for hippocampus in lexico-semantic mapping between spoken words and simple pictures.

Partner-Specific Communication Deficits in Individuals With Alzheimer's Disease

Purpose Speakers adjust referential expressions to the listeners' knowledge while communicating, a phenomenon called "audience design." While individuals with Alzheimer's disease (AD) show difficulties in discourse production, it is unclear whether they exhibit preserved partner-specific audience design. The current study examined if individuals with AD demonstrate partner-specific audience design skills. Method Ten adults with mild-to-moderate AD and 12 healthy older adults performed a referential communication task with two experimenters (E1 and E2). At first, E1 and participants completed an image-sorting task, allowing them to establish shared labels. Then, during testing, both experimenters were present in the room, and participants described images to either E1 or E2 (randomly alternating). Analyses focused on the number of words participants used to describe each image and whether they reused shared labels. Results During testing, participants in both groups produced shorter descriptions when describing familiar images versus new images, demonstrating their ability to learn novel knowledge. When they described familiar images, healthy older adults modified their expressions depending on the current partner's knowledge, producing shorter expressions and more established labels for the knowledgeable partner (E1) versus the naïve partner (E2), but individuals with AD were less likely to do so. Conclusions The current study revealed that both individuals with AD and the control participants were able to acquire novel knowledge, but individuals with AD tended not to flexibly adjust expressions depending on the partner's knowledge state. Conversational inefficiency and difficulties observed in AD may, in part, stem from disrupted audience design skills.

Caregivers use gesture contingently to support word learning

Children learn words in environments where there is considerable variability, both in terms of the number of possible referents for novel words, and the availability of cues to support word-referent mappings. How caregivers adapt their gestural cues to referential uncertainty has not yet been explored. We tested a computational model of cross-situational word learning that examined the value of a variable gesture cue during training across conditions of varying referential uncertainty. We found that gesture had a greater benefit for referential uncertainty, but unexpectedly also found that learning was best when there was variability in both the environment (number of referents) and gestural cue use. We demonstrated that these results are reflected behaviourally in an experimental word-learning study involving children aged 18-24-month-olds and their caregivers. Under similar conditions to the computational model, caregivers not only used gesture more when there were more potential referents for novel words, but children also learned best when there was some referential ambiguity for words. Thus, caregivers are sensitive to referential uncertainty in the environment and adapt their gestures accordingly, and children are able to respond to environmental variability to learn more robustly. These results imply that training under variable circumstances may actually benefit learning, rather than hinder it.

Processing Speaker-Specific Information in Two Stages During the Interpretation of Referential Precedents

To reduce ambiguity across a conversation, interlocutors reach temporary conventions or referential precedents on how to refer to an entity. Despite their central role in communication, the cognitive underpinnings of the interpretation of precedents remain unclear, specifically the role and mechanisms by which information related to the speaker is integrated. We contrast predictions of one-stage, original two-stage, and extended two-stage models for the processing of speaker information and provide evidence favoring the latter: we show that both stages are sensitive to speaker-specific information. Using an experimental paradigm based on visual-world eye tracking in the context of a referential communication task, we look at the moment-by-moment interpretation of precedents and focus on the temporal profile of the influence of the speaker and linguistic information when facing ambiguity. We find two clearly identifiable moments where speaker-specific information has its effects on reference resolution. We conclude that these two stages reflect two distinct cognitive mechanisms, with different timings, and rely on different representational formats for encoding and accessing information about the speaker: a cue-driven memory retrieval process that mediates language processing and an inferential mechanism based on perspective-taking abilities.

The Role of Gesture in Communication and Cognition: Implications for Understanding and Treating Neurogenic Communication Disorders

When people talk, they gesture. Gesture is a fundamental component of language that contributes meaningful and unique information to a spoken message and reflects the speaker's underlying knowledge and experiences. Theoretical perspectives of speech and gesture propose that they share a common conceptual origin and have a tightly integrated relationship, overlapping in time, meaning, and function to enrich the communicative context. We review a robust literature from the field of psychology documenting the benefits of gesture for communication for both speakers and listeners, as well as its important cognitive functions for organizing spoken language, and facilitating problem-solving, learning, and memory. Despite this evidence, gesture has been relatively understudied in populations with neurogenic communication disorders. While few studies have examined the rehabilitative potential of gesture in these populations, others have ignored gesture entirely or even discouraged its use. We review the literature characterizing gesture production and its role in intervention for people with aphasia, as well as describe the much sparser literature on gesture in cognitive communication disorders including right hemisphere damage, traumatic brain injury, and Alzheimer's disease. The neuroanatomical and behavioral profiles of these patient populations provide a unique opportunity to test theories of the relationship of speech and gesture and advance our understanding of their neural correlates. This review highlights several gaps in the field of communication disorders which may serve as a bridge for applying the psychological literature of gesture to the study of language disorders. Such future work would benefit from considering theoretical perspectives of gesture and using more rigorous and quantitative empirical methods in its approaches. We discuss implications for leveraging gesture to explore its untapped potential in understanding and rehabilitating neurogenic communication disorders.

Gesture height reflects common ground status even in patients with amnesia

When we communicate, we alter our language and gesture based on the mutually shared knowledge - common ground - that we have with our listener. How memory supports these alterations remain unclear. We asked healthy adults and patients with hippocampal amnesia to engage in a referential communication task. Previous work suggests that common ground can be encoded by distinct memory systems; Amnesic patients show normal learning and referential label use as common ground increases, but inconsistently mark these labels with definite determiners (e.g., the vs. a windmill). Which memory systems support the ability to mark common ground via hand gesture? We found that gestures of both healthy participants and amnesic patients reflected common ground status. Both groups produced high gestures when common ground was lacking, and were less likely to do so as common ground increased. These findings suggest that gesture can reflect common ground status during conversation, potentially via non-declarative memory.

Collaborative Remembering in Conversational Narration

This chapter introduces an epistemic perspective on narration and illustrates, based on data from storytelling in free natural conversation, how collaborative remembering can instantiate distributed cognition: first, when tellers deploy expressions of forgetfulness or explicitly enlist the aid of other participants; second, during various forms of collaborative turn sequencing with rapid speaker shift and a high degree of overlap; third, when conversational participants cooperate to produce a mosaic-like narrative consisting of their partially separate perspectives and contributions; and fourth, when they weave their voices into a single narrative thread in a Goffmanian team performance.

Changes in discourse structure over time following traumatic brain injury

The objective of the present study was to investigate structural changes in the narrative discourse of individuals with penetrating traumatic brain injury (pTBI) following immediate and delayed story retellings. Additionally, the potential influence of immediate memory, working memory, and executive functions on narrative discourse performance were examined. The narrative discourse of two groups, 123 with pTBI and 44 non-brain injured (NBI), was sampled. Participants were asked to retell a wordless picture story immediately after viewing it and again 30-min later. Story narratives were analyzed using a variety of microlinguistic and macrostructural measures. Results revealed significant group differences of both microlinguistic and macrostructural measures following the immediate retell, but not following the delayed retell. Regression analyses revealed that immediate memory accounted for a modest degree of the explained variance for the production of critical content during narrative discourse. The observed deficits were relatively stable over a short delay (30 min) and appeared attributable, in part, to difficulty with the encoding and consolidation of story content.

Knowledge and learning of verb biases in amnesia

Verb bias-the co-occurrence frequencies between a verb and the syntactic structures it may appear with-is a critical and reliable linguistic cue for online sentence processing. In particular, listeners use this information to disambiguate sentences with multiple potential syntactic parses (e.g., Feel the frog with the feather.). Further, listeners dynamically update their representations of specific verbs in the face of new evidence about verb-structure co-occurrence. Yet, little is known about the biological memory systems that support the use and dynamic updating of verb bias. We propose that hippocampal-dependent declarative (relational) memory represents a likely candidate system because it has been implicated in the flexible binding of relational co-occurrences and in statistical learning. We explore this question by testing patients with severe and selective deficits in declarative memory (anterograde amnesia), and demographically matched healthy participants, in their on-line interpretation of ambiguous sentences and the ability to update their verb bias with experience. We find that (1) patients and their healthy counterparts use existing verb bias to successfully interpret on-line ambiguity, however (2) unlike healthy young adults, neither group updated these biases in response to recent exposure. These findings demonstrate that using existing representations of verb bias does not necessitate involvement of the declarative memory system, but leave open the question of whether the ability to update representations of verb-specific biases requires hippocampal engagement.

The influence of the hippocampus and declarative memory on word use: Patients with amnesia use less imageable words

Hippocampal functioning contributes to our ability to generate multifaceted, imagistic event representations. Patients with hippocampal damage produce event narratives that contain fewer details and fewer imagistic features. We hypothesized that impoverished memory representations would influence language at the word level, yielding words lower in imageability and concreteness. We tested this by examining language produced by patients with bilateral hippocampal damage and severe declarative memory impairment, and brain-damaged and healthy comparison groups. Participants described events from the real past, imagined past, imagined present, and imagined future. We analyzed the imageability and concreteness of words used. Patients with amnesia used words that were less imageable than those of comparison groups across time periods, even when accounting for the amount of episodic detail in narratives. Moreover, all participants used words that were relatively more imageable when discussing real past events than other time periods. Taken together, these findings suggest that the memory that we have for an event affects how we talk about that event, and this extends all the way to the individual words that we use.

Learning and using knowledge about what other people do and don't know despite amnesia

Successful communication requires keeping track of what other people do and do not know, and how this differs from our own knowledge. Here we ask how knowledge of what others know is stored in memory. We take a neuropsychological approach, comparing healthy adults to patients with severe declarative memory impairment (amnesia). We evaluate whether this memory impairment disrupts the ability to successfully acquire and use knowledge about what other people know when communicating with them. We tested participants in a referential communication task in which the participants described a series of abstract "tangram" images for a partner. Participants then repeated the task with the same partner or a new partner. Findings show that much like healthy individuals, individuals with amnesia successfully tailored their communicative language to the knowledge shared with their conversational partner-their common ground. They produced brief descriptions of the tangram images for the familiar partner and provided more descriptive, longer expressions for the new partner. These findings demonstrate remarkable sparing in amnesia of the acquisition and use of partner-specific knowledge that underlies common ground, and have important implications for understanding the memory systems that support conversational language.