1
|
Pulvermüller F. Neurobiological mechanisms for language, symbols and concepts: Clues from brain-constrained deep neural networks. Prog Neurobiol 2023; 230:102511. [PMID: 37482195 PMCID: PMC10518464 DOI: 10.1016/j.pneurobio.2023.102511] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 05/02/2023] [Accepted: 07/18/2023] [Indexed: 07/25/2023]
Abstract
Neural networks are successfully used to imitate and model cognitive processes. However, to provide clues about the neurobiological mechanisms enabling human cognition, these models need to mimic the structure and function of real brains. Brain-constrained networks differ from classic neural networks by implementing brain similarities at different scales, ranging from the micro- and mesoscopic levels of neuronal function, local neuronal links and circuit interaction to large-scale anatomical structure and between-area connectivity. This review shows how brain-constrained neural networks can be applied to study in silico the formation of mechanisms for symbol and concept processing and to work towards neurobiological explanations of specifically human cognitive abilities. These include verbal working memory and learning of large vocabularies of symbols, semantic binding carried by specific areas of cortex, attention focusing and modulation driven by symbol type, and the acquisition of concrete and abstract concepts partly influenced by symbols. Neuronal assembly activity in the networks is analyzed to deliver putative mechanistic correlates of higher cognitive processes and to develop candidate explanations founded in established neurobiological principles.
Collapse
Affiliation(s)
- Friedemann Pulvermüller
- Brain Language Laboratory, Department of Philosophy and Humanities, WE4, Freie Universität Berlin, 14195 Berlin, Germany; Berlin School of Mind and Brain, Humboldt Universität zu Berlin, 10099 Berlin, Germany; Einstein Center for Neurosciences Berlin, 10117 Berlin, Germany; Cluster of Excellence 'Matters of Activity', Humboldt Universität zu Berlin, 10099 Berlin, Germany.
| |
Collapse
|
2
|
Henningsen-Schomers MR, Pulvermüller F. Modelling concrete and abstract concepts using brain-constrained deep neural networks. PSYCHOLOGICAL RESEARCH 2021; 86:2533-2559. [PMID: 34762152 DOI: 10.1007/s00426-021-01591-6] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Abstract
A neurobiologically constrained deep neural network mimicking cortical areas relevant for sensorimotor, linguistic and conceptual processing was used to investigate the putative biological mechanisms underlying conceptual category formation and semantic feature extraction. Networks were trained to learn neural patterns representing specific objects and actions relevant to semantically 'ground' concrete and abstract concepts. Grounding sets consisted of three grounding patterns with neurons representing specific perceptual or action-related features; neurons were either unique to one pattern or shared between patterns of the same set. Concrete categories were modelled as pattern triplets overlapping in their 'shared neurons', thus implementing semantic feature sharing of all instances of a category. In contrast, abstract concepts had partially shared feature neurons common to only pairs of category instances, thus, exhibiting family resemblance, but lacking full feature overlap. Stimulation with concrete and abstract conceptual patterns and biologically realistic unsupervised learning caused formation of strongly connected cell assemblies (CAs) specific to individual grounding patterns, whose neurons were spread out across all areas of the deep network. After learning, the shared neurons of the instances of concrete concepts were more prominent in central areas when compared with peripheral sensorimotor ones, whereas for abstract concepts the converse pattern of results was observed, with central areas exhibiting relatively fewer neurons shared between pairs of category members. We interpret these results in light of the current knowledge about the relative difficulty children show when learning abstract words. Implications for future neurocomputational modelling experiments as well as neurobiological theories of semantic representation are discussed.
Collapse
Affiliation(s)
- Malte R Henningsen-Schomers
- Department of Philosophy of Humanities, Brain Language Laboratory, Freie Universität Berlin, Habelschwerdter Allee 45, 14195, Berlin, Germany.
- Cluster of Excellence 'Matters of Activity. Image Space Material', Humboldt-Universität zu Berlin, Berlin, Germany.
| | - Friedemann Pulvermüller
- Department of Philosophy of Humanities, Brain Language Laboratory, Freie Universität Berlin, Habelschwerdter Allee 45, 14195, Berlin, Germany
- Berlin School of Mind and Brain, Humboldt-Universität zu Berlin, Berlin, Germany
- Einstein Center for Neurosciences, Berlin, Germany
- Cluster of Excellence 'Matters of Activity. Image Space Material', Humboldt-Universität zu Berlin, Berlin, Germany
| |
Collapse
|
3
|
|
4
|
|
5
|
Förster F, Saunders J, Lehmann H, Nehaniv CL. Robots Learning to Say “No”. ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2019. [DOI: 10.1145/3359618] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Abstract
“No” is one of the first ten words used by children and embodies the first form of linguistic negation. Despite its early occurrence, the details of its acquisition remain largely unknown. The circumstance that “no” cannot be construed as a label for perceptible objects or events puts it outside the scope of most modern accounts of language acquisition. Moreover, most symbol grounding architectures will struggle to ground the word due to its non-referential character. The presented work extends symbol grounding to encompass affect and motivation. In a study involving the child-like robot iCub, we attempt to illuminate the acquisition process of negation words. The robot is deployed in speech-wise unconstrained interaction with participants acting as its language teachers. The results corroborate the hypothesis that affect or volition plays a pivotal role in the acquisition process. Negation words are prosodically salient within prohibitive utterances and negative intent interpretations such that they can be easily isolated from the teacher’s speech signal. These words subsequently may be grounded in negative affective states. However, observations of the nature of prohibition and the temporal relationships between its linguistic and extra-linguistic components raise questions over the suitability of Hebbian-type algorithms for certain types of language grounding.
Collapse
Affiliation(s)
| | | | - Hagen Lehmann
- Università di Macerata, Italy and University of Hertfordshire, Hatfield, UK
| | | |
Collapse
|
6
|
Hagiwara Y, Kobayashi H, Taniguchi A, Taniguchi T. Symbol Emergence as an Interpersonal Multimodal Categorization. Front Robot AI 2019; 6:134. [PMID: 33501149 PMCID: PMC7805687 DOI: 10.3389/frobt.2019.00134] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2019] [Accepted: 11/19/2019] [Indexed: 11/13/2022] Open
Abstract
This study focuses on category formation for individual agents and the dynamics of symbol emergence in a multi-agent system through semiotic communication. In this study, the semiotic communication refers to exchanging signs composed of the signifier (i.e., words) and the signified (i.e., categories). We define the generation and interpretation of signs associated with the categories formed through the agent's own sensory experience or by exchanging signs with other agents as basic functions of the semiotic communication. From the viewpoint of language evolution and symbol emergence, organization of a symbol system in a multi-agent system (i.e., agent society) is considered as a bottom-up and dynamic process, where individual agents share the meaning of signs and categorize sensory experience. A constructive computational model can explain the mutual dependency of the two processes and has mathematical support that guarantees a symbol system's emergence and sharing within the multi-agent system. In this paper, we describe a new computational model that represents symbol emergence in a two-agent system based on a probabilistic generative model for multimodal categorization. It models semiotic communication via a probabilistic rejection based on the receiver's own belief. We have found that the dynamics by which cognitively independent agents create a symbol system through their semiotic communication can be regarded as the inference process of a hidden variable in an interpersonal multimodal categorizer, i.e., the complete system can be regarded as a single agent performing multimodal categorization using the sensors of all agents, if we define the rejection probability based on the Metropolis-Hastings algorithm. The validity of the proposed model and algorithm for symbol emergence, i.e., forming and sharing signs and categories, is also verified in an experiment with two agents observing daily objects in the real-world environment. In the experiment, we compared three communication algorithms: no communication, no rejection, and the proposed algorithm. The experimental results demonstrate that our model reproduces the phenomena of symbol emergence, which does not require a teacher who would know a pre-existing symbol system. Instead, the multi-agent system can form and use a symbol system without having pre-existing categories.
Collapse
Affiliation(s)
- Yoshinobu Hagiwara
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Hiroyoshi Kobayashi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Akira Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tadahiro Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| |
Collapse
|
7
|
Pulvermüller F. Neurobiological Mechanisms for Semantic Feature Extraction and Conceptual Flexibility. Top Cogn Sci 2018; 10:590-620. [DOI: 10.1111/tops.12367] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2017] [Revised: 05/02/2018] [Accepted: 05/09/2018] [Indexed: 11/30/2022]
Affiliation(s)
- Friedemann Pulvermüller
- Brain Language Laboratory Department of Philosophy and Humanities WE4, Freie Universität Berlin
- Berlin School of Mind and Brain Humboldt Universität zu Berlin
- Einstein Center for Neurosciences Berlin
| |
Collapse
|
8
|
Pulvermüller F. The case of CAUSE: neurobiological mechanisms for grounding an abstract concept. Philos Trans R Soc Lond B Biol Sci 2018; 373:20170129. [PMID: 29914997 PMCID: PMC6015827 DOI: 10.1098/rstb.2017.0129] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/08/2018] [Indexed: 11/19/2022] Open
Abstract
How can we understand causal relationships and how can we understand words such as 'cause'? Some theorists assume that the underlying abstract concept is given to us, and that perceptual correlation provides the relevant hints towards inferring causation from perceived real-life events. A different approach emphasizes the role of actions and their typical consequences for the emergence of the concept of causation and the application of the related term. A model of causation is proposed that highlights the family resemblance between causal actions and postulates that symbols are necessary for binding together the different partially shared semantic features of subsets of causal actions and their goals. Linguistic symbols are proposed to play a key role in binding the different subsets of semantic features of the abstract concept. The model is spelt out at the neuromechanistic level of distributed cortical circuits and the cognitive functions they carry. The model is discussed in light of behavioural and neuroscience evidence, and questions for future research are highlighted. In sum, taking causation as a concrete example, I argue that abstract concepts and words can be learnt and grounded in real-life interaction, and that the neurobiological mechanisms realizing such abstract semantic grounding are within our grasp.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.
Collapse
Affiliation(s)
- Friedemann Pulvermüller
- Brain Language Laboratory, Department of Philosophy and Humanities, WE4, Freie Universität Berlin, 14195 Berlin, Germany
- Berlin School of Mind and Brain, Humboldt-Universität zu Berlin, 10099 Berlin, Germany
- Einstein Center for Neurosciences Berlin, Charitéplatz 1, 10117 Berlin, Germany
| |
Collapse
|
9
|
Cangelosi A, Stramandinoli F. A review of abstract concept learning in embodied agents and robots. Philos Trans R Soc Lond B Biol Sci 2018; 373:20170131. [PMID: 29914999 PMCID: PMC6015819 DOI: 10.1098/rstb.2017.0131] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/22/2017] [Indexed: 11/12/2022] Open
Abstract
This paper reviews computational modelling approaches to the learning of abstract concepts and words in embodied agents such as humanoid robots. This will include a discussion of the learning of abstract words such as 'use' and 'make' in humanoid robot experiments, and the acquisition of numerical concepts via gesture and finger counting strategies. The current approaches share a strong emphasis on embodied cognition aspects for the grounding of abstract concepts, and a continuum, rather than dichotomy, view of concrete/abstract concepts differences.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.
Collapse
Affiliation(s)
- Angelo Cangelosi
- Centre for Robotics and Neural Systems, Plymouth University, Plymouth PL4 8AA, UK
| | - Francesca Stramandinoli
- iCub Facility Department, Istituto Italiano di Tecnologia, Via Morego 30, 16163 Genoa, Italy
| |
Collapse
|
10
|
Taniguchi A, Taniguchi T, Cangelosi A. Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots. Front Neurorobot 2018; 11:66. [PMID: 29311888 PMCID: PMC5742219 DOI: 10.3389/fnbot.2017.00066] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2017] [Accepted: 11/21/2017] [Indexed: 11/24/2022] Open
Abstract
In this paper, we propose a Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of the four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations rather than conventional situations of cross-situational learning. We conducted a learning scenario using a simulator and a real humanoid iCub robot. In the scenario, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The scenario was set as follows: the number of words per sensory-channel was three or four, and the number of trials for learning was 20 and 40 for the simulator and 25 and 40 for the real robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning scenario. The experimental results showed that the robot could successfully use the word meanings learned by using the proposed method.
Collapse
Affiliation(s)
- Akira Taniguchi
- Emergent Systems Laboratory, Ritsumeikan University, Kusatsu, Japan
| | | | - Angelo Cangelosi
- The Centre for Robotics and Neural Systems, Plymouth University, Plymouth, United Kingdom
| |
Collapse
|
11
|
Yamada T, Murata S, Arie H, Ogata T. Representation Learning of Logic Words by an RNN: From Word Sequences to Robot Actions. Front Neurorobot 2017; 11:70. [PMID: 29311891 PMCID: PMC5744442 DOI: 10.3389/fnbot.2017.00070] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2017] [Accepted: 12/14/2017] [Indexed: 11/13/2022] Open
Abstract
An important characteristic of human language is compositionality. We can efficiently express a wide variety of real-world situations, events, and behaviors by compositionally constructing the meaning of a complex expression from a finite number of elements. Previous studies have analyzed how machine-learning models, particularly neural networks, can learn from experience to represent compositional relationships between language and robot actions with the aim of understanding the symbol grounding structure and achieving intelligent communicative agents. Such studies have mainly dealt with the words (nouns, adjectives, and verbs) that directly refer to real-world matters. In addition to these words, the current study deals with logic words, such as “not,” “and,” and “or” simultaneously. These words are not directly referring to the real world, but are logical operators that contribute to the construction of meaning in sentences. In human–robot communication, these words may be used often. The current study builds a recurrent neural network model with long short-term memory units and trains it to learn to translate sentences including logic words into robot actions. We investigate what kind of compositional representations, which mediate sentences and robot actions, emerge as the network's internal states via the learning process. Analysis after learning shows that referential words are merged with visual information and the robot's own current state, and the logical words are represented by the model in accordance with their functions as logical operators. Words such as “true,” “false,” and “not” work as non-linear transformations to encode orthogonal phrases into the same area in a memory cell state space. The word “and,” which required a robot to lift up both its hands, worked as if it was a universal quantifier. The word “or,” which required action generation that looked apparently random, was represented as an unstable space of the network's dynamical system.
Collapse
Affiliation(s)
- Tatsuro Yamada
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| | - Shingo Murata
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Hiroaki Arie
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Tetsuya Ogata
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| |
Collapse
|