1
|
Pedrelli L, Hinaut X. Hierarchical-Task Reservoir for Online Semantic Analysis From Continuous Speech. IEEE Trans Neural Netw Learn Syst 2022; 33:2654-2663. [PMID: 34570710 DOI: 10.1109/tnnls.2021.3095140] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
In this article, we propose a novel architecture called hierarchical-task reservoir (HTR) suitable for real-time applications for which different levels of abstraction are available. We apply it to semantic role labeling (SRL) based on continuous speech recognition. Taking inspiration from the brain, this demonstrates the hierarchies of representations from perceptive to integrative areas, and we consider a hierarchy of four subtasks with increasing levels of abstraction (phone, word, part-of-speech (POS), and semantic role tags). These tasks are progressively learned by the layers of the HTR architecture. Interestingly, quantitative and qualitative results show that the hierarchical-task approach provides an advantage to improve the prediction. In particular, the qualitative results show that a shallow or a hierarchical reservoir, considered as baselines, does not produce estimations as good as the HTR model would. Moreover, we show that it is possible to further improve the accuracy of the model by designing skip connections and by considering word embedding (WE) in the internal representations. Overall, the HTR outperformed the other state-of-the-art reservoir-based approaches and it resulted in extremely efficient with respect to typical recurrent neural networks (RNNs) in deep learning (DL) [e.g., long short term memory (LSTMs)]. The HTR architecture is proposed as a step toward the modeling of online and hierarchical processes at work in the brain during language comprehension.
Collapse
|
6
|
Abstract
Gated working memory is defined as the capacity of holding arbitrary information at any time in order to be used at a later time. Based on electrophysiological recordings, several computational models have tackled the problem using dedicated and explicit mechanisms. We propose instead to consider an implicit mechanism based on a random recurrent neural network. We introduce a robust yet simple reservoir model of gated working memory with instantaneous updates. The model is able to store an arbitrary real value at random time over an extended period of time. The dynamics of the model is a line attractor that learns to exploit reentry and a nonlinearity during the training phase using only a few representative values. A deeper study of the model shows that there is actually a large range of hyperparameters for which the results hold (e.g., number of neurons, sparsity, global weight scaling) such that any large enough population, mixing excitatory and inhibitory neurons, can quickly learn to realize such gated working memory. In a nutshell, with a minimal set of hypotheses, we show that we can have a robust model of working memory. This suggests this property could be an implicit property of any random population, that can be acquired through learning. Furthermore, considering working memory to be a physically open but functionally closed system, we give account on some counterintuitive electrophysiological recordings.
Collapse
Affiliation(s)
- Anthony Strock
- Inria Bordeaux Sud-Ouest, 33405 Talence Cedex, France; LaBRI, Université de Bordeaux, Institut Polytechnique de Bordeaux, Centre National de la Recherche Scientifique, 33405 Talence Cedex, France; and Institut des Maladies Neurodégénératives, Université de Bordeaux, Centre National de la Recherche Scientifique, 33076 Cedex, Bordeaux, France
| | - Xavier Hinaut
- Inria Bordeaux Sud-Ouest, 33405 Talence Cedex, France; LaBRI, Université de Bordeaux, Institut Polytechnique de Bordeaux, Centre National de la Recherche Scientifique, 33405 Talence Cedex, France; and Institut des Maladies Neurodégénératives, Université de Bordeaux, Centre National de la Recherche Scientifique, 33076 Cedex, Bordeaux, France
| | - Nicolas P Rougier
- Inria Bordeaux Sud-Ouest, 33405 Talence Cedex, France; LaBRI, Université de Bordeaux, Institut Polytechnique de Bordeaux, Centre National de la Recherche Scientifique, 33405 Talence Cedex, France; and Institut des Maladies Neurodégénératives, Université de Bordeaux, Centre National de la Recherche Scientifique, 33076 Cedex, Bordeaux, France
| |
Collapse
|
8
|
Rougier NP, Hinsen K, Alexandre F, Arildsen T, Barba LA, Benureau FC, Brown CT, de Buyl P, Caglayan O, Davison AP, Delsuc MA, Detorakis G, Diem AK, Drix D, Enel P, Girard B, Guest O, Hall MG, Henriques RN, Hinaut X, Jaron KS, Khamassi M, Klein A, Manninen T, Marchesi P, McGlinn D, Metzner C, Petchey O, Plesser HE, Poisot T, Ram K, Ram Y, Roesch E, Rossant C, Rostami V, Shifman A, Stachelek J, Stimberg M, Stollmeier F, Vaggi F, Viejo G, Vitay J, Vostinar AE, Yurchak R, Zito T. Sustainable computational science: the ReScience initiative. PeerJ Comput Sci 2017; 3:e142. [PMID: 34722870 PMCID: PMC8530091 DOI: 10.7717/peerj-cs.142] [Citation(s) in RCA: 42] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2017] [Accepted: 11/15/2017] [Indexed: 05/30/2023]
Abstract
Computer science offers a large set of tools for prototyping, writing, running, testing, validating, sharing and reproducing results; however, computational science lags behind. In the best case, authors may provide their source code as a compressed archive and they may feel confident their research is reproducible. But this is not exactly true. James Buckheit and David Donoho proposed more than two decades ago that an article about computational results is advertising, not scholarship. The actual scholarship is the full software environment, code, and data that produced the result. This implies new workflows, in particular in peer-reviews. Existing journals have been slow to adapt: source codes are rarely requested and are hardly ever actually executed to check that they produce the results advertised in the article. ReScience is a peer-reviewed journal that targets computational research and encourages the explicit replication of already published research, promoting new and open-source implementations in order to ensure that the original research can be replicated from its description. To achieve this goal, the whole publishing chain is radically different from other traditional scientific journals. ReScience resides on GitHub where each new implementation of a computational study is made available together with comments, explanations, and software tests.
Collapse
Affiliation(s)
| | - Konrad Hinsen
- Centre de Biophysique Moléculaire UPR4301, CNRS, Orléans, France
| | | | - Thomas Arildsen
- Department of Electronic Systems, Technical Faculty of IT and Design, Aalborg University, Aalborg, Denmark
| | - Lorena A. Barba
- Department of Mechanical and Aerospace Engineering, The George Washington University, Washington, D.C., USA
| | | | - C. Titus Brown
- Department of Population Health and Reproduction, University of California Davis, Davis, CA, USA
| | - Pierre de Buyl
- Instituut voor Theoretische Fysica, KU Leuven, Leuven, Belgium
| | - Ozan Caglayan
- Laboratoire d’Informatique (LIUM), Le Mans University, Le Mans, France
| | | | - Marc-André Delsuc
- Institut de Génétique et de Biologie Moléculaire et Cellulaire, Illkirch, France
| | - Georgios Detorakis
- Department of Cognitive Sciences, University of California Irvine, Irvine, CA, USA
| | - Alexandra K. Diem
- Computational Engineering and Design, University of Southampton, Southampton, United Kingdom
| | - Damien Drix
- Humboldt Universität zu Berlin, Berlin, Germany
| | - Pierre Enel
- Department of Neuroscience, Mount Sinai School of Medicine, New York, NY, USA
| | - Benoît Girard
- Institute of Intelligent Systems and Robotics, Sorbonne Universités - UPMC Univ Paris 06 - CNRS, Paris, France
| | - Olivia Guest
- Experimental Psychology, University College London, London, Greater London, United Kingdom
| | - Matt G. Hall
- UCL Great Ormond St Institute of Child Health, London, United Kingdom
| | - Rafael N. Henriques
- Champalimaud Centre for the Unknown, Champalimaud Neuroscience Program, Lisbon, Portugal
| | | | - Kamil S. Jaron
- Department of Ecology and Evolution, University of Lausanne, Lausanne, Switzerland
| | - Mehdi Khamassi
- Institute of Intelligent Systems and Robotics, Sorbonne Universités - UPMC Univ Paris 06 - CNRS, Paris, France
| | - Almar Klein
- Independent scholar, Enschede, The Netherlands
| | - Tiina Manninen
- BioMediTech Institute and Faculty of Biomedical Sciences and Engineering, Tampere University of Technology, Tampere, Finland
| | - Pietro Marchesi
- Swammerdam Institute for Life Sciences, University of Amsterdam, Amsterdam, The Netherlands
| | - Daniel McGlinn
- Department of Biology, College of Charleston, Charleston, SC, USA
| | - Christoph Metzner
- Centre for Computer Science and Informatics Research, University of Hertfordshire, Hatfield, United Kingdom
| | - Owen Petchey
- Department of Evolutionary Biology and Environmental Studies, University of Zurich, Zurich, Switzerland
| | - Hans Ekkehard Plesser
- Faculty of Science and Technology, Norwegian University of Life Sciences, Aas, Norway
| | - Timothée Poisot
- Département de Sciences Biologiques, Université de Montréal, Montréal, QC, Canada
| | - Karthik Ram
- Berkeley Institute for Data Science, University of California, Berkeley, CA, USA
| | - Yoav Ram
- Department of Biology, Stanford University, Stanford, CA, USA
| | - Etienne Roesch
- Centre for Integrative Neuroscience, University of Reading, Reading, United Kingdom
| | - Cyrille Rossant
- Institute of Neurology, University College London, London, United Kingdom
| | - Vahid Rostami
- Institute of Neuroscience & Medicine, Juelich Forschungszentrum, Jülich, Germany
| | - Aaron Shifman
- Department of Biology, University of Ottawa, Ottawa, Ontario, Canada
| | - Jemma Stachelek
- Department of Fisheries and Wildlife, Michigan State University, East Lansing, MI, USA
| | - Marcel Stimberg
- Sorbonne Universités/UPMC Univ Paris 06/INSERM/CNRS/Institut de la Vision, Paris, France
| | - Frank Stollmeier
- Max Planck Institute for Dynamics and Self-Organization, Göttingen, Lower Saxony, Germany
| | | | - Guillaume Viejo
- Institute of Intelligent Systems and Robotics, Sorbonne Universités - UPMC Univ Paris 06 - CNRS, Paris, France
| | - Julien Vitay
- Department of Computer Science, Chemnitz University of Technology, Chemnitz, Saxony, Germany
| | - Anya E. Vostinar
- Department of Computer Science, Grinnell College, Grinnell, IA, USA
| | | | - Tiziano Zito
- Neural Information Processing Group, Eberhard Karls Universität Tübingen, Tübingen, Germany
| |
Collapse
|
9
|
Hinaut X, Lance F, Droin C, Petit M, Pointeau G, Dominey PF. Corticostriatal response selection in sentence production: Insights from neural network simulation with reservoir computing. Brain Lang 2015; 150:54-68. [PMID: 26335997 DOI: 10.1016/j.bandl.2015.08.002] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/16/2015] [Revised: 07/18/2015] [Accepted: 08/04/2015] [Indexed: 06/05/2023]
Abstract
Language production requires selection of the appropriate sentence structure to accommodate the communication goal of the speaker - the transmission of a particular meaning. Here we consider event meanings, in terms of predicates and thematic roles, and we address the problem that a given event can be described from multiple perspectives, which poses a problem of response selection. We present a model of response selection in sentence production that is inspired by the primate corticostriatal system. The model is implemented in the context of reservoir computing where the reservoir - a recurrent neural network with fixed connections - corresponds to cortex, and the readout corresponds to the striatum. We demonstrate robust learning, and generalization properties of the model, and demonstrate its cross linguistic capabilities in English and Japanese. The results contribute to the argument that the corticostriatal system plays a role in response selection in language production, and to the stance that reservoir computing is a valid potential model of corticostriatal processing.
Collapse
Affiliation(s)
- Xavier Hinaut
- CNPS, UMR CNRS 8195, University Paris-Sud, Orsay, France
| | - Florian Lance
- INSERM Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 18 Ave Lepine, 69675 Bron Cedex, France
| | - Colas Droin
- INSERM Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 18 Ave Lepine, 69675 Bron Cedex, France
| | - Maxime Petit
- INSERM Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 18 Ave Lepine, 69675 Bron Cedex, France
| | - Gregoire Pointeau
- INSERM Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 18 Ave Lepine, 69675 Bron Cedex, France
| | - Peter Ford Dominey
- INSERM Stem Cell and Brain Research Institute, Human and Robot Cognitive Systems, 18 Ave Lepine, 69675 Bron Cedex, France.
| |
Collapse
|
10
|
Hinaut X, Petit M, Pointeau G, Dominey PF. Exploring the acquisition and production of grammatical constructions through human-robot interaction with echo state networks. Front Neurorobot 2014; 8:16. [PMID: 24834050 PMCID: PMC4018555 DOI: 10.3389/fnbot.2014.00016] [Citation(s) in RCA: 36] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2014] [Accepted: 03/27/2014] [Indexed: 11/13/2022] Open
Abstract
One of the principal functions of human language is to allow people to coordinate joint action. This includes the description of events, requests for action, and their organization in time. A crucial component of language acquisition is learning the grammatical structures that allow the expression of such complex meaning related to physical events. The current research investigates the learning of grammatical constructions and their temporal organization in the context of human-robot physical interaction with the embodied sensorimotor humanoid platform, the iCub. We demonstrate three noteworthy phenomena. First, a recurrent network model is used in conjunction with this robotic platform to learn the mappings between grammatical forms and predicate-argument representations of meanings related to events, and the robot's execution of these events in time. Second, this learning mechanism functions in the inverse sense, i.e., in a language production mode, where rather than executing commanded actions, the robot will describe the results of human generated actions. Finally, we collect data from naïve subjects who interact with the robot via spoken language, and demonstrate significant learning and generalization results. This allows us to conclude that such a neural language learning system not only helps to characterize and understand some aspects of human language acquisition, but also that it can be useful in adaptive human-robot interaction.
Collapse
Affiliation(s)
- Xavier Hinaut
- Stem Cell and Brain Research Institute, INSERM U846 Bron, France ; Université de Lyon, Université Lyon I Lyon, France
| | - Maxime Petit
- Stem Cell and Brain Research Institute, INSERM U846 Bron, France ; Université de Lyon, Université Lyon I Lyon, France
| | - Gregoire Pointeau
- Stem Cell and Brain Research Institute, INSERM U846 Bron, France ; Université de Lyon, Université Lyon I Lyon, France
| | - Peter Ford Dominey
- Stem Cell and Brain Research Institute, INSERM U846 Bron, France ; Université de Lyon, Université Lyon I Lyon, France ; Centre National de la Recherche Scientifique Bron, France
| |
Collapse
|
11
|
Hinaut X, Dominey PF. A three-layered model of primate prefrontal cortex encodes identity and abstract categorical structure of behavioral sequences. ACTA ACUST UNITED AC 2011; 105:16-24. [PMID: 21939760 DOI: 10.1016/j.jphysparis.2011.07.010] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2011] [Revised: 06/03/2011] [Accepted: 07/13/2011] [Indexed: 01/21/2023]
Abstract
Categorical encoding is crucial for mastering large bodies of related sensory-motor experiences, but what is its neural substrate? In an effort to respond to this question, recent single-unit recording studies in the macaque lateral prefrontal cortex (LPFC) have demonstrated two characteristic forms of neural encoding of the sequential structure of the animal's sensory-motor experience. One population of neurons encodes the specific behavioral sequences. A second population of neurons encodes the sequence category (e.g. ABAB, AABB or AAAA) and does not differentiate sequences within the category (Shima, K., Isoda, M., Mushiake, H., Tanji, J., 2007. Categorization of behavioural sequences in the prefrontal cortex. Nature 445, 315-318.). Interestingly these neurons are intermingled in the lateral prefrontal cortex, and not topographically segregated. Thus, LPFC may provide a neurophysiological basis for sensorimotor categorization. Here we report on a neural network simulation study that reproduces and explains these results. We model a cortical circuit composed of three layers (infragranular, granular, and supragranular) of 5*5 leaky integrator neurons with a sigmoidal output function, and we examine 1000 such circuits running in parallel. Crucially the three layers are interconnected with recurrent connections, thus producing a dynamical system that is inherently sensitive to the spatiotemporal structure of the sequential inputs. The model is presented with 11 four-element sequences following Shima et al. We isolated one subpopulation of neurons each of whose activity predicts individual sequences, and a second population that predicts category independent of the specific sequence. We argue that a richly interconnected cortical circuit is capable of internally generating a neural representation of category membership, thus significantly extending the scope of recurrent network computation. In order to demonstrate that these representations can be used to create an explicit categorization capability, we introduced an additional neural structure corresponding to the striatum. We showed that via cortico-striatal plasticity, neurons in the striatum could produce an explicit representation both of the identity of each sequence, and its category membership.
Collapse
Affiliation(s)
- Xavier Hinaut
- Stem Cell and Brain Research Institute, INSERM U846, 18 Avenue du Doyen, Lepine, 69500 Bron, France.
| | | |
Collapse
|