1
|
Taniguchi A, Ito S, Taniguchi T. Hierarchical path planning from speech instructions with spatial concept-based topometric semantic mapping. Front Robot AI 2024; 11:1291426. [PMID: 39148580 PMCID: PMC11324419 DOI: 10.3389/frobt.2024.1291426] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2023] [Accepted: 06/20/2024] [Indexed: 08/17/2024] Open
Abstract
Assisting individuals in their daily activities through autonomous mobile robots is a significant concern, especially for users without specialized knowledge. Specifically, the capability of a robot to navigate to destinations based on human speech instructions is crucial. Although robots can take different paths toward the same objective, the shortest path is not always the most suitable. A preferred approach would be to accommodate waypoint specifications flexibly for planning an improved alternative path even with detours. Furthermore, robots require real-time inference capabilities. In this sense, spatial representations include semantic, topological, and metric-level representations, each capturing different aspects of the environment. This study aimed to realize a hierarchical spatial representation using a topometric semantic map and path planning with speech instructions by including waypoints. Thus, we present a hierarchical path planning method called spatial concept-based topometric semantic mapping for hierarchical path planning (SpCoTMHP), which integrates place connectivity. This approach provides a novel integrated probabilistic generative model and fast approximate inferences with interactions among the hierarchy levels. A formulation based on "control as probabilistic inference" theoretically supports the proposed path planning algorithm. We conducted experiments in a home environment using the Toyota human support robot on the SIGVerse simulator and in a lab-office environment with the real robot Albert. Here, the user issues speech commands that specify the waypoint and goal, such as "Go to the bedroom via the corridor." Navigation experiments were performed using speech instructions with a waypoint to demonstrate the performance improvement of the SpCoTMHP over the baseline hierarchical path planning method with heuristic path costs (HPP-I) in terms of the weighted success rate at which the robot reaches the closest target (0.590) and passes the correct waypoints. The computation time was significantly improved by 7.14 s with the SpCoTMHP than the baseline HPP-I in advanced tasks. Thus, hierarchical spatial representations provide mutually understandable instruction forms for both humans and robots, thus enabling language-based navigation.
Collapse
Affiliation(s)
- Akira Taniguchi
- Emergent Systems Laboratory, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Shuya Ito
- Emergent Systems Laboratory, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Tadahiro Taniguchi
- Emergent Systems Laboratory, Ritsumeikan University, Kusatsu, Shiga, Japan
| |
Collapse
|
2
|
Hoang NL, Taniguchi T, Hagiwara Y, Taniguchi A. Emergent communication of multimodal deep generative models based on Metropolis-Hastings naming game. Front Robot AI 2024; 10:1290604. [PMID: 38356917 PMCID: PMC10864618 DOI: 10.3389/frobt.2023.1290604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Accepted: 12/18/2023] [Indexed: 02/16/2024] Open
Abstract
Deep generative models (DGM) are increasingly employed in emergent communication systems. However, their application in multimodal data contexts is limited. This study proposes a novel model that combines multimodal DGM with the Metropolis-Hastings (MH) naming game, enabling two agents to focus jointly on a shared subject and develop common vocabularies. The model proves that it can handle multimodal data, even in cases of missing modalities. Integrating the MH naming game with multimodal variational autoencoders (VAE) allows agents to form perceptual categories and exchange signs within multimodal contexts. Moreover, fine-tuning the weight ratio to favor a modality that the model could learn and categorize more readily improved communication. Our evaluation of three multimodal approaches - mixture-of-experts (MoE), product-of-experts (PoE), and mixture-of-product-of-experts (MoPoE)-suggests an impact on the creation of latent spaces, the internal representations of agents. Our results from experiments with the MNIST + SVHN and Multimodal165 datasets indicate that combining the Gaussian mixture model (GMM), PoE multimodal VAE, and MH naming game substantially improved information sharing, knowledge formation, and data reconstruction.
Collapse
Affiliation(s)
- Nguyen Le Hoang
- Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Yoshinobu Hagiwara
- Research Organization of Science and Technology, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Akira Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| |
Collapse
|
3
|
Okumura R, Taniguchi T, Hagiwara Y, Taniguchi A. Metropolis-Hastings algorithm in joint-attention naming game: experimental semiotics study. Front Artif Intell 2023; 6:1235231. [PMID: 38116389 PMCID: PMC10728479 DOI: 10.3389/frai.2023.1235231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Accepted: 11/13/2023] [Indexed: 12/21/2023] Open
Abstract
We explore the emergence of symbols during interactions between individuals through an experimental semiotic study. Previous studies have investigated how humans organize symbol systems through communication using artificially designed subjective experiments. In this study, we focused on a joint-attention-naming game (JA-NG) in which participants independently categorized objects and assigned names while assuming their joint attention. In the Metropolis-Hastings naming game (MHNG) theory, listeners accept provided names according to the acceptance probability computed using the Metropolis-Hastings (MH) algorithm. The MHNG theory suggests that symbols emerge as an approximate decentralized Bayesian inference of signs, which is represented as a shared prior variable if the conditions of the MHNG are satisfied. This study examines whether human participants exhibit behavior consistent with the MHNG theory when playing the JA-NG. By comparing human acceptance decisions of a partner's naming with acceptance probabilities computed in the MHNG, we tested whether human behavior is consistent with the MHNG theory. The main contributions of this study are twofold. First, we reject the null hypothesis that humans make acceptance judgments with a constant probability, regardless of the acceptance probability calculated by the MH algorithm. The results of this study show that the model with acceptance probability computed by the MH algorithm predicts human behavior significantly better than the model with a constant probability of acceptance. Second, the MH-based model predicted human acceptance/rejection behavior more accurately than four other models (i.e., Constant, Numerator, Subtraction, Binary). Among the models compared, the model using the MH algorithm, which is the only model with the mathematical support of decentralized Bayesian inference, predicted human behavior most accurately, suggesting that symbol emergence in the JA-NG can be explained by the MHNG.
Collapse
Affiliation(s)
- Ryota Okumura
- Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Yoshinobu Hagiwara
- Research Organization of Science and Technology, Ritsumeikan University, Kusatsu, Japan
| | - Akira Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| |
Collapse
|
4
|
Inukai J, Taniguchi T, Taniguchi A, Hagiwara Y. Recursive Metropolis-Hastings naming game: symbol emergence in a multi-agent system based on probabilistic generative models. Front Artif Intell 2023; 6:1229127. [PMID: 37920571 PMCID: PMC10619661 DOI: 10.3389/frai.2023.1229127] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Accepted: 09/22/2023] [Indexed: 11/04/2023] Open
Abstract
In the studies on symbol emergence and emergent communication in a population of agents, a computational model was employed in which agents participate in various language games. Among these, the Metropolis-Hastings naming game (MHNG) possesses a notable mathematical property: symbol emergence through MHNG is proven to be a decentralized Bayesian inference of representations shared by the agents. However, the previously proposed MHNG is limited to a two-agent scenario. This paper extends MHNG to an N-agent scenario. The main contributions of this paper are twofold: (1) we propose the recursive Metropolis-Hastings naming game (RMHNG) as an N-agent version of MHNG and demonstrate that RMHNG is an approximate Bayesian inference method for the posterior distribution over a latent variable shared by agents, similar to MHNG; and (2) we empirically evaluate the performance of RMHNG on synthetic and real image data, i.e., YCB object dataset, enabling multiple agents to develop and share a symbol system. Furthermore, we introduce two types of approximations-one-sample and limited-length-to reduce computational complexity while maintaining the ability to explain communication in a population of agents. The experimental findings showcased the efficacy of RMHNG as a decentralized Bayesian inference for approximating the posterior distribution concerning latent variables, which are jointly shared among agents, akin to MHNG, although the improvement in ARI and κ coefficient is smaller in the real image dataset condition. Moreover, the utilization of RMHNG elucidated the agents' capacity to exchange symbols. Furthermore, the study discovered that even the computationally simplified version of RMHNG could enable symbols to emerge among the agents.
Collapse
Affiliation(s)
- Jun Inukai
- Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Tadahiro Taniguchi
- Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Akira Taniguchi
- Graduate School of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Yoshinobu Hagiwara
- Research Organization of Science and Technology, Ritsumeikan University, Kusatsu, Japan
| |
Collapse
|
5
|
Bolis D, Dumas G, Schilbach L. Interpersonal attunement in social interactions: from collective psychophysiology to inter-personalized psychiatry and beyond. Philos Trans R Soc Lond B Biol Sci 2023; 378:20210365. [PMID: 36571122 PMCID: PMC9791489 DOI: 10.1098/rstb.2021.0365] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022] Open
Abstract
In this article, we analyse social interactions, drawing on diverse points of views, ranging from dialectics, second-person neuroscience and enactivism to dynamical systems, active inference and machine learning. To this end, we define interpersonal attunement as a set of multi-scale processes of building up and materializing social expectations-put simply, anticipating and interacting with others and ourselves. While cultivating and negotiating common ground, via communication and culture-building activities, are indispensable for the survival of the individual, the relevant multi-scale mechanisms have been largely considered in isolation. Here, collective psychophysiology, we argue, can lend itself to the fine-tuned analysis of social interactions, without neglecting the individual. On the other hand, an interpersonal mismatch of expectations can lead to a breakdown of communication and social isolation known to negatively affect mental health. In this regard, we review psychopathology in terms of interpersonal misattunement, conceptualizing psychiatric disorders as disorders of social interaction, to describe how individual mental health is inextricably linked to social interaction. By doing so, we foresee avenues for an inter-personalized psychiatry, which moves from a static spectrum of disorders to a dynamic relational space, focusing on how the multi-faceted processes of social interaction can help to promote mental health. This article is part of the theme issue 'Concepts in interaction: social engagement and inner experiences'.
Collapse
Affiliation(s)
- Dimitris Bolis
- Independent Max Planck Research Group for Social Neuroscience, Max Planck Institute of Psychiatry, Kraepelinstrasse 2–10, Muenchen-Schwabing 80804, Germany,Centre for Philosophy of Science, University of Lisbon, Campo Grande, 1749-016 Lisbon, Portugal,Department of System Neuroscience, National Institute for Physiological Sciences (NIPS), Okazaki 444-0867, Japan
| | - Guillaume Dumas
- Precision Psychiatry and Social Physiology Laboratory, CHU Ste-Justine Research Center, Department of Psychiatry, University of Montreal, Quebec, Canada H3T 1J4,Mila - Quebec AI Institute, University of Montreal, Quebec, Canada H2S 3H1,Culture Mind and Brain Program, Department of Psychiatry, McGill University, Montreal, Quebec, Canada H3A 1A1
| | - Leonhard Schilbach
- Independent Max Planck Research Group for Social Neuroscience, Max Planck Institute of Psychiatry, Kraepelinstrasse 2–10, Muenchen-Schwabing 80804, Germany,Department of Psychiatry and Psychotherapy, University Hospital, Ludwig Maximilians Universität, Munich 40629, Germany,Department of General Psychiatry 2, LVR-Klinikum Düsseldorf, Düsseldorf 80336, Germany
| |
Collapse
|
6
|
Miyazawa K, Nagai T. Concept formation through multimodal integration using multimodal BERT and VQ-VAE. Adv Robot 2022. [DOI: 10.1080/01691864.2022.2141583] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Affiliation(s)
- Kazuki Miyazawa
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
| | - Takayuki Nagai
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
- Artificial Intelligence Exploration Research Center, The University of Electro-Communications, Tokyo, Japan
| |
Collapse
|
7
|
El Hafi L, Garcia Ricardez GA, von Drigalski F, Inoue Y, Yamamoto M, Yamamoto T. Software development environment for collaborative research workflow in robotic system integration. Adv Robot 2022. [DOI: 10.1080/01691864.2022.2068353] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Affiliation(s)
- Lotfi El Hafi
- Ritsumeikan Global Innovation Research Organization, Ritsumeikan University, Kusatsu, Japan
- Coarobo GK, Seika, Japan
| | - Gustavo Alfonso Garcia Ricardez
- Ritsumeikan Global Innovation Research Organization, Ritsumeikan University, Kusatsu, Japan
- Coarobo GK, Seika, Japan
- Division of Information Science, Nara Institute of Science and Technology, Ikoma, Japan
| | - Felix von Drigalski
- Ritsumeikan Global Innovation Research Organization, Ritsumeikan University, Kusatsu, Japan
- Coarobo GK, Seika, Japan
- Robotics Group, OMRON SINIC X Corporation, Tokyo, Japan
| | - Yuki Inoue
- Faculty of Robotics and Design, Osaka Institute of Technology, Osaka, Japan
| | - Masaki Yamamoto
- Manufacturing Innovation Division, Panasonic Corporation, Osaka, Japan
| | - Takashi Yamamoto
- Frontier Research Center, Toyota Motor Corporation, Toyota, Japan
| |
Collapse
|
8
|
Taniguchi T, Yamakawa H, Nagai T, Doya K, Sakagami M, Suzuki M, Nakamura T, Taniguchi A. A whole brain probabilistic generative model: Toward realizing cognitive architectures for developmental robots. Neural Netw 2022; 150:293-312. [PMID: 35339010 DOI: 10.1016/j.neunet.2022.02.026] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 02/25/2022] [Accepted: 02/28/2022] [Indexed: 01/08/2023]
Abstract
Building a human-like integrative artificial cognitive system, that is, an artificial general intelligence (AGI), is the holy grail of the artificial intelligence (AI) field. Furthermore, a computational model that enables an artificial system to achieve cognitive development will be an excellent reference for brain and cognitive science. This paper describes an approach to develop a cognitive architecture by integrating elemental cognitive modules to enable the training of the modules as a whole. This approach is based on two ideas: (1) brain-inspired AI, learning human brain architecture to build human-level intelligence, and (2) a probabilistic generative model (PGM)-based cognitive architecture to develop a cognitive system for developmental robots by integrating PGMs. The proposed development framework is called a whole brain PGM (WB-PGM), which differs fundamentally from existing cognitive architectures in that it can learn continuously through a system based on sensory-motor information. In this paper, we describe the rationale for WB-PGM, the current status of PGM-based elemental cognitive modules, their relationship with the human brain, the approach to the integration of the cognitive modules, and future challenges. Our findings can serve as a reference for brain studies. As PGMs describe explicit informational relationships between variables, WB-PGM provides interpretable guidance from computational sciences to brain science. By providing such information, researchers in neuroscience can provide feedback to researchers in AI and robotics on what the current models lack with reference to the brain. Further, it can facilitate collaboration among researchers in neuro-cognitive sciences as well as AI and robotics.
Collapse
Affiliation(s)
| | - Hiroshi Yamakawa
- The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, Japan; The Whole Brain Architecture Initiative, 2-19-21 Nishikoiwa , Edogawa-ku, Tokyo, Japan; RIKEN, 6-2-3 Furuedai, Suita, Osaka, Japan
| | - Takayuki Nagai
- Osaka University, 1-3 Machikane-yama, Toyonaka, Osaka, Japan
| | - Kenji Doya
- Okinawa Institute of Science and Technology Graduate University, 1919-1 Tancha, Onna-son, Kunigami, Okinawa, Japan
| | | | - Masahiro Suzuki
- The University of Tokyo, 7-3-1, Hongo, Bunkyo-ku, Tokyo, Japan
| | - Tomoaki Nakamura
- The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo, Japan
| | | |
Collapse
|
9
|
Nishikawa J, Morita J. Cognitive model of phonological awareness focusing on errors and formation process through Shiritori. Adv Robot 2022. [DOI: 10.1080/01691864.2022.2029763] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Affiliation(s)
- Jumpei Nishikawa
- Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka, Japan
| | - Junya Morita
- Graduate School of Integrated Science and Technology, Shizuoka University, Shizuoka, Japan
| |
Collapse
|
10
|
Affiliation(s)
- Tatsuya Sakai
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
| | - Takayuki Nagai
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
- Artificial Intelligence Exploration Research Center, University of Electro-Communications, Tokyo, Japan
| |
Collapse
|
11
|
Hagiwara Y, Furukawa K, Taniguchi A, Taniguchi T. Multiagent multimodal categorization for symbol emergence: emergent communication via interpersonal cross-modal inference. Adv Robot 2022. [DOI: 10.1080/01691864.2022.2029721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Affiliation(s)
- Yoshinobu Hagiwara
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Kazuma Furukawa
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Akira Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| |
Collapse
|
12
|
Hieida C, Nagai T. Survey and perspective on social emotions in robotics. Adv Robot 2022. [DOI: 10.1080/01691864.2021.2012512] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Affiliation(s)
- Chie Hieida
- Division of Information Science, Graduate School of Science and Technology, Nara Institute of Science and Technology, Ikoma, Japan
| | - Takayuki Nagai
- Department of Systems Innovation, Graduate School of Engineering Science, Osaka University, Suita, Japan
- The University of Electro-Communications, Chofu, Japan
| |
Collapse
|
13
|
Sagara R, Taguchi R, Taniguchi A, Taniguchi T, Hattori K, Hoguro M, Umezaki T. Unsupervised lexical acquisition of relative spatial concepts using spoken user utterances. Adv Robot 2021. [DOI: 10.1080/01691864.2021.2007168] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Rikunari Sagara
- Taguchi Laboratory, Department of Computer Science, Nagoya Institute of Technology, Aichi, Japan
| | - Ryo Taguchi
- Taguchi Laboratory, Department of Computer Science, Nagoya Institute of Technology, Aichi, Japan
| | - Akira Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tadahiro Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | | | | | - Taizo Umezaki
- College of Engineering, Chubu University, Aichi, Japan
| |
Collapse
|
14
|
Noguchi W, Iizuka H, Yamamoto M. Multi-modal shared module that enables the bottom-up formation of map representation and top-down map reading. Adv Robot 2021. [DOI: 10.1080/01691864.2021.1993334] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Wataru Noguchi
- Faculty of Information Science and Technology, Hokkaido University, Sapporo, Japan
| | - Hiroyuki Iizuka
- Faculty of Information Science and Technology, Hokkaido University, Sapporo, Japan
- Center for Human Nature, Artificial Intelligence, and Neuroscience, Hokkaido University, Sapporo, Japan
| | - Masahito Yamamoto
- Faculty of Information Science and Technology, Hokkaido University, Sapporo, Japan
- Center for Human Nature, Artificial Intelligence, and Neuroscience, Hokkaido University, Sapporo, Japan
| |
Collapse
|
15
|
Friston K, Moran RJ, Nagai Y, Taniguchi T, Gomi H, Tenenbaum J. World model learning and inference. Neural Netw 2021; 144:573-590. [PMID: 34634605 DOI: 10.1016/j.neunet.2021.09.011] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2021] [Revised: 07/28/2021] [Accepted: 09/09/2021] [Indexed: 11/19/2022]
Abstract
Understanding information processing in the brain-and creating general-purpose artificial intelligence-are long-standing aspirations of scientists and engineers worldwide. The distinctive features of human intelligence are high-level cognition and control in various interactions with the world including the self, which are not defined in advance and are vary over time. The challenge of building human-like intelligent machines, as well as progress in brain science and behavioural analyses, robotics, and their associated theoretical formalisations, speaks to the importance of the world-model learning and inference. In this article, after briefly surveying the history and challenges of internal model learning and probabilistic learning, we introduce the free energy principle, which provides a useful framework within which to consider neuronal computation and probabilistic world models. Next, we showcase examples of human behaviour and cognition explained under that principle. We then describe symbol emergence in the context of probabilistic modelling, as a topic at the frontiers of cognitive robotics. Lastly, we review recent progress in creating human-like intelligence by using novel probabilistic programming languages. The striking consensus that emerges from these studies is that probabilistic descriptions of learning and inference are powerful and effective ways to create human-like artificial intelligent machines and to understand intelligence in the context of how humans interact with their world.
Collapse
Affiliation(s)
- Karl Friston
- Wellcome Centre for Human Neuroimaging, Institute of Neurology, University College London (UCL), WC1N 3BG, UK.
| | - Rosalyn J Moran
- Department of Neuroimaging, Institute of Psychiatry, Psychology & Neuroscience, King's College London, SE5 8AF, UK.
| | - Yukie Nagai
- International Research Center for Neurointelligence (IRCN), The University of Tokyo, Tokyo, Japan.
| | - Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan.
| | - Hiroaki Gomi
- NTT Communication Science Labs., Nippon Telegraph and Telephone, Kanawaga, Japan.
| | - Josh Tenenbaum
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA; The Center for Brains, Minds and Machines, MIT, Cambridge, MA, USA.
| |
Collapse
|
16
|
Kuniyasu R, Nakamura T, Taniguchi T, Nagai T. Robot Concept Acquisition Based on Interaction Between Probabilistic and Deep Generative Models. FRONTIERS IN COMPUTER SCIENCE 2021. [DOI: 10.3389/fcomp.2021.618069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
We propose a method for multimodal concept formation. In this method, unsupervised multimodal clustering and cross-modal inference, as well as unsupervised representation learning, can be performed by integrating the multimodal latent Dirichlet allocation (MLDA)-based concept formation and variational autoencoder (VAE)-based feature extraction. Multimodal clustering, representation learning, and cross-modal inference are critical for robots to form multimodal concepts from sensory data. Various models have been proposed for concept formation. However, in previous studies, features were extracted using manually designed or pre-trained feature extractors and representation learning was not performed simultaneously. Moreover, the generative probabilities of the features extracted from the sensory data could be predicted, but the sensory data could not be predicted in the cross-modal inference. Therefore, a method that can perform clustering, feature learning, and cross-modal inference among multimodal sensory data is required for concept formation. To realize such a method, we extend the VAE to the multinomial VAE (MNVAE), the latent variables of which follow a multinomial distribution, and construct a model that integrates the MNVAE and MLDA. In the experiments, the multimodal information of the images and words acquired by a robot was classified using the integrated model. The results demonstrated that the integrated model can classify the multimodal information as accurately as the previous model despite the feature extractor learning in an unsupervised manner, suitable image features for clustering can be learned, and cross-modal inference from the words to images is possible.
Collapse
|
17
|
Taniguchi T, El Hafi L, Hagiwara Y, Taniguchi A, Shimada N, Nishiura T. Semiotically adaptive cognition: toward the realization of remotely-operated service robots for the new normal symbiotic society. Adv Robot 2021. [DOI: 10.1080/01691864.2021.1928552] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 09/30/2022]
Affiliation(s)
- Tadahiro Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Lotfi El Hafi
- Ritsumeikan Global Innovation Research Organization, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Yoshinobu Hagiwara
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Akira Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Nobutaka Shimada
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| | - Takanobu Nishiura
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Shiga, Japan
| |
Collapse
|
18
|
Novoa J, Mahu R, Wuth J, Escudero JP, Fredes J, Yoma NB. Automatic Speech Recognition for Indoor HRI Scenarios. ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2021. [DOI: 10.1145/3442629] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/22/2022]
Abstract
This article presents a stand-alone automatic speech recognition system that accounts for listener movement, time-varying reverberation effects, environmental noise, and user position information for beamforming approaches in an HRI setting. We raise the importance of replacing the classical black-box integration of automatic speech recognition technology in HRI applications with the incorporation of the acoustic environment representation and modeling, and of the target source direction. Test data were recorded on a real robot under various moving conditions. For addressing the time-varying acoustic channel problem and incorporating environmental effect during training, clean speech samples were passed through estimated static channel responses and noise was added. Beamforming is investigated regarding oracle source tracking using, for instance, image processing. The proposed strategy is interesting for the robotics community, because it allows the development of voice-based HRI with limited training data and without relying on third-party technologies or Internet access eliminating the need to upload data to the cloud. In our mobile HRI scenario, the resulting speech recognition engine provided an average word error rate that is at least 19% and 34% lower than publicly available speech recognition APIs with the playback (i.e., loudspeaker) and human testing modalities, respectively.
Collapse
Affiliation(s)
- José Novoa
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| | - Rodrigo Mahu
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| | - Jorge Wuth
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| | - Juan Pablo Escudero
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| | - Josué Fredes
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| | - Néstor Becerra Yoma
- Speech Processing and Transmission Laboratory, University of Chile, Santiago, Chile
| |
Collapse
|
19
|
Taniguchi T, Horii T, Hinaut X, Spranger M, Mochihashi D, Nagai T. Editorial: Language and Robotics. Front Robot AI 2021; 8:674832. [PMID: 33912598 PMCID: PMC8072269 DOI: 10.3389/frobt.2021.674832] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2021] [Accepted: 03/08/2021] [Indexed: 11/13/2022] Open
Affiliation(s)
- Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| | - Takato Horii
- Graduate School of Engineering Science, Osaka University, Suita, Japan
| | - Xavier Hinaut
- Institut National de Recherche en Informatique et en Automatique (INRIA), Rocquencourt, Île-de-France, France
| | | | - Daichi Mochihashi
- Department of Statistical Inference and Mathematics, The Institute of Statistical Mathematics, Tokyo, Japan
| | - Takayuki Nagai
- Graduate School of Engineering Science, Osaka University, Suita, Japan
| |
Collapse
|
20
|
Bennett MT, Maruyama Y. Philosophical Specification of Empathetic Ethical Artificial Intelligence. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2021.3099945] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
21
|
Taniguchi A, Hagiwara Y, Taniguchi T, Inamura T. Spatial concept-based navigation with human speech instructions via probabilistic inference on Bayesian generative model. Adv Robot 2020. [DOI: 10.1080/01691864.2020.1817777] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
Affiliation(s)
- Akira Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Yoshinobu Hagiwara
- College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tadahiro Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tetsunari Inamura
- National Institute of Informatics, The Graduate University for Advanced Studies, SOKENDAI, Tokyo, Japan
| |
Collapse
|
22
|
Hinaut X, Twiefel J. Teach Your Robot Your Language! Trainable Neural Parser for Modeling Human Sentence Processing: Examples for 15 Languages. IEEE Trans Cogn Dev Syst 2020. [DOI: 10.1109/tcds.2019.2957006] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
23
|
Kalantari S, Nazemi E, Masoumi B. Emergence phenomena in self-organizing systems: a systematic literature review of concepts, researches, and future prospects. JOURNAL OF ORGANIZATIONAL COMPUTING AND ELECTRONIC COMMERCE 2020. [DOI: 10.1080/10919392.2020.1748977] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Affiliation(s)
- Somayeh Kalantari
- Faculty of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran
| | - Eslam Nazemi
- Faculty of Computer Science & Engineering, Shahid Beheshti University, Tehran, Iran
| | - Behrooz Masoumi
- Faculty of Computer and Information Technology Engineering, Qazvin Branch, Islamic Azad University, Qazvin, Iran
| |
Collapse
|
24
|
Gärdenfors P. Using Event Representations to Generate Robot Semantics. ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2019. [DOI: 10.1145/3341167] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Abstract
Most semantic models employed in human-robot interactions concern how a robot can understand commands, but in this article the aim is to present a framework that allows dialogic interaction. The key idea is to use events as the fundamental structures for the semantic representations of a robot. Events are modeled in terms of conceptual spaces and mappings between spaces. It is shown how the semantics of major word classes can be described with the aid of conceptual spaces in a way that is amenable to computer implementations. An event is represented by two vectors, one force vector representing an action and one result vector representing the effect of the action. The two-vector model is then extended by the thematic roles so an event is built up from an agent, an action, a patient, and a result. It is shown how the components of an event can be put together to semantic structures that represent the meanings of sentences. It is argued that a semantic framework based on events can generate a general representational framework for human-robot communication. An implementation of the framework involving communication with an iCub will be described.
Collapse
Affiliation(s)
- Peter Gärdenfors
- Lund University and University of Technology Sydney, Lund, Sweden
| |
Collapse
|
25
|
Hagiwara Y, Kobayashi H, Taniguchi A, Taniguchi T. Symbol Emergence as an Interpersonal Multimodal Categorization. Front Robot AI 2019; 6:134. [PMID: 33501149 PMCID: PMC7805687 DOI: 10.3389/frobt.2019.00134] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2019] [Accepted: 11/19/2019] [Indexed: 11/13/2022] Open
Abstract
This study focuses on category formation for individual agents and the dynamics of symbol emergence in a multi-agent system through semiotic communication. In this study, the semiotic communication refers to exchanging signs composed of the signifier (i.e., words) and the signified (i.e., categories). We define the generation and interpretation of signs associated with the categories formed through the agent's own sensory experience or by exchanging signs with other agents as basic functions of the semiotic communication. From the viewpoint of language evolution and symbol emergence, organization of a symbol system in a multi-agent system (i.e., agent society) is considered as a bottom-up and dynamic process, where individual agents share the meaning of signs and categorize sensory experience. A constructive computational model can explain the mutual dependency of the two processes and has mathematical support that guarantees a symbol system's emergence and sharing within the multi-agent system. In this paper, we describe a new computational model that represents symbol emergence in a two-agent system based on a probabilistic generative model for multimodal categorization. It models semiotic communication via a probabilistic rejection based on the receiver's own belief. We have found that the dynamics by which cognitively independent agents create a symbol system through their semiotic communication can be regarded as the inference process of a hidden variable in an interpersonal multimodal categorizer, i.e., the complete system can be regarded as a single agent performing multimodal categorization using the sensors of all agents, if we define the rejection probability based on the Metropolis-Hastings algorithm. The validity of the proposed model and algorithm for symbol emergence, i.e., forming and sharing signs and categories, is also verified in an experiment with two agents observing daily objects in the real-world environment. In the experiment, we compared three communication algorithms: no communication, no rejection, and the proposed algorithm. The experimental results demonstrate that our model reproduces the phenomena of symbol emergence, which does not require a teacher who would know a pre-existing symbol system. Instead, the multi-agent system can form and use a symbol system without having pre-existing categories.
Collapse
Affiliation(s)
- Yoshinobu Hagiwara
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Hiroyoshi Kobayashi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Akira Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tadahiro Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| |
Collapse
|
26
|
Taniguchi T, Piater J, Worgotter F, Ugur E, Hoffmann M, Jamone L, Nagai T, Rosman B, Matsuka T, Iwahashi N, Oztop E. Symbol Emergence in Cognitive Developmental Systems: A Survey. IEEE Trans Cogn Dev Syst 2019. [DOI: 10.1109/tcds.2018.2867772] [Citation(s) in RCA: 32] [Impact Index Per Article: 6.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
27
|
Miyazawa K, Horii T, Aoki T, Nagai T. Integrated Cognitive Architecture for Robot Learning of Action and Language. Front Robot AI 2019; 6:131. [PMID: 33501146 PMCID: PMC7805838 DOI: 10.3389/frobt.2019.00131] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2019] [Accepted: 11/13/2019] [Indexed: 11/13/2022] Open
Abstract
The manner in which humans learn, plan, and decide actions is a very compelling subject. Moreover, the mechanism behind high-level cognitive functions, such as action planning, language understanding, and logical thinking, has not yet been fully implemented in robotics. In this paper, we propose a framework for the simultaneously comprehension of concepts, actions, and language as a first step toward this goal. This can be achieved by integrating various cognitive modules and leveraging mainly multimodal categorization by using multilayered multimodal latent Dirichlet allocation (mMLDA). The integration of reinforcement learning and mMLDA enables actions based on understanding. Furthermore, the mMLDA, in conjunction with grammar learning and based on the Bayesian hidden Markov model (BHMM), allows the robot to verbalize its own actions and understand user utterances. We verify the potential of the proposed architecture through experiments using a real robot.
Collapse
Affiliation(s)
- Kazuki Miyazawa
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
| | - Takato Horii
- Graduate School of Engineering Science, Osaka University, Osaka, Japan
| | - Tatsuya Aoki
- Graduate School of Engineering Science, Osaka University, Osaka, Japan.,Graduate School of Informatics and Engineering, The University of Electro-Communications, Tokyo, Japan
| | - Takayuki Nagai
- Graduate School of Engineering Science, Osaka University, Osaka, Japan.,Artificial Intelligence Exploration Research Center, The University of Electro-Communications, Tokyo, Japan
| |
Collapse
|
28
|
El Hafi L, Isobe S, Tabuchi Y, Katsumata Y, Nakamura H, Fukui T, Matsuo T, Garcia Ricardez GA, Yamamoto M, Taniguchi A, Hagiwara Y, Taniguchi T. System for augmented human–robot interaction through mixed reality and robot training by non-experts in customer service environments. Adv Robot 2019. [DOI: 10.1080/01691864.2019.1694068] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
Affiliation(s)
- L. El Hafi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - S. Isobe
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Y. Tabuchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Y. Katsumata
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - H. Nakamura
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - T. Fukui
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - T. Matsuo
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - G. A. Garcia Ricardez
- Division of Information Science, Nara Institute of Science and Technology, Ikoma, Japan
| | - M. Yamamoto
- Business Innovation Division, Panasonic Corporation, Osaka, Japan
| | - A. Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - Y. Hagiwara
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - T. Taniguchi
- College of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| |
Collapse
|
29
|
Taniguchi T, Ugur E, Ogata T, Nagai T, Demiris Y. Editorial: Machine Learning Methods for High-Level Cognitive Capabilities in Robotics. Front Neurorobot 2019; 13:83. [PMID: 31695604 PMCID: PMC6817914 DOI: 10.3389/fnbot.2019.00083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/19/2019] [Accepted: 09/25/2019] [Indexed: 11/18/2022] Open
Affiliation(s)
- Tadahiro Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| | - Emre Ugur
- Department of Computer Engineering, Boǧaziçi University, Istanbul, Turkey
| | - Tetsuya Ogata
- Department of Intermedia Art and Science, School of Fundamental Science and Engineering, Waseda University, Tokyo, Japan
| | - Takayuki Nagai
- Department of Systems Innovation, Graduate School of Engineering Science, Osaka University, Osaka, Japan
| | - Yiannis Demiris
- Department of Electrical and Electronic Engineering, Imperial College London, London, United Kingdom
| |
Collapse
|
30
|
|
31
|
Nakashima R, Ozaki R, Taniguchi T. Unsupervised Phoneme and Word Discovery From Multiple Speakers Using Double Articulation Analyzer and Neural Network With Parametric Bias. Front Robot AI 2019; 6:92. [PMID: 33501107 PMCID: PMC7805918 DOI: 10.3389/frobt.2019.00092] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2019] [Accepted: 09/09/2019] [Indexed: 11/13/2022] Open
Abstract
This paper describes a new unsupervised machine-learning method for simultaneous phoneme and word discovery from multiple speakers. Phoneme and word discovery from multiple speakers is a more challenging problem than that from one speaker, because the speech signals from different speakers exhibit different acoustic features. The existing method, a nonparametric Bayesian double articulation analyzer (NPB-DAA) with deep sparse autoencoder (DSAE) only performed phoneme and word discovery from a single speaker. Extending NPB-DAA with DSAE to a multi-speaker scenario is, therefore, the research problem of this paper.This paper proposes the employment of a DSAE with parametric bias in the hidden layer (DSAE-PBHL) as a feature extractor for unsupervised phoneme and word discovery. DSAE-PBHL is designed to subtract speaker-dependent acoustic features and speaker-independent features by introducing parametric bias input to the DSAE hidden layer. An experiment demonstrated that DSAE-PBHL could subtract distributed representations of acoustic signals, enabling extraction based on the types of phonemes rather than the speakers. Another experiment demonstrated that a combination of NPB-DAA and DSAE-PBHL outperformed other available methods accomplishing phoneme and word discovery tasks involving speech signals with Japanese vowel sequences from multiple speakers.
Collapse
Affiliation(s)
- Ryo Nakashima
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Ryo Ozaki
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| | - Tadahiro Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| |
Collapse
|
32
|
Mici L, Parisi GI, Wermter S. Compositional Learning of Human Activities With a Self-Organizing Neural Architecture. Front Robot AI 2019; 6:72. [PMID: 33501087 PMCID: PMC7805845 DOI: 10.3389/frobt.2019.00072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2019] [Accepted: 07/30/2019] [Indexed: 12/02/2022] Open
Abstract
An important step for assistive systems and robot companions operating in human environments is to learn the compositionality of human activities, i.e., recognize both activities and their comprising actions. Most existing approaches address action and activity recognition as separate tasks, i.e., actions need to be inferred before the activity labels, and are thus highly sensitive to the correct temporal segmentation of the activity sequences. In this paper, we present a novel learning approach that jointly learns human activities on two levels of semantic and temporal complexity: (1) transitive actions such as reaching and opening, e.g., a cereal box, and (2) high-level activities such as having breakfast. Our model consists of a hierarchy of GWR networks which process and learn inherent spatiotemporal dependencies of multiple visual cues extracted from the human body skeletal representation and the interaction with objects. The neural architecture learns and semantically segments input RGB-D sequences of high-level activities into their composing actions, without supervision. We investigate the performance of our architecture with a set of experiments on a publicly available benchmark dataset. The experimental results show that our approach outperforms the state of the art with respect to the classification of the high-level activities. Additionally, we introduce a novel top-down modulation mechanism to the architecture which uses the actions and activity labels as constraints during the learning phase. In our experiments, we show how this mechanism can be used to control the network's neural growth without decreasing the overall performance.
Collapse
|
33
|
Taniguchi T, Mochihashi D, Nagai T, Uchida S, Inoue N, Kobayashi I, Nakamura T, Hagiwara Y, Iwahashi N, Inamura T. Survey on frontiers of language and robotics. Adv Robot 2019. [DOI: 10.1080/01691864.2019.1632223] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/09/2023]
Affiliation(s)
- T. Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - D. Mochihashi
- The Institute of Statistical Mathematics, Tachikawa, Japan
- SOKENDAI (The Graduate University for Advanced Studies), Tokyo, Japan
| | - T. Nagai
- Graduate School of Engineering Science, Osaka University, Toyonaka, Osaka, Japan
| | - S. Uchida
- Faculty of Languages and Cultures, Kyushu University, Fukuoka, Japan
| | - N. Inoue
- Graduate School of Information Sciences, Tohoku University, Sendai, Japan
- RIKEN Center for Advanced Intelligence Project, Chuo-ku, Japan
| | - I. Kobayashi
- Advanced Sciences, Graduate School of Humanities and Sciences, Ochanomizu University, Tokyo, Japan
| | - T. Nakamura
- Department of Mechanical Engineering and Intelligent Systems, The University of Electro-Communications, Chofu, Japan
| | - Y. Hagiwara
- Department of Information Science and Engineering, Ritsumeikan University, Kusatsu, Japan
| | - N. Iwahashi
- Department of Information and Communication Engineering, Okayama Prefectural University, Okayama, Japan
| | - T. Inamura
- SOKENDAI (The Graduate University for Advanced Studies), Tokyo, Japan
- National Institute of Informatics, Tokyo, Japan
| |
Collapse
|
34
|
Katsumata Y, Taniguchi A, Hagiwara Y, Taniguchi T. Semantic Mapping Based on Spatial Concepts for Grounding Words Related to Places in Daily Environments. Front Robot AI 2019; 6:31. [PMID: 33501047 PMCID: PMC7805848 DOI: 10.3389/frobt.2019.00031] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2018] [Accepted: 04/08/2019] [Indexed: 11/13/2022] Open
Abstract
An autonomous robot performing tasks in a human environment needs to recognize semantic information about places. Semantic mapping is a task in which suitable semantic information is assigned to an environmental map so that a robot can communicate with people and appropriately perform tasks requested by its users. We propose a novel statistical semantic mapping method called SpCoMapping, which integrates probabilistic spatial concept acquisition based on multimodal sensor information and a Markov random field applied for learning the arbitrary shape of a place on a map.SpCoMapping can connect multiple words to a place in a semantic mapping process using user utterances without pre-setting the list of place names. We also develop a nonparametric Bayesian extension of SpCoMapping that can automatically estimate an adequate number of categories. In the experiment in the simulation environments, we showed that the proposed method generated better semantic maps than previous semantic mapping methods; our semantic maps have categories and shapes similar to the ground truth provided by the user. In addition, we showed that SpCoMapping could generate appropriate semantic maps in a real-world environment.
Collapse
Affiliation(s)
- Yuki Katsumata
- Department of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| | - Akira Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| | - Yoshinobu Hagiwara
- Department of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| | - Tadahiro Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Kyoto, Japan
| |
Collapse
|
35
|
Nakamura T, Nagai T. Ensemble-of-Concept Models for Unsupervised Formation of Multiple Categories. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2745502] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
36
|
Moulin-Frier C, Fischer T, Petit M, Pointeau G, Puigbo JY, Pattacini U, Low SC, Camilleri D, Nguyen P, Hoffmann M, Chang HJ, Zambelli M, Mealier AL, Damianou A, Metta G, Prescott TJ, Demiris Y, Dominey PF, Verschure PFMJ. DAC-h3: A Proactive Robot Cognitive Architecture to Acquire and Express Knowledge About the World and the Self. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2754143] [Citation(s) in RCA: 29] [Impact Index Per Article: 4.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
37
|
Sturdivant RL, Chong EKP. The Necessary and Sufficient Conditions for Emergence in Systems Applied to Symbol Emergence in Robots. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2731361] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
38
|
Nakamura T, Nagai T, Taniguchi T. SERKET: An Architecture for Connecting Stochastic Models to Realize a Large-Scale Cognitive Model. Front Neurorobot 2018; 12:25. [PMID: 29997493 PMCID: PMC6028621 DOI: 10.3389/fnbot.2018.00025] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2017] [Accepted: 05/14/2018] [Indexed: 11/24/2022] Open
Abstract
To realize human-like robot intelligence, a large-scale cognitive architecture is required for robots to understand their environment through a variety of sensors with which they are equipped. In this paper, we propose a novel framework named Serket that enables the construction of a large-scale generative model and its inferences easily by connecting sub-modules to allow the robots to acquire various capabilities through interaction with their environment and others. We consider that large-scale cognitive models can be constructed by connecting smaller fundamental models hierarchically while maintaining their programmatic independence. Moreover, the connected modules are dependent on each other and their parameters must be optimized as a whole. Conventionally, the equations for parameter estimation have to be derived and implemented depending on the models. However, it has become harder to derive and implement equations of large-scale models. Thus, in this paper, we propose a parameter estimation method that communicates the minimum parameters between various modules while maintaining their programmatic independence. Therefore, Serket makes it easy to construct large-scale models and estimate their parameters via the connection of modules. Experimental results demonstrated that the model can be constructed by connecting modules, the parameters can be optimized as a whole, and they are comparable with the original models that we have proposed.
Collapse
Affiliation(s)
- Tomoaki Nakamura
- Department of Mechanical Engineering and Intelligent Systems, University of Electro-Communications, Tokyo, Japan
| | - Takayuki Nagai
- Department of Mechanical Engineering and Intelligent Systems, University of Electro-Communications, Tokyo, Japan
| | - Tadahiro Taniguchi
- Department of Information Science and Engineering, Ritsumeikan University, Shiga, Japan
| |
Collapse
|
39
|
Hwang J, Tani J. Seamless Integration and Coordination of Cognitive Skills in Humanoid Robots: A Deep Learning Approach. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2714170] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
40
|
Taniguchi T, Yoshino R, Takano T. Multimodal Hierarchical Dirichlet Process-Based Active Perception by a Robot. Front Neurorobot 2018; 12:22. [PMID: 29872389 PMCID: PMC5972223 DOI: 10.3389/fnbot.2018.00022] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2017] [Accepted: 04/23/2018] [Indexed: 11/13/2022] Open
Abstract
In this paper, we propose an active perception method for recognizing object categories based on the multimodal hierarchical Dirichlet process (MHDP). The MHDP enables a robot to form object categories using multimodal information, e.g., visual, auditory, and haptic information, which can be observed by performing actions on an object. However, performing many actions on a target object requires a long time. In a real-time scenario, i.e., when the time is limited, the robot has to determine the set of actions that is most effective for recognizing a target object. We propose an active perception for MHDP method that uses the information gain (IG) maximization criterion and lazy greedy algorithm. We show that the IG maximization criterion is optimal in the sense that the criterion is equivalent to a minimization of the expected Kullback-Leibler divergence between a final recognition state and the recognition state after the next set of actions. However, a straightforward calculation of IG is practically impossible. Therefore, we derive a Monte Carlo approximation method for IG by making use of a property of the MHDP. We also show that the IG has submodular and non-decreasing properties as a set function because of the structure of the graphical model of the MHDP. Therefore, the IG maximization problem is reduced to a submodular maximization problem. This means that greedy and lazy greedy algorithms are effective and have a theoretical justification for their performance. We conducted an experiment using an upper-torso humanoid robot and a second one using synthetic data. The experimental results show that the method enables the robot to select a set of actions that allow it to recognize target objects quickly and accurately. The numerical experiment using the synthetic data shows that the proposed method can work appropriately even when the number of actions is large and a set of target objects involves objects categorized into multiple classes. The results support our theoretical outcomes.
Collapse
Affiliation(s)
- Tadahiro Taniguchi
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, KsatsuJapan
| | - Ryo Yoshino
- Emergent Systems Laboratory, College of Information Science and Engineering, Ritsumeikan University, KsatsuJapan
| | - Toshiaki Takano
- Adaptive Systems Laboratory, Department of Computer Science, Shizuoka Institute of Science and Technology, Fukuroi, Japan
| |
Collapse
|
41
|
Taniguchi A, Taniguchi T, Cangelosi A. Cross-Situational Learning with Bayesian Generative Models for Multimodal Category and Word Learning in Robots. Front Neurorobot 2018; 11:66. [PMID: 29311888 PMCID: PMC5742219 DOI: 10.3389/fnbot.2017.00066] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2017] [Accepted: 11/21/2017] [Indexed: 11/24/2022] Open
Abstract
In this paper, we propose a Bayesian generative model that can form multiple categories based on each sensory-channel and can associate words with any of the four sensory-channels (action, position, object, and color). This paper focuses on cross-situational learning using the co-occurrence between words and information of sensory-channels in complex situations rather than conventional situations of cross-situational learning. We conducted a learning scenario using a simulator and a real humanoid iCub robot. In the scenario, a human tutor provided a sentence that describes an object of visual attention and an accompanying action to the robot. The scenario was set as follows: the number of words per sensory-channel was three or four, and the number of trials for learning was 20 and 40 for the simulator and 25 and 40 for the real robot. The experimental results showed that the proposed method was able to estimate the multiple categorizations and to learn the relationships between multiple sensory-channels and words accurately. In addition, we conducted an action generation task and an action description task based on word meanings learned in the cross-situational learning scenario. The experimental results showed that the robot could successfully use the word meanings learned by using the proposed method.
Collapse
Affiliation(s)
- Akira Taniguchi
- Emergent Systems Laboratory, Ritsumeikan University, Kusatsu, Japan
| | | | - Angelo Cangelosi
- The Centre for Robotics and Neural Systems, Plymouth University, Plymouth, United Kingdom
| |
Collapse
|
42
|
Yamada T, Murata S, Arie H, Ogata T. Representation Learning of Logic Words by an RNN: From Word Sequences to Robot Actions. Front Neurorobot 2017; 11:70. [PMID: 29311891 PMCID: PMC5744442 DOI: 10.3389/fnbot.2017.00070] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2017] [Accepted: 12/14/2017] [Indexed: 11/13/2022] Open
Abstract
An important characteristic of human language is compositionality. We can efficiently express a wide variety of real-world situations, events, and behaviors by compositionally constructing the meaning of a complex expression from a finite number of elements. Previous studies have analyzed how machine-learning models, particularly neural networks, can learn from experience to represent compositional relationships between language and robot actions with the aim of understanding the symbol grounding structure and achieving intelligent communicative agents. Such studies have mainly dealt with the words (nouns, adjectives, and verbs) that directly refer to real-world matters. In addition to these words, the current study deals with logic words, such as “not,” “and,” and “or” simultaneously. These words are not directly referring to the real world, but are logical operators that contribute to the construction of meaning in sentences. In human–robot communication, these words may be used often. The current study builds a recurrent neural network model with long short-term memory units and trains it to learn to translate sentences including logic words into robot actions. We investigate what kind of compositional representations, which mediate sentences and robot actions, emerge as the network's internal states via the learning process. Analysis after learning shows that referential words are merged with visual information and the robot's own current state, and the logical words are represented by the model in accordance with their functions as logical operators. Words such as “true,” “false,” and “not” work as non-linear transformations to encode orthogonal phrases into the same area in a memory cell state space. The word “and,” which required a robot to lift up both its hands, worked as if it was a universal quantifier. The word “or,” which required action generation that looked apparently random, was represented as an unstable space of the network's dynamical system.
Collapse
Affiliation(s)
- Tatsuro Yamada
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| | - Shingo Murata
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Hiroaki Arie
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Tetsuya Ogata
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| |
Collapse
|
43
|
Kamper H, Jansen A, Goldwater S. A segmental framework for fully-unsupervised large-vocabulary speech recognition. COMPUT SPEECH LANG 2017. [DOI: 10.1016/j.csl.2017.04.008] [Citation(s) in RCA: 28] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
44
|
Yamada T, Murata S, Arie H, Ogata T. Dynamical Integration of Language and Behavior in a Recurrent Neural Network for Human-Robot Interaction. Front Neurorobot 2016; 10:5. [PMID: 27471463 PMCID: PMC4946379 DOI: 10.3389/fnbot.2016.00005] [Citation(s) in RCA: 22] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2016] [Accepted: 06/23/2016] [Indexed: 12/03/2022] Open
Abstract
To work cooperatively with humans by using language, robots must not only acquire a mapping between language and their behavior but also autonomously utilize the mapping in appropriate contexts of interactive tasks online. To this end, we propose a novel learning method linking language to robot behavior by means of a recurrent neural network. In this method, the network learns from correct examples of the imposed task that are given not as explicitly separated sets of language and behavior but as sequential data constructed from the actual temporal flow of the task. By doing this, the internal dynamics of the network models both language-behavior relationships and the temporal patterns of interaction. Here, "internal dynamics" refers to the time development of the system defined on the fixed-dimensional space of the internal states of the context layer. Thus, in the execution phase, by constantly representing where in the interaction context it is as its current state, the network autonomously switches between recognition and generation phases without any explicit signs and utilizes the acquired mapping in appropriate contexts. To evaluate our method, we conducted an experiment in which a robot generates appropriate behavior responding to a human's linguistic instruction. After learning, the network actually formed the attractor structure representing both language-behavior relationships and the task's temporal pattern in its internal dynamics. In the dynamics, language-behavior mapping was achieved by the branching structure. Repetition of human's instruction and robot's behavioral response was represented as the cyclic structure, and besides, waiting to a subsequent instruction was represented as the fixed-point attractor. Thanks to this structure, the robot was able to interact online with a human concerning the given task by autonomously switching phases.
Collapse
Affiliation(s)
- Tatsuro Yamada
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| | - Shingo Murata
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Hiroaki Arie
- Department of Modern Mechanical Engineering, Waseda University, Tokyo, Japan
| | - Tetsuya Ogata
- Department of Intermedia Art and Science, Waseda University, Tokyo, Japan
| |
Collapse
|
45
|
Uragami D, Kohno Y, Takahashi T. Robotic action acquisition with cognitive biases in coarse-grained state space. Biosystems 2016; 145:41-52. [PMID: 27195484 DOI: 10.1016/j.biosystems.2016.05.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2016] [Revised: 05/14/2016] [Accepted: 05/15/2016] [Indexed: 10/21/2022]
Abstract
Some of the authors have previously proposed a cognitively inspired reinforcement learning architecture (LS-Q) that mimics cognitive biases in humans. LS-Q adaptively learns under uniform, coarse-grained state division and performs well without parameter tuning in a giant-swing robot task. However, these results were shown only in simulations. In this study, we test the validity of the LS-Q implemented in a robot in a real environment. In addition, we analyze the learning process to elucidate the mechanism by which the LS-Q adaptively learns under the partially observable environment. We argue that the LS-Q may be a versatile reinforcement learning architecture, which is, despite its simplicity, easily applicable and does not require well-prepared settings.
Collapse
Affiliation(s)
- Daisuke Uragami
- College of Industrial Technology, Nihon University, 1-2-1, Izumi, Narashino, Chiba, 275-8575, Japan.
| | - Yu Kohno
- Graduate School of Advanced Science and Technology, Tokyo Denki University, Hatoyama, Hiki, Saitama, 350-0394, Japan.
| | - Tatsuji Takahashi
- School of Science and Technology, Tokyo Denki University, Hatoyama, Hiki, Saitama, 350-0394, Japan.
| |
Collapse
|