1
|
Rayyes R. Intrinsic motivation learning for real robot applications. Front Robot AI 2023; 10:1102438. [PMID: 36845331 PMCID: PMC9950409 DOI: 10.3389/frobt.2023.1102438] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 01/13/2023] [Indexed: 02/12/2023] Open
|
2
|
Aubret A, Matignon L, Hassas S. An Information-Theoretic Perspective on Intrinsic Motivation in Reinforcement Learning: A Survey. ENTROPY (BASEL, SWITZERLAND) 2023; 25:327. [PMID: 36832693 PMCID: PMC9954873 DOI: 10.3390/e25020327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/28/2022] [Revised: 01/23/2023] [Accepted: 02/07/2023] [Indexed: 06/18/2023]
Abstract
The reinforcement learning (RL) research area is very active, with an important number of new contributions, especially considering the emergent field of deep RL (DRL). However, a number of scientific and technical challenges still need to be resolved, among which we acknowledge the ability to abstract actions or the difficulty to explore the environment in sparse-reward settings which can be addressed by intrinsic motivation (IM). We propose to survey these research works through a new taxonomy based on information theory: we computationally revisit the notions of surprise, novelty, and skill-learning. This allows us to identify advantages and disadvantages of methods and exhibit current outlooks of research. Our analysis suggests that novelty and surprise can assist the building of a hierarchy of transferable skills which abstracts dynamics and makes the exploration process more robust.
Collapse
|
3
|
Ten A, Kaushik P, Oudeyer PY, Gottlieb J. Humans monitor learning progress in curiosity-driven exploration. Nat Commun 2021; 12:5972. [PMID: 34645800 PMCID: PMC8514490 DOI: 10.1038/s41467-021-26196-w] [Citation(s) in RCA: 20] [Impact Index Per Article: 6.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2020] [Accepted: 09/02/2021] [Indexed: 11/09/2022] Open
Abstract
Curiosity-driven learning is foundational to human cognition. By enabling humans to autonomously decide when and what to learn, curiosity has been argued to be crucial for self-organizing temporally extended learning curricula. However, the mechanisms driving people to set intrinsic goals, when they are free to explore multiple learning activities, are still poorly understood. Computational theories propose different heuristics, including competence measures (e.g., percent correct) and learning progress, that could be used as intrinsic utility functions to efficiently organize exploration. Such intrinsic utilities constitute computationally cheap but smart heuristics to prevent people from laboring in vain on unlearnable activities, while still motivating them to self-challenge on difficult learnable activities. Here, we provide empirical evidence for these ideas by means of a free-choice experimental paradigm and computational modeling. We show that while humans rely on competence information to avoid easy tasks, models that include a learning-progress component provide the best fit to task selection data. These results bridge the research in artificial and biological curiosity, reveal strategies that are used by humans but have not been considered in computational research, and introduce tools for probing how humans become intrinsically motivated to learn and acquire interests and skills on extended time scales.
Collapse
Affiliation(s)
- Alexandr Ten
- INRIA Bordeaux Sud-Ouest, 200 Avenue de la Vieille Tour, 33405, Talence, France.
| | - Pramod Kaushik
- INRIA Bordeaux Sud-Ouest, 200 Avenue de la Vieille Tour, 33405, Talence, France
| | - Pierre-Yves Oudeyer
- INRIA Bordeaux Sud-Ouest, 200 Avenue de la Vieille Tour, 33405, Talence, France
| | - Jacqueline Gottlieb
- Department of Neuroscience & The Kavli Institute for Brain Science, Columbia University, 1051 Riverside Drive, Kolb Research Annex, Rm. 569, New York, NY, 10032, USA
| |
Collapse
|
4
|
Bugur S, Oztop E, Nagai Y, Ugur E. Effect Regulated Projection of Robot’s Action Space for Production and Prediction of Manipulation Primitives Through Learning Progress and Predictability-Based Exploration. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2019.2933900] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
5
|
Dhakan P, Kasmarik K, Rano I, Siddique N. Open-Ended Continuous Learning of Compound Goals. IEEE Trans Cogn Dev Syst 2021. [DOI: 10.1109/tcds.2019.2947896] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
6
|
Jacquey L, Baldassarre G, Santucci VG, O’Regan JK. Sensorimotor Contingencies as a Key Drive of Development: From Babies to Robots. Front Neurorobot 2019; 13:98. [PMID: 31866848 PMCID: PMC6904889 DOI: 10.3389/fnbot.2019.00098] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2019] [Accepted: 11/06/2019] [Indexed: 01/22/2023] Open
Abstract
Much current work in robotics focuses on the development of robots capable of autonomous unsupervised learning. An essential prerequisite for such learning to be possible is that the agent should be sensitive to the link between its actions and the consequences of its actions, called sensorimotor contingencies. This sensitivity, and more particularly its role as a key drive of development, has been widely studied by developmental psychologists. However, the results of these studies may not necessarily be accessible or intelligible to roboticians. In this paper, we review the main experimental data demonstrating the role of sensitivity to sensorimotor contingencies in infants' acquisition of four fundamental motor and cognitive abilities: body knowledge, memory, generalization, and goal-directedness. We relate this data from developmental psychology to work in robotics, highlighting the links between these two domains of research. In the last part of the article we present a blueprint architecture demonstrating how exploitation of sensitivity to sensorimotor contingencies, combined with the notion of "goal," allows an agent to develop new sensorimotor skills. This architecture can be used to guide the design of specific computational models, and also to possibly envisage new empirical experiments.
Collapse
Affiliation(s)
- Lisa Jacquey
- Integrative Neuroscience and Cognition Center, UMR 8002, CNRS, Université Paris Descartes, Paris, France
- Laboratoire Ethologie Cognition Développement, Université Paris Nanterre, Nanterre, France
| | - Gianluca Baldassarre
- Laboratory of Computational Embodied Neuroscience, Istituto di Scienze e Tecnologie della Cognizione, Consiglio Nazionale delle Ricerche, Rome, Italy
| | - Vieri Giuliano Santucci
- Laboratory of Computational Embodied Neuroscience, Istituto di Scienze e Tecnologie della Cognizione, Consiglio Nazionale delle Ricerche, Rome, Italy
| | - J. Kevin O’Regan
- Integrative Neuroscience and Cognition Center, UMR 8002, CNRS, Université Paris Descartes, Paris, France
| |
Collapse
|
7
|
Baldassarre G, Lord W, Granato G, Santucci VG. An Embodied Agent Learning Affordances With Intrinsic Motivations and Solving Extrinsic Tasks With Attention and One-Step Planning. Front Neurorobot 2019; 13:45. [PMID: 31402859 PMCID: PMC6676802 DOI: 10.3389/fnbot.2019.00045] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2018] [Accepted: 06/11/2019] [Indexed: 11/18/2022] Open
Abstract
We propose an architecture for the open-ended learning and control of embodied agents. The architecture learns action affordances and forward models based on intrinsic motivations and can later use the acquired knowledge to solve extrinsic tasks by decomposing them into sub-tasks, each solved with one-step planning. An affordance is here operationalized as the agent's estimate of the probability of success of an action performed on a given object. The focus of the work is on the overall architecture while single sensorimotor components are simplified. A key element of the architecture is the use of “active vision” that plays two functions, namely to focus on single objects and to factorize visual information into the object appearance and object position. These processes serve both the acquisition and use of object-related affordances, and the decomposition of extrinsic goals (tasks) into multiple sub-goals (sub-tasks). The architecture gives novel contributions on three problems: (a) the learning of affordances based on intrinsic motivations; (b) the use of active vision to decompose complex extrinsic tasks; (c) the possible role of affordances within planning systems endowed with models of the world. The architecture is tested in a simulated stylized 2D scenario in which objects need to be moved or “manipulated” in order to accomplish new desired overall configurations of the objects (extrinsic goals). The results show the utility of using intrinsic motivations to support affordance learning; the utility of active vision to solve composite tasks; and the possible utility of affordances for solving utility-based planning problems.
Collapse
Affiliation(s)
- Gianluca Baldassarre
- Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, National Research Council of Italy, Rome, Italy
| | - William Lord
- School of Engineering Sciences, KTH Royal Institute of Technology, Stockholm, Sweden
| | - Giovanni Granato
- Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, National Research Council of Italy, Rome, Italy
| | - Vieri Giuliano Santucci
- Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, National Research Council of Italy, Rome, Italy
| |
Collapse
|
8
|
A Reinforcement Learning Architecture That Transfers Knowledge Between Skills When Solving Multiple Tasks. IEEE Trans Cogn Dev Syst 2019. [DOI: 10.1109/tcds.2016.2607018] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
9
|
Khan MM, Kasmarik K, Barlow M. Toward Computational Motivation for Multi-Agent Systems and Swarms. Front Robot AI 2018; 5:134. [PMID: 33501012 PMCID: PMC7806096 DOI: 10.3389/frobt.2018.00134] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2018] [Accepted: 12/03/2018] [Indexed: 11/30/2022] Open
Abstract
Motivation is a crucial part of animal and human mental development, fostering competence, autonomy, and open-ended development. Motivational constructs have proved to be an integral part of explaining human and animal behavior. Computer scientists have proposed various computational models of motivation for artificial agents, with the aim of building artificial agents capable of autonomous goal generation. Multi-agent systems and swarm intelligence are natural extensions to the individual agent setting. However, there are only a few works that focus on motivation theories in multi-agent or swarm settings. In this study, we review current computational models of motivation settings, mechanisms, functions and evaluation methods and discuss how we can produce systems with new kinds of functions not possible using individual agents. We describe in detail this open area of research and the major research challenges it holds.
Collapse
Affiliation(s)
- Md Mohiuddin Khan
- School of Engineering and Information Technology, University of New South Wales, Canberra, ACT, Australia
| | - Kathryn Kasmarik
- School of Engineering and Information Technology, University of New South Wales, Canberra, ACT, Australia
| | - Michael Barlow
- School of Engineering and Information Technology, University of New South Wales, Canberra, ACT, Australia
| |
Collapse
|
10
|
Biehl M, Guckelsberger C, Salge C, Smith SC, Polani D. Expanding the Active Inference Landscape: More Intrinsic Motivations in the Perception-Action Loop. Front Neurorobot 2018; 12:45. [PMID: 30214404 PMCID: PMC6125413 DOI: 10.3389/fnbot.2018.00045] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2018] [Accepted: 07/02/2018] [Indexed: 11/13/2022] Open
Abstract
Active inference is an ambitious theory that treats perception, inference, and action selection of autonomous agents under the heading of a single principle. It suggests biologically plausible explanations for many cognitive phenomena, including consciousness. In active inference, action selection is driven by an objective function that evaluates possible future actions with respect to current, inferred beliefs about the world. Active inference at its core is independent from extrinsic rewards, resulting in a high level of robustness across e.g., different environments or agent morphologies. In the literature, paradigms that share this independence have been summarized under the notion of intrinsic motivations. In general and in contrast to active inference, these models of motivation come without a commitment to particular inference and action selection mechanisms. In this article, we study if the inference and action selection machinery of active inference can also be used by alternatives to the originally included intrinsic motivation. The perception-action loop explicitly relates inference and action selection to the environment and agent memory, and is consequently used as foundation for our analysis. We reconstruct the active inference approach, locate the original formulation within, and show how alternative intrinsic motivations can be used while keeping many of the original features intact. Furthermore, we illustrate the connection to universal reinforcement learning by means of our formalism. Active inference research may profit from comparisons of the dynamics induced by alternative intrinsic motivations. Research on intrinsic motivations may profit from an additional way to implement intrinsically motivated agents that also share the biological plausibility of active inference.
Collapse
Affiliation(s)
| | - Christian Guckelsberger
- Computational Creativity Group, Department of Computing, Goldsmiths, University of London, London, United Kingdom
| | - Christoph Salge
- Game Innovation Lab, Department of Computer Science and Engineering, New York University, New York, NY, United States
- Sepia Lab, Adaptive Systems Research Group, Department of Computer Science, University of Hertfordshire, Hatfield, United Kingdom
| | - Simón C. Smith
- Sepia Lab, Adaptive Systems Research Group, Department of Computer Science, University of Hertfordshire, Hatfield, United Kingdom
- Institute of Perception, Action and Behaviour, School of Informatics, The University of Edinburgh, Edinburgh, United Kingdom
| | - Daniel Polani
- Sepia Lab, Adaptive Systems Research Group, Department of Computer Science, University of Hertfordshire, Hatfield, United Kingdom
| |
Collapse
|
11
|
Mannella F, Santucci VG, Somogyi E, Jacquey L, O'Regan KJ, Baldassarre G. Know Your Body Through Intrinsic Goals. Front Neurorobot 2018; 12:30. [PMID: 30018547 PMCID: PMC6037791 DOI: 10.3389/fnbot.2018.00030] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2017] [Accepted: 05/17/2018] [Indexed: 11/25/2022] Open
Abstract
The first “object” that newborn children play with is their own body. This activity allows them to autonomously form a sensorimotor map of their own body and a repertoire of actions supporting future cognitive and motor development. Here we propose the theoretical hypothesis, operationalized as a computational model, that this acquisition of body knowledge is not guided by random motor-babbling, but rather by autonomously generated goals formed on the basis of intrinsic motivations. Motor exploration leads the agent to discover and form representations of the possible sensory events it can cause with its own actions. When the agent realizes the possibility of improving the competence to re-activate those representations, it is intrinsically motivated to select and pursue them as goals. The model is based on four components: (1) a self-organizing neural network, modulated by competence-based intrinsic motivations, that acquires abstract representations of experienced sensory (touch) changes; (2) a selector that selects the goal to pursue, and the motor resources to train to pursue it, on the basis of competence improvement; (3) an echo-state neural network that controls and learns, through goal-accomplishment and competence, the agent's motor skills; (4) a predictor of the accomplishment of the selected goals generating the competence-based intrinsic motivation signals. The model is tested as the controller of a simulated simple planar robot composed of a torso and two kinematic 3-DoF 2D arms. The robot explores its body covered by touch sensors by moving its arms. The results, which might be used to guide future empirical experiments, show how the system converges to goals and motor skills allowing it to touch the different parts of own body and how the morphology of the body affects the formed goals. The convergence is strongly dependent on competence-based intrinsic motivations affecting not only skill learning and the selection of formed goals, but also the formation of the goal representations themselves.
Collapse
Affiliation(s)
- Francesco Mannella
- Institute of Cognitive Sciences and Technologies, National Research Council - CNR, Rome, Italy
| | - Vieri G Santucci
- Institute of Cognitive Sciences and Technologies, National Research Council - CNR, Rome, Italy
| | - Eszter Somogyi
- Laboratoire Psychologie de la Perception (UMR 8242), Paris Descartes - CPSC, Paris, France
| | - Lisa Jacquey
- Laboratoire Psychologie de la Perception (UMR 8242), Paris Descartes - CPSC, Paris, France
| | - Kevin J O'Regan
- Laboratoire Psychologie de la Perception (UMR 8242), Paris Descartes - CPSC, Paris, France
| | - Gianluca Baldassarre
- Institute of Cognitive Sciences and Technologies, National Research Council - CNR, Rome, Italy
| |
Collapse
|
12
|
Sperati V, Baldassarre G. Bio-Inspired Model Learning Visual Goals and Attention Skills Through Contingencies and Intrinsic Motivations. IEEE Trans Cogn Dev Syst 2018. [DOI: 10.1109/tcds.2017.2772908] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
13
|
Wu R, Zhou C, Chao F, Zhu Z, Lin CM, Yang L. A Developmental Learning Approach of Mobile Manipulator via Playing. Front Neurorobot 2017; 11:53. [PMID: 29046632 PMCID: PMC5632655 DOI: 10.3389/fnbot.2017.00053] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2017] [Accepted: 09/19/2017] [Indexed: 11/13/2022] Open
Abstract
Inspired by infant development theories, a robotic developmental model combined with game elements is proposed in this paper. This model does not require the definition of specific developmental goals for the robot, but the developmental goals are implied in the goals of a series of game tasks. The games are characterized into a sequence of game modes based on the complexity of the game tasks from simple to complex, and the task complexity is determined by the applications of developmental constraints. Given a current mode, the robot switches to play in a more complicated game mode when it cannot find any new salient stimuli in the current mode. By doing so, the robot gradually achieves it developmental goals by playing different modes of games. In the experiment, the game was instantiated into a mobile robot with the playing task of picking up toys, and the game is designed with a simple game mode and a complex game mode. A developmental algorithm, "Lift-Constraint, Act and Saturate," is employed to drive the mobile robot move from the simple mode to the complex one. The experimental results show that the mobile manipulator is able to successfully learn the mobile grasping ability after playing simple and complex games, which is promising in developing robotic abilities to solve complex tasks using games.
Collapse
Affiliation(s)
- Ruiqi Wu
- Fujian Provincal Key Lab of Brain-Inspired Computing, Department of Cognitive Science, School of Informatics, Xiamen University, Xiamen, China
| | - Changle Zhou
- Fujian Provincal Key Lab of Brain-Inspired Computing, Department of Cognitive Science, School of Informatics, Xiamen University, Xiamen, China
| | - Fei Chao
- Fujian Provincal Key Lab of Brain-Inspired Computing, Department of Cognitive Science, School of Informatics, Xiamen University, Xiamen, China
| | - Zuyuan Zhu
- Department of Computer Science, School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom
| | - Chih-Min Lin
- Fujian Provincal Key Lab of Brain-Inspired Computing, Department of Cognitive Science, School of Informatics, Xiamen University, Xiamen, China.,Department of Electrical Engineering, Yuan Ze University, Tao-Yuan, Taiwan
| | - Longzhi Yang
- Department of Computer and Information Sciences, Faculty of Engineering and Environment, Northumbria University, Newcastle upon Tyne, United Kingdom
| |
Collapse
|
14
|
Meola VC, Caligiore D, Sperati V, Zollo L, Ciancio AL, Taffoni F, Guglielmelli E, Baldassarre G. Interplay of Rhythmic and Discrete Manipulation Movements During Development: A Policy-Search Reinforcement-Learning Robot Model. IEEE Trans Cogn Dev Syst 2016. [DOI: 10.1109/tamd.2015.2494460] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
15
|
Santucci VG, Baldassarre G, Mirolli M. GRAIL: A Goal-Discovering Robotic Architecture for Intrinsically-Motivated Learning. IEEE Trans Cogn Dev Syst 2016. [DOI: 10.1109/tcds.2016.2538961] [Citation(s) in RCA: 40] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
16
|
|
17
|
Clark D, Schumann F, Mostofsky SH. Mindful movement and skilled attention. Front Hum Neurosci 2015; 9:297. [PMID: 26190986 PMCID: PMC4484342 DOI: 10.3389/fnhum.2015.00297] [Citation(s) in RCA: 32] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/28/2014] [Accepted: 05/09/2015] [Indexed: 12/04/2022] Open
Abstract
Bodily movement has long been employed as a foundation for cultivating mental skills such as attention, self-control or mindfulness, with recent studies documenting the positive impacts of mindful movement training, such as yoga and tai chi. A parallel “mind-body connection” has also been observed in many developmental disorders. We elaborate a spectrum of mindfulness by considering ADHD, in which deficient motor control correlates with impaired (disinhibited) behavioral control contributing to defining features of excessive distractibility and impulsivity. These data provide evidence for an important axis of variation for wellbeing, in which skillful cognitive control covaries with a capacity for skillful movement. We review empirical and theoretical literature on attention, cognitive control, mind wandering, mindfulness and skill learning, endorsing a model of skilled attention in which motor plans, attention, and executive goals are seen as mutually co-defining aspects of skilled behavior that are linked by reciprocal inhibitory and excitatory connections. Thus, any movement training should engage “higher-order” inhibition and selection and develop a repertoire of rehearsed procedures that coordinate goals, attention and motor plans. However, we propose that mindful movement practice may improve the functional quality of rehearsed procedures, cultivating a transferrable skill of attention. We adopt Langer’s spectrum of mindful learning that spans from “mindlessness” to engagement with the details of the present task and contrast this with the mental attitudes cultivated in standard mindfulness meditation. We particularly follow Feldenkrais’ suggestion that mindful learning of skills for organizing the body in movement might transfer to other forms of mental activity. The results of mindful movement training should be observed in multiple complementary measures, and may have tremendous potential benefit for individuals with ADHD and other populations.
Collapse
Affiliation(s)
- Dav Clark
- D-Lab, University of California, Berkeley Berkeley, CA, USA ; Berkeley Institute for Data Science, University of California, Berkeley Berkeley, CA, USA
| | - Frank Schumann
- Laboratoire Psychologie de la Perception, Université Paris Descartes Paris, France
| | - Stewart H Mostofsky
- Center for Neurodevelopmental Medicine and Research, Kennedy Krieger Institute Baltimore, MD, USA ; Departments of Neurology and Psychiatry and Behavioral Sciences, Johns Hopkins University School of Medicine Baltimore, MD, USA
| |
Collapse
|
18
|
Caligiore D, Mustile M, Cipriani D, Redgrave P, Triesch J, De Marsico M, Baldassarre G. Intrinsic motivations drive learning of eye movements: an experiment with human adults. PLoS One 2015; 10:e0118705. [PMID: 25775248 PMCID: PMC4361314 DOI: 10.1371/journal.pone.0118705] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/08/2014] [Accepted: 01/07/2015] [Indexed: 11/19/2022] Open
Abstract
Intrinsic motivations drive the acquisition of knowledge and skills on the basis of novel or surprising stimuli or the pleasure to learn new skills. In so doing, they are different from extrinsic motivations that are mainly linked to drives that promote survival and reproduction. Intrinsic motivations have been implicitly exploited in several psychological experiments but, due to the lack of proper paradigms, they are rarely a direct subject of investigation. This article investigates how different intrinsic motivation mechanisms can support the learning of visual skills, such as "foveate a particular object in space", using a gaze contingency paradigm. In the experiment participants could freely foveate objects shown in a computer screen. Foveating each of two "button" pictures caused different effects: one caused the appearance of a simple image (blue rectangle) in unexpected positions, while the other evoked the appearance of an always-novel picture (objects or animals). The experiment studied how two possible intrinsic motivation mechanisms might guide learning to foveate one or the other button picture. One mechanism is based on the sudden, surprising appearance of a familiar image at unpredicted locations, and a second one is based on the content novelty of the images. The results show the comparative effectiveness of the mechanism based on image novelty, whereas they do not support the operation of the mechanism based on the surprising location of the image appearance. Interestingly, these results were also obtained with participants that, according to a post experiment questionnaire, had not understood the functions of the different buttons suggesting that novelty-based intrinsic motivation mechanisms might operate even at an unconscious level.
Collapse
Affiliation(s)
- Daniele Caligiore
- Laboratory of Computational Embodied Neuroscience, Istituto di Scienze e Tecnologie della Cognizione, Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR), Rome, Italy
| | - Magda Mustile
- Laboratory of Computational Embodied Neuroscience, Istituto di Scienze e Tecnologie della Cognizione, Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR), Rome, Italy
| | - Daniele Cipriani
- Dipartimento di Informatica, Università di Roma “Sapienza”, Rome, Italy
| | - Peter Redgrave
- Department of Psychology, University of Sheffield, Sheffield, United Kingdom
| | - Jochen Triesch
- Frankfurt Institute for Advanced Studies, Frankfurt, Germany
| | - Maria De Marsico
- Dipartimento di Informatica, Università di Roma “Sapienza”, Rome, Italy
| | - Gianluca Baldassarre
- Laboratory of Computational Embodied Neuroscience, Istituto di Scienze e Tecnologie della Cognizione, Consiglio Nazionale delle Ricerche (LOCEN-ISTC-CNR), Rome, Italy
| |
Collapse
|
19
|
Baldassarre G, Stafford T, Mirolli M, Redgrave P, Ryan RM, Barto A. Intrinsic motivations and open-ended development in animals, humans, and robots: an overview. Front Psychol 2014; 5:985. [PMID: 25249998 PMCID: PMC4158798 DOI: 10.3389/fpsyg.2014.00985] [Citation(s) in RCA: 38] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2014] [Accepted: 08/19/2014] [Indexed: 11/13/2022] Open
Affiliation(s)
- Gianluca Baldassarre
- Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, National Research Council Rome, Italy
| | - Tom Stafford
- Department of Psychology, University of Sheffield Sheffield, UK
| | - Marco Mirolli
- Laboratory of Computational Embodied Neuroscience, Institute of Cognitive Sciences and Technologies, National Research Council Rome, Italy
| | - Peter Redgrave
- Department of Psychology, University of Sheffield Sheffield, UK
| | - Richard M Ryan
- Department of Clinical and Social Sciences in Psychology, University of Rochester River, New York, USA
| | - Andrew Barto
- Department of Computer Science, University of Massachusetts Amherst Massachusetts, USA
| |
Collapse
|
20
|
Cyr A, Boukadoum M, Thériault F. Operant conditioning: a minimal components requirement in artificial spiking neurons designed for bio-inspired robot's controller. Front Neurorobot 2014; 8:21. [PMID: 25120464 PMCID: PMC4110879 DOI: 10.3389/fnbot.2014.00021] [Citation(s) in RCA: 16] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2013] [Accepted: 07/02/2014] [Indexed: 01/03/2023] Open
Abstract
In this paper, we investigate the operant conditioning (OC) learning process within a bio-inspired paradigm, using artificial spiking neural networks (ASNN) to act as robot brain controllers. In biological agents, OC results in behavioral changes learned from the consequences of previous actions, based on progressive prediction adjustment from rewarding or punishing signals. In a neurorobotics context, virtual and physical autonomous robots may benefit from a similar learning skill when facing unknown and unsupervised environments. In this work, we demonstrate that a simple invariant micro-circuit can sustain OC in multiple learning scenarios. The motivation for this new OC implementation model stems from the relatively complex alternatives that have been described in the computational literature and recent advances in neurobiology. Our elementary kernel includes only a few crucial neurons, synaptic links and originally from the integration of habituation and spike-timing dependent plasticity as learning rules. Using several tasks of incremental complexity, our results show that a minimal neural component set is sufficient to realize many OC procedures. Hence, with the proposed OC module, designing learning tasks with an ASNN and a bio-inspired robot context leads to simpler neural architectures for achieving complex behaviors.
Collapse
Affiliation(s)
- André Cyr
- Computer Science Department, Cognitive and Computer science, Université du Québec à Montréal Montréal, QC, Canada
| | - Mounir Boukadoum
- Computer Science Department, Cognitive and Computer science, Université du Québec à Montréal Montréal, QC, Canada
| | - Frédéric Thériault
- Computer Science Department, Cognitive and Computer science, Université du Québec à Montréal Montréal, QC, Canada
| |
Collapse
|
21
|
Taffoni F, Tamilia E, Focaroli V, Formica D, Ricci L, Di Pino G, Baldassarre G, Mirolli M, Guglielmelli E, Keller F. Development of goal-directed action selection guided by intrinsic motivations: an experiment with children. Exp Brain Res 2014; 232:2167-77. [DOI: 10.1007/s00221-014-3907-z] [Citation(s) in RCA: 17] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2013] [Accepted: 03/05/2014] [Indexed: 11/24/2022]
|