Task-Oriented Robot Cognitive Manipulation Planning Using Affordance Segmentation and Logic Reasoning

The purpose of task-oriented robot cognitive manipulation planning is to enable robots to select appropriate actions to manipulate appropriate parts of an object according to different tasks, so as to complete the human-like task execution. This ability is crucial for robots to understand how to manipulate and grasp objects under given tasks. This article proposes a task-oriented robot cognitive manipulation planning method using affordance segmentation and logic reasoning, which can provide robots with semantic reasoning skills about the most appropriate parts of the object to be manipulated and oriented by tasks. Object affordance can be obtained by constructing a convolutional neural network based on the attention mechanism. In view of the diversity of service tasks and objects in service environments, object/task ontologies are constructed to realize the management of objects and tasks, and the object-task affordances are established through causal probability logic. On this basis, the Dempster-Shafer theory is used to design a robot cognitive manipulation planning framework, which can reason manipulation regions' configuration for the intended task. The experimental results demonstrate that our proposed method can effectively improve the cognitive manipulation ability of robots and make robots preform various tasks more intelligently.

Curiosity: primate neural circuits for novelty and information seeking

For many years, neuroscientists have investigated the behavioural, computational and neurobiological mechanisms that support value-based decisions, revealing how humans and animals make choices to obtain rewards. However, many decisions are influenced by factors other than the value of physical rewards or second-order reinforcers (such as money). For instance, animals (including humans) frequently explore novel objects that have no intrinsic value solely because they are novel and they exhibit the desire to gain information to reduce their uncertainties about the future, even if this information cannot lead to reward or assist them in accomplishing upcoming tasks. In this Review, I discuss how circuits in the primate brain responsible for detecting, predicting and assessing novelty and uncertainty regulate behaviour and give rise to these behavioural components of curiosity. I also briefly discuss how curiosity-related behaviours arise during postnatal development and point out some important reasons for the persistence of curiosity across generations.

Tool mastering today - an interdisciplinary perspective

Tools have coined human life, living conditions, and culture. Recognizing the cognitive architecture underlying tool use would allow us to comprehend its evolution, development, and physiological basis. However, the cognitive underpinnings of tool mastering remain little understood in spite of long-time research in neuroscientific, psychological, behavioral and technological fields. Moreover, the recent transition of tool use to the digital domain poses new challenges for explaining the underlying processes. In this interdisciplinary review, we propose three building blocks of tool mastering: (A) perceptual and motor abilities integrate to tool manipulation knowledge, (B) perceptual and cognitive abilities to functional tool knowledge, and (C) motor and cognitive abilities to means-end knowledge about tool use. This framework allows for integrating and structuring research findings and theoretical assumptions regarding the functional architecture of tool mastering via behavior in humans and non-human primates, brain networks, as well as computational and robotic models. An interdisciplinary perspective also helps to identify open questions and to inspire innovative research approaches. The framework can be applied to studies on the transition from classical to modern, non-mechanical tools and from analogue to digital user-tool interactions in virtual reality, which come with increased functional opacity and sensorimotor decoupling between tool user, tool, and target. By working towards an integrative theory on the cognitive architecture of the use of tools and technological assistants, this review aims at stimulating future interdisciplinary research avenues.

Robot tool use: A survey

Using human tools can significantly benefit robots in many application domains. Such ability would allow robots to solve problems that they were unable to without tools. However, robot tool use is a challenging task. Tool use was initially considered to be the ability that distinguishes human beings from other animals. We identify three skills required for robot tool use: perception, manipulation, and high-level cognition skills. While both general manipulation tasks and tool use tasks require the same level of perception accuracy, there are unique manipulation and cognition challenges in robot tool use. In this survey, we first define robot tool use. The definition highlighted the skills required for robot tool use. The skills coincide with an affordance model which defined a three-way relation between actions, objects, and effects. We also compile a taxonomy of robot tool use with insights from animal tool use literature. Our definition and taxonomy lay a theoretical foundation for future robot tool use studies and also serve as practical guidelines for robot tool use applications. We first categorize tool use based on the context of the task. The contexts are highly similar for the same task (e.g., cutting) in non-causal tool use, while the contexts for causal tool use are diverse. We further categorize causal tool use based on the task complexity suggested in animal tool use studies into single-manipulation tool use and multiple-manipulation tool use. Single-manipulation tool use are sub-categorized based on tool features and prior experiences of tool use. This type of tool may be considered as building blocks of causal tool use. Multiple-manipulation tool use combines these building blocks in different ways. The different combinations categorize multiple-manipulation tool use. Moreover, we identify different skills required in each sub-type in the taxonomy. We then review previous studies on robot tool use based on the taxonomy and describe how the relations are learned in these studies. We conclude with a discussion of the current applications of robot tool use and open questions to address future robot tool use.

The hiatus between organism and machine evolution: Contrasting mixed microbial communities with robots

Mixed microbial communities, usually composed of various bacterial and fungal species, are fundamental in a plethora of environments, from soil to human gut and skin. Their evolution is a paradigmatic example of intertwined dynamics, where not just the relations among species plays a role, but also the opportunities - and possible harms - that each species presents to the others. These opportunities are in fact affordances, which can be seized by heritable variations and selection. In this paper, starting from a systemic viewpoint of mixed microbial communities, we focus on the pivotal role of affordances in evolution and we contrast it to the artificial evolution of programs and robots. We maintain that the two realms are neatly separated, in that natural evolution proceeds by extending the space of its possibilities in a completely open way, while the latter is inherently limited by the algorithmic framework in which it is defined. This discrepancy characterizes also an envisioned setting in which robots evolve in the physical world. We present arguments supporting our claim and we propose an experimental setting for assessing our statements. Rather than just discussing the limitations of the artificial evolution of machines, the aim of this contribution is to emphasize the tremendous potential of the evolution of the biosphere, beautifully represented by the evolution of communities of microbes.

What is consciousness? Artificial intelligence, real intelligence, quantum mind and qualia

We approach the question ‘What is consciousness?’ in a new way, not as Descartes’ ‘systematic doubt’, but as how organisms find their way in their world. Finding one’s way involves finding possible uses of features of the world that might be beneficial or avoiding those that might be harmful. ‘Possible uses of X to accomplish Y’ are ‘affordances’. The number of uses of X is indefinite (or unknown), the different uses are unordered, are not listable, and are not deducible from one another. All biological adaptations are either affordances seized by heritable variation and selection or, far faster, by the organism acting in its world finding uses of X to accomplish Y. Based on this, we reach rather astonishing conclusions:

1. Artificial general intelligence based on universal Turing machines (UTMs) is not possible, since UTMs cannot ‘find’ novel affordances.

2. Brain-mind is not purely classical physics for no classical physics system can be an analogue computer whose dynamical behaviour can be isomorphic to ‘possible uses’.

3. Brain-mind must be partly quantum—supported by increasing evidence at 6.0 to 7.3 sigma.

4. Based on Heisenberg’s interpretation of the quantum state as ‘potentia’ converted to ‘actuals’ by measurement, where this interpretation is not a substance dualism, a natural hypothesis is that mind actualizes potentia. This is supported at 5.2 sigma. Then mind’s actualizations of entangled brain-mind-world states are experienced as qualia and allow ‘seeing’ or ‘perceiving’ of uses of X to accomplish Y. We can and do jury-rig. Computers cannot.

5. Beyond familiar quantum computers, we discuss the potentialities of trans-Turing systems.

Developing Intelligent Robots that Grasp Affordance

Humans and robots operating in unstructured environments both need to classify objects through haptic exploration and use them in various tasks, but currently they differ greatly in their strategies for acquiring such capabilities. This review explores nascent technologies that promise more convergence. A novel form of artificial intelligence classifies objects according to sensory percepts during active exploration and decides on efficient sequences of exploratory actions to identify objects. Representing objects according to the collective experience of manipulating them provides a substrate for discovering causality and affordances. Such concepts that generalize beyond explicit training experiences are an important aspect of human intelligence that has eluded robots. For robots to acquire such knowledge, they will need an extended period of active exploration and manipulation similar to that employed by infants. The efficacy, efficiency and safety of such behaviors depends on achieving smooth transitions between movements that change quickly from exploratory to executive to reflexive. Animals achieve such smoothness by using a hierarchical control scheme that is fundamentally different from those of conventional robotics. The lowest level of that hierarchy, the spinal cord, starts to self-organize during spontaneous movements in the fetus. This allows its connectivity to reflect the mechanics of the musculoskeletal plant, a bio-inspired process that could be used to adapt spinal-like middleware for robots. Implementation of these extended and essential stages of fetal and infant development is impractical, however, for mechatronic hardware that does not heal and replace itself like biological tissues. Instead such development can now be accomplished in silico and then cloned into physical robots, a strategy that could transcend human performance.

Neural Correlates of Motor Imagery during Action Observation in Affordance-based Actions: Preliminary Results

Object affordance, a characterization of the different functionalities of an object, refers to an object's numerous possibilities of interaction. It has a significant part to play in priming motoric actions which depends on the actor's spontaneous neurological behaviour. Action Observation (AO) and Motor Imagery (MI) also lead to the stimulation of motor system. In fact, AO and MI result in activation of overlapping brain areas as the actual motor task. AO combined with MI (referred to as AO+MI) initiates higher cortical activity in comparison with either MI or AO alone. In this paper, we investigate the influence of affordance driven motor priming during AO, MI and AO + MI. Source current density as an EEG parameter is estimated by Low Resolution Electromagnetic Tomography (LORETA). Our results demonstrate that affordance driven motor priming during AO+MI indicates stronger electrophysiological and behavioural changes. This is evident from the N2 ERP component. Further, the current source density (in brain regions associated with motor planning) during affordance driven AO+MI is found to be maximum.

Building an Affordances Map With Interactive Perception

Robots need to understand their environment to perform their task. If it is possible to pre-program a visual scene analysis process in closed environments, robots operating in an open environment would benefit from the ability to learn it through their interaction with their environment. This ability furthermore opens the way to the acquisition of affordances maps in which the action capabilities of the robot structure its visual scene understanding. We propose an approach to build such affordances maps by relying on an interactive perception approach and an online classification for a real robot equipped with two arms with 7 degrees of freedom. Our system is modular and permits to learn maps from different skills. In the proposed formalization of affordances, actions and effects are related to visual features, not objects, thus our approach does not need a prior definition of the concept of object. We have tested the approach on three action primitives and on a real PR2 robot.

Event-driven proto-object based saliency in 3D space to attract a robot's attention

To interact with its environment, a robot working in 3D space needs to organise its visual input in terms of objects or their perceptual precursors, proto-objects. Among other visual cues, depth is a submodality used to direct attention to visual features and objects. Current depth-based proto-object attention models have been implemented for standard RGB-D cameras that produce synchronous frames. In contrast, event cameras are neuromorphic sensors that loosely mimic the function of the human retina by asynchronously encoding per-pixel brightness changes at very high temporal resolution, thereby providing advantages like high dynamic range, efficiency (thanks to their high degree of signal compression), and low latency. We propose a bio-inspired bottom-up attention model that exploits event-driven sensing to generate depth-based saliency maps that allow a robot to interact with complex visual input. We use event-cameras mounted in the eyes of the iCub humanoid robot to directly extract edge, disparity and motion information. Real-world experiments demonstrate that our system robustly selects salient objects near the robot in the presence of clutter and dynamic scene changes, for the benefit of downstream applications like object segmentation, tracking and robot interaction with external objects.

How Organisms Come to Know the World: Fundamental Limits on Artificial General Intelligence

Artificial intelligence has made tremendous advances since its inception about seventy years ago. Self-driving cars, programs beating experts at complex games, and smart robots capable of assisting people that need care are just some among the successful examples of machine intelligence. This kind of progress might entice us to envision a society populated by autonomous robots capable of performing the same tasks humans do in the near future. This prospect seems limited only by the power and complexity of current computational devices, which is improving fast. However, there are several significant obstacles on this path. General intelligence involves situational reasoning, taking perspectives, choosing goals, and an ability to deal with ambiguous information. We observe that all of these characteristics are connected to the ability of identifying and exploiting new affordances—opportunities (or impediments) on the path of an agent to achieve its goals. A general example of an affordance is the use of an object in the hands of an agent. We show that it is impossible to predefine a list of such uses. Therefore, they cannot be treated algorithmically. This means that “AI agents” and organisms differ in their ability to leverage new affordances. Only organisms can do this. This implies that true AGI is not achievable in the current algorithmic frame of AI research. It also has important consequences for the theory of evolution. We argue that organismic agency is strictly required for truly open-ended evolution through radical emergence. We discuss the diverse ramifications of this argument, not only in AI research and evolution, but also for the philosophy of science.

Rapidly Learning Generalizable and Robot-Agnostic Tool-Use Skills for a Wide Range of Tasks

Many real-world applications require robots to use tools. However, robots lack the skills necessary to learn and perform many essential tool-use tasks. To this end, we present the TRansferrIng Skilled Tool Use Acquired Rapidly (TRI-STAR) framework for task-general robot tool use. TRI-STAR has three primary components: 1) the ability to learn and apply tool-use skills to a wide variety of tasks from a minimal number of training demonstrations, 2) the ability to generalize learned skills to other tools and manipulated objects, and 3) the ability to transfer learned skills to other robots. These capabilities are enabled by TRI-STAR’s task-oriented approach, which identifies and leverages structural task knowledge through the use of our goal-based task taxonomy. We demonstrate this framework with seven tasks that impose distinct requirements on the usages of the tools, six of which were each performed on three physical robots with varying kinematic configurations. Our results demonstrate that TRI-STAR can learn effective tool-use skills from only 20 training demonstrations. In addition, our framework generalizes tool-use skills to morphologically distinct objects and transfers them to new platforms, with minor performance degradation.

The World Is Not a Theorem

The evolution of the biosphere unfolds as a luxuriant generative process of new living forms and functions. Organisms adapt to their environment, exploit novel opportunities that are created in this continuous blooming dynamics. Affordances play a fundamental role in the evolution of the biosphere, for organisms can exploit them for new morphological and behavioral adaptations achieved by heritable variations and selection. This way, the opportunities offered by affordances are then actualized as ever novel adaptations. In this paper, we maintain that affordances elude a formalization that relies on set theory: we argue that it is not possible to apply set theory to affordances; therefore, we cannot devise a set-based mathematical theory to deduce the diachronic evolution of the biosphere.

Tool-Use Model to Reproduce the Goal Situations Considering Relationship Among Tools, Objects, Actions and Effects Using Multimodal Deep Neural Networks

We propose a tool-use model that enables a robot to act toward a provided goal. It is important to consider features of the four factors; tools, objects actions, and effects at the same time because they are related to each other and one factor can influence the others. The tool-use model is constructed with deep neural networks (DNNs) using multimodal sensorimotor data; image, force, and joint angle information. To allow the robot to learn tool-use, we collect training data by controlling the robot to perform various object operations using several tools with multiple actions that leads different effects. Then the tool-use model is thereby trained and learns sensorimotor coordination and acquires relationships among tools, objects, actions and effects in its latent space. We can give the robot a task goal by providing an image showing the target placement and orientation of the object. Using the goal image with the tool-use model, the robot detects the features of tools and objects, and determines how to act to reproduce the target effects automatically. Then the robot generates actions adjusting to the real time situations even though the tools and objects are unknown and more complicated than trained ones.

Safety Concerns Emerging from Robots Navigating in Crowded Pedestrian Areas

The slogan “robots will pervade our environment” has become a reality. Drones and ground robots are used for commercial purposes while semi-autonomous driving systems are standard accessories to traditional cars. However, while our eyes have been riveted on dangers and accidents arising from drones falling and autonomous cars’ crashing, much less attention has been ported to dangers arising from the imminent arrival of robots that share the floor with pedestrians and will mix with human crowds. These robots range from semi or autonomous mobile platforms designed for providing several kinds of service, such as assistant, patrolling, tour-guide, delivery, human transportation, etc. We highlight and discuss potential sources of injury emerging from contacts of robots with pedestrians through a set of case studies. We look specifically at dangers deriving from robots moving in dense crowds. In such situations, contact will not only be unavoidable, but may be desirable to ensure that the robot moves with the flow. As an outlook toward the future, we also offer some thoughts on the psychological risks, beyond the physical hazards, arising from the robot’s appearance and behaviour. We also advocate for new policies to regulate mobile robots traffic and enforce proper end user’s training.

Learning Actions From Natural Language Instructions Using an ON-World Embodied Cognitive Architecture

Endowing robots with the ability to view the world the way humans do, to understand natural language and to learn novel semantic meanings when they are deployed in the physical world, is a compelling problem. Another significant aspect is linking language to action, in particular, utterances involving abstract words, in artificial agents. In this work, we propose a novel methodology, using a brain-inspired architecture, to model an appropriate mapping of language with the percept and internal motor representation in humanoid robots. This research presents the first robotic instantiation of a complex architecture based on the Baddeley's Working Memory (WM) model. Our proposed method grants a scalable knowledge representation of verbal and non-verbal signals in the cognitive architecture, which supports incremental open-ended learning. Human spoken utterances about the workspace and the task are combined with the internal knowledge map of the robot to achieve task accomplishment goals. We train the robot to understand instructions involving higher-order (abstract) linguistic concepts of developmental complexity, which cannot be directly hooked in the physical world and are not pre-defined in the robot's static self-representation. Our proposed interactive learning method grants flexible run-time acquisition of novel linguistic forms and real-world information, without training the cognitive model anew. Hence, the robot can adapt to new workspaces that include novel objects and task outcomes. We assess the potential of the proposed methodology in verification experiments with a humanoid robot. The obtained results suggest robust capabilities of the model to link language bi-directionally with the physical environment and solve a variety of manipulation tasks, starting with limited knowledge and gradually learning from the run-time interaction with the tutor, past the pre-trained stage.

Using environmental nudges to reduce academic cheating in young children

Previous research on nudges conducted with adults suggests that the accessibility of behavioral options can influence people's decisions. The present study examined whether accessibility can be used to reduce academic cheating among young children. We gave children a challenging math test in the presence of an answer key they were instructed not to peek at, and manipulated the accessibility of the answer key by placing various familiar objects on top of it. In Study 1, we used an opaque sheet of paper as a two-dimensional occluder, and found that it significantly reduced cheating compared to a transparent plastic sheet. In Study 2, we used a three-dimensional occluder in the form of a tissue box to make the answer key appear even less accessible, and found it was significantly more effective in reducing cheating than the opaque paper. In Study 3, we used two symbolic representations of the tissue box: a realistic color photo and a line drawing. Both representations were effective in reducing cheating, but the realistic photo was more effective than the drawing. These findings demonstrate that manipulating accessibility can be an effective strategy to nudge children away from cheating in an academic context. They further suggest that different types of everyday objects and their symbolic representations can differentially impact children's moral behavior.

Robot Tutoring of Multiplication: Over One-Third Learning Gain for Most, Learning Loss for Some

In the design of educational robots, it appears to be undecided as to whether robots should show social behaviors and look human-like or whether such cues are insignificant for learning. We conducted an experiment with different designs of social robots built from the same materials, which is unique in robotics research. The robots rehearsed multiplication tables with primary school children in Hong Kong, which is a user group not easily or often accessed. The results show that affective bonding tendencies may occur but did not significantly contribute to the learning progress of these children, which was perhaps due to the short interaction period. Nonetheless, 5 min of robot tutoring improved their scores by about 30%, while performance dropped only for a few challenged children. We discuss topics, such as teaching language skills, which may be fostered by human likeness in appearance and behaviors; however, for Science, Technology, Engineering, and Mathematics (STEM)-related subjects, the social aspects of robots hardly seem to matter.

I Think Therefore You Are

Cyber-physical systems operate in our real world, constantly interacting with the environment and collaborating with other systems. The increasing number of devices will make it infeasible to control each one individually. It will also be infeasible to prepare each of them for every imaginable rapidly unfolding situation. Therefore, we must increase the autonomy of future Cyber-physical Systems. Making these systems self-aware allows them to reason about their own capabilities and their immediate environment. In this article, we extend the idea of the self-awareness of individual systems toward networked self-awareness . This gives systems the ability to reason about how they are being affected by the actions and interactions of others within their perceived environment, as well as in the extended environment that is beyond their direct perception. We propose that different levels of networked self-awareness can develop over time in systems as they do in humans. Furthermore, we propose that this could have the same benefits for networks of systems that it has had for communities of humans, increasing performance and adaptability.

Emergence of Organisms

Since early cybernetics studies by Wiener, Pask, and Ashby, the properties of living systems are subject to deep investigations. The goals of this endeavour are both understanding and building: abstract models and general principles are sought for describing organisms, their dynamics and their ability to produce adaptive behavior. This research has achieved prominent results in fields such as artificial intelligence and artificial life. For example, today we have robots capable of exploring hostile environments with high level of self-sufficiency, planning capabilities and able to learn. Nevertheless, the discrepancy between the emergence and evolution of life and artificial systems is still huge. In this paper, we identify the fundamental elements that characterize the evolution of the biosphere and open-ended evolution, and we illustrate their implications for the evolution of artificial systems. Subsequently, we discuss the most relevant issues and questions that this viewpoint poses both for biological and artificial systems.

Geometric Affordance Perception: Leveraging Deep 3D Saliency With the Interaction Tensor

Agents that need to act on their surroundings can significantly benefit from the perception of their interaction possibilities or affordances. In this paper we combine the benefits of the Interaction Tensor, a straight-forward geometrical representation that captures multiple object-scene interactions, with deep learning saliency for fast parsing of affordances in the environment. Our approach works with visually perceived 3D pointclouds and enables to query a 3D scene for locations that support affordances such as sitting or riding, as well as interactions for everyday objects like the where to hang an umbrella or place a mug. Crucially, the nature of the interaction description exhibits one-shot generalization. Experiments with numerous synthetic and real RGB-D scenes and validated by human subjects, show that the representation enables the prediction of affordance candidate locations in novel environments from a single training example. The approach also allows for a highly parallelizable, multiple-affordance representation, and works at fast rates. The combination of the deep neural network that learns to estimate scene saliency with the one-shot geometric representation aligns well with the expectation that computational models for affordance estimation should be perceptually direct and economical.

Plant Bioinspired Ecological Robotics

Plants are movers, but the nature of their movement differs dramatically from that of creatures that move their whole body from point A to point B. Plants grow to where they are going. Bio-inspired robotics sometimes emulates plants' growth-based movement; but growing is part of a broader system of movement guidance and control. We argue that ecological psychology's conception of "information" and "control" can simultaneously make sense of what it means for a plant to navigate its environment and provide a control scheme for the design of ecological plant-inspired robotics. In this effort, we will outline several control laws and give special consideration to the class of control laws identified by tau theory, such as time to contact.

Automatic Generation of Object Shapes With Desired Affordances Using Voxelgrid Representation

3D objects (artifacts) are made to fulfill functions. Designing an object often starts with defining a list of functionalities or affordances (action possibilities) that it should provide, known as functional requirements. Today, designing 3D object models is still a slow and difficult activity, with few Computer-Aided Design (CAD) tools capable to explore the design solution space. The purpose of this study is to explore shape generation conditioned on desired affordances. We introduce an algorithm for generating voxelgrid object shapes which afford the desired functionalities. We follow the principle form follows function, and assume that object forms are related to affordances they provide (their functions). First, we use an artificial neural network to learn a function-to-form mapping from a dataset of affordance-labeled objects. Then, we combine forms providing one or more desired affordances, generating an object shape expected to provide all of them. Finally, we verify in simulation whether the generated object indeed possesses the desired affordances, by defining and executing affordance tests on it. Examples are provided using the affordances contain-ability, sit-ability, and support-ability.

Integrated Cognitive Architecture for Robot Learning of Action and Language

The manner in which humans learn, plan, and decide actions is a very compelling subject. Moreover, the mechanism behind high-level cognitive functions, such as action planning, language understanding, and logical thinking, has not yet been fully implemented in robotics. In this paper, we propose a framework for the simultaneously comprehension of concepts, actions, and language as a first step toward this goal. This can be achieved by integrating various cognitive modules and leveraging mainly multimodal categorization by using multilayered multimodal latent Dirichlet allocation (mMLDA). The integration of reinforcement learning and mMLDA enables actions based on understanding. Furthermore, the mMLDA, in conjunction with grammar learning and based on the Bayesian hidden Markov model (BHMM), allows the robot to verbalize its own actions and understand user utterances. We verify the potential of the proposed architecture through experiments using a real robot.

Compositional Learning of Human Activities With a Self-Organizing Neural Architecture

An important step for assistive systems and robot companions operating in human environments is to learn the compositionality of human activities, i.e., recognize both activities and their comprising actions. Most existing approaches address action and activity recognition as separate tasks, i.e., actions need to be inferred before the activity labels, and are thus highly sensitive to the correct temporal segmentation of the activity sequences. In this paper, we present a novel learning approach that jointly learns human activities on two levels of semantic and temporal complexity: (1) transitive actions such as reaching and opening, e.g., a cereal box, and (2) high-level activities such as having breakfast. Our model consists of a hierarchy of GWR networks which process and learn inherent spatiotemporal dependencies of multiple visual cues extracted from the human body skeletal representation and the interaction with objects. The neural architecture learns and semantically segments input RGB-D sequences of high-level activities into their composing actions, without supervision. We investigate the performance of our architecture with a set of experiments on a publicly available benchmark dataset. The experimental results show that our approach outperforms the state of the art with respect to the classification of the high-level activities. Additionally, we introduce a novel top-down modulation mechanism to the architecture which uses the actions and activity labels as constraints during the learning phase. In our experiments, we show how this mechanism can be used to control the network's neural growth without decreasing the overall performance.

Action Generation Adapted to Low-Level and High-Level Robot-Object Interaction States

Our daily environments are complex, composed of objects with different features. These features can be categorized into low-level features, e.g., an object position or temperature, and high-level features resulting from a pre-processing of low-level features for decision purposes, e.g., a binary value saying if it is too hot to be grasped. Besides, our environments are dynamic, i.e., object states can change at any moment. Therefore, robots performing tasks in these environments must have the capacity to (i) identify the next action to execute based on the available low-level and high-level object states, and (ii) dynamically adapt their actions to state changes. We introduce a method named Interaction State-based Skill Learning (IS²L), which builds skills to solve tasks in realistic environments. A skill is a Bayesian Network that infers actions composed of a sequence of movements of the robot's end-effector, which locally adapt to spatio-temporal perturbations using a dynamical system. In the current paper, an external agent performs one or more kinesthetic demonstrations of an action generating a dataset of high-level and low-level states of the robot and the environment objects. First, the method transforms each interaction to represent (i) the relationship between the robot and the object and (ii) the next robot end-effector movement to perform at consecutive instants of time. Then, the skill is built, i.e., the Bayesian network is learned. While generating an action this skill relies on the robot and object states to infer the next movement to execute. This movement selection gets inspired by a type of predictive models for action selection usually called affordances. The main contribution of this paper is combining the main features of dynamical systems and affordances in a unique method to build skills that solve tasks in realistic scenarios. More precisely, combining the low-level movement generation of the dynamical systems, to adapt to local perturbations, with the next movement selection simultaneously based on high-level and low-level states. This contribution was assessed in three experiments in realistic environments using both high-level and low-level states. The built skills solved the respective tasks relying on both types of states, and adapting to external perturbations.

Allocentric Emotional Affordances in HRI: The Multimodal Binding

The concept of affordance perception is one of the distinctive traits of human cognition; and its application to robots can dramatically improve the quality of human-robot interaction (HRI). In this paper we explore and discuss the idea of “emotional affordances” by proposing a viable model for implementation into HRI; which considers allocentric and multimodal perception. We consider “2-ways” affordances: perceived object triggering an emotion; and perceived human emotion expression triggering an action. In order to make the implementation generic; the proposed model includes a library that can be customised depending on the specific robot and application scenario. We present the AAA (Affordance-Appraisal-Arousal) model; which incorporates Plutchik’s Wheel of Emotions; and we outline some numerical examples of how it can be used in different scenarios.