Infants infer and predict coherent event interactions: Modeling cognitive development

Mental representations of the environment in infants are sparse and grow richer during their development. Anticipatory eye fixation studies show that infants aged around 7 months start to predict the goal of an observed action, e.g., an object targeted by a reaching hand. Interestingly, goal-predictive gaze shifts occur at an earlier age when the hand subsequently manipulates an object and later when an action is performed by an inanimate actor, e.g., a mechanical claw. We introduce CAPRI2 (Cognitive Action PRediction and Inference in Infants), a computational model that explains this development from a functional, algorithmic perspective. It is based on the theory that infants learn object files and events as they develop a physical reasoning system. In particular, CAPRI2 learns a generative event-predictive model, which it uses to both interpret sensory information and infer goal-directed behavior. When observing object interactions, CAPRI2 (i) interprets the unfolding interactions in terms of event-segmented dynamics, (ii) maximizes the coherence of its event interpretations, updating its internal estimates and (iii) chooses gaze behavior to minimize expected uncertainty. As a result, CAPRI2 mimics the developmental pathway of infants' goal-predictive gaze behavior. Our modeling work suggests that the involved event-predictive representations, longer-term generative model learning, and shorter-term retrospective and active inference principles constitute fundamental building blocks for the effective development of goal-predictive capacities.

Evidence for core social goal understanding (and, perhaps, core morality) in preverbal infants

Spelke's What Babies Know masterfully describes infants' impressive repertoire of core cognitive concepts, from which the suite of human knowledge is eventually built. The current commentary argues for the existence of a core concept that Spelke claims preverbal infants lack: social goal. Core social goal concepts, operative extremely early in human development, underlie infants' basic abilities to interpret and evaluate entities within the moral world; such abilities support claims for a core moral domain.

What is in an action? Preschool children predict that agents take previous paths and not previous goals

Developmental science has experienced a vivid debate on whether young children prioritize goals over means in their prediction of others' actions. Influential developmental theories highlight the role of goal objects for action understanding. Yet, recent infant studies report evidence for the opposite. The empirical evidence is therefore inconclusive. The current study advanced this debate by assessing preschool children's verbal predictions of others' actions. In five experiments (N = 302), we investigated whether preschool children and adults predict agents to move towards their previous goal (that is, show goal-related predictions) or predict agents to move along the same movement path that they pursued before. While Experiments 1a, 1b and 1c presented young children and adults with animated agents, Experiments 2a and 2b presented participants with human grasping action. An integrative analysis across experiments revealed that children were more likely to predict the agent to move along the same movement path, Z = -4.574, p ≤ 0.0001 (r = 0.304). That is, preschool children were more likely to predict that agents would move along the same trajectory even though this action would lead to a new goal object. Thus, our findings suggest that young children's action prediction relies on the detection of spatial and movement information. Overall, we discuss our findings in terms of theoretical frameworks that conceive of action understanding as an umbrella term that comprises different forms and facets in which humans understand others' actions. RESEARCH HIGHLIGHTS: We investigated whether preschool children predict agents to move towards their previous goal or to move along the same movement path that they pursued before. Unlike adults, preschool children predicted that agents would move along the same trajectory even though this action would lead to a new goal. Adults' goal-based predictions were affected from contextual details, whereas children systematically made path-based predictions. Young children's action prediction relies on the detection of spatial and movement information.

Do dogs preferentially encode the identity of the target object or the location of others' actions?

The ability to make sense of and predict others' actions is foundational for many socio-cognitive abilities. Dogs (Canis familiaris) constitute interesting comparative models for the study of action perception due to their marked sensitivity to human actions. We tested companion dogs (N = 21) in two screen-based eye-tracking experiments, adopting a task previously used with human infants and apes, to assess which aspects of an agent's action dogs consider relevant to the agent's underlying intentions. An agent was shown repeatedly acting upon the same one of two objects, positioned in the same location. We then presented the objects in swapped locations and the agent approached the objects centrally (Experiment 1) or the old object in the new location or the new object in the old location (Experiment 2). Dogs' anticipatory fixations and looking times did not reflect an expectation that agents should have continued approaching the same object nor the same location as witnessed during the brief familiarization phase; this contrasts with some findings with infants and apes, but aligns with findings in younger infants before they have sufficient motor experience with the observed action. However, dogs' pupil dilation and latency to make an anticipatory fixation suggested that, if anything, dogs expected the agents to keep approaching the same location rather than the same object, and their looking times showed sensitivity to the animacy of the agents. We conclude that dogs, lacking motor experience with the observed actions of grasping or kicking performed by a human or inanimate agent, might interpret such actions as directed toward a specific location rather than a specific object. Future research will need to further probe the suitability of anticipatory looking as measure of dogs' socio-cognitive abilities given differences between the visual systems of dogs and primates.

Reading minds or reading scripts? De-intellectualising theory of mind

Understanding the origins of human social cognition is a central challenge in contemporary science. In recent decades, the idea of a 'Theory of Mind' (ToM) has emerged as the most popular way of explaining unique features of human social cognition. This default view has been progressively undermined by research on 'implicit' ToM, which suggests that relevant precursor abilities may already be present in preverbal human infants and great apes. However, this area of research suffers from conceptual difficulties and empirical limitations, including explanatory circularity, over-intellectualisation, and inconsistent empirical replication. Our article breaks new ground by adapting 'script theory' for application to both linguistic and non-linguistic agents. It thereby provides a new theoretical framework able to resolve the aforementioned issues, generate novel predictions, and provide a plausible account of how individuals make sense of the behaviour of others. Script theory is based on the premise that pre-verbal infants and great apes are capable of basic forms of agency-detection and non-mentalistic goal understanding, allowing individuals to form event-schemata that are then used to make sense of the behaviour of others. We show how script theory circumvents fundamental problems created by ToM-based frameworks, explains patterns of inconsistent replication, and offers important novel predictions regarding how humans and other animals understand and predict the behaviour of others.

Quantifying interaction with robotic toys in pre-term and full-term infants

Infants born pre-term are at an increased risk for developmental, behavioral, and motor delay and subsequent disability. When these problems are detected early, clinical intervention can be effective at improving functional outcomes. Current methods of early clinical assessment are resource intensive, require extensive training, and do not always capture infants' behavior in natural play environments. We developed the Play and Neuro Development Assessment (PANDA) Gym, an affordable, mechatronic, sensor-based play environment that can be used outside clinical settings to capture infant visual and motor behavior. Using a set of classification codes developed from the literature, we analyzed videos from 24 pre-term and full-term infants as they played with each of three robotic toys designed to elicit different types of interactions-a lion, an orangutan, and an elephant. We manually coded for frequency and duration of toy interactions such as kicking, grasping, touching, and gazing. Pre-term infants gazed at the toys with similar frequency as full-term infants, but infants born full-term physically engaged more frequently and for longer durations with the robotic toys than infants born pre-term. While we showed we could detect differences between full-term and pre-term infants, further work is needed to determine whether differences seen were primarily due to age, developmental delays, or a combination.

The use of gaze to study cognition: limitations, solutions, and applications to animal welfare

The study of gaze responses, typically using looking time paradigms, has become a popular approach to improving our understanding of cognitive processes in non-verbal individuals. Our interpretation of data derived from these paradigms, however, is constrained by how we conceptually and methodologically approach these problems. In this perspective paper, we outline the application of gaze studies in comparative cognitive and behavioral research and highlight current limitations in the interpretation of commonly used paradigms. Further, we propose potential solutions, including improvements to current experimental approaches, as well as broad-scale benefits of technology and collaboration. Finally, we outline the potential benefits of studying gaze responses from an animal welfare perspective. We advocate the implementation of these proposals across the field of animal behavior and cognition to aid experimental validity, and further advance our knowledge on a variety of cognitive processes and welfare outcomes.

Infants’ Prediction of Humanoid Robot’s Goal-Directed Action

Several studies have shown that infants anticipate human goal-directed actions, but not robot’s ones. However, the studies focusing on the robot goal-directed actions have mainly analyzed the effect of mechanical arms on infant’s attention. To date, the prediction of goal-directed actions in infants has not yet been studied when the agent is a humanoid robot. Given this lack of evidence in infancy research, the present study aims at analyzing infants’ action anticipation of both a human’s and a humanoid robot’s goal-directed action. Data were acquired on thirty 17-month-old infants, watching four video clips, where either a human or a humanoid robot performed a goal-directed action, i.e. reaching a target. Infants looking behavior was measured through the eye-tracking technique. The results showed that infants anticipated the goal-directed action of both the human and the robot and there were no differences in the anticipatory gaze behavior between the two agents. Furthermore, the findings indicated different attentional patterns for the human and the robot, showing a greater attention paid to the robot's face than the human’s face. Overall, the results suggest that 17-month-old infants may infer also humanoid robot’ underlying action goals.

Perspective Taking and Avatar-Self Merging

Today, avatars often represent users in digital worlds such as in video games or workplace applications. Avatars embody the user and perform their actions in these artificial environments. As a result, users sometimes develop the feeling that their self merges with their avatar. The user realizes that they are the avatar, but the avatar is also the user-meaning that avatar's appearance, character, and actions also affect their self. In the present paper, we first introduce the event-coding approach of the self and then argue based on the reviewed literature on human-avatar interaction that a self-controlled avatar can lead to avatar-self merging: the user sets their own goals in the virtual environment, plans and executes the avatar's actions, and compares the predicted with the actual motion outcomes of the avatar. This makes the user feel body ownership and agency over the avatar's action. Following the event-coding account, avatar-self merging should not be seen as an all-or-nothing process, but rather as a continuous process to which various factors contribute, including successfully taking the perspective of the avatar. Against this background, we discuss affective, cognitive, and visuo-spatial perspective taking of the avatar. As evidence for avatar-self merging, we present findings showing that when users take the avatar's perspective, they can show spontaneous behavioral tendencies that run counter to their own.

Children do not distinguish efficient from inefficient actions during observation

Observation is a powerful way to learn efficient actions from others. However, the role of observers' motor skill in assessing efficiency of others is unknown. Preschoolers are notoriously poor at performing multi-step actions like grasping the handle of a tool. Preschoolers (N = 22) and adults (N = 22) watched video-recorded actors perform efficient and inefficient tool use. Eye tracking showed that preschoolers and adults looked equally long at the videos, but adults looked longer than children at how actors grasped the tool. Deep learning analyses of participants' eye gaze distinguished efficient from inefficient grasps for adults, but not for children. Moreover, only adults showed differential action-related pupil dilation and neural activity (suppressed oscillation power in the mu frequency) while observing efficient vs. inefficient grasps. Thus, children observe multi-step actions without "seeing" whether the initial step is efficient. Findings suggest that observer's own motor efficiency determines whether they can perceive action efficiency in others.

The Impact of Action Effects on Infants' Predictive Gaze Shifts for a Non-Human Grasping Action at 7, 11, and 18 Months

During the observation of goal-directed actions, infants usually predict the goal at an earlier age when the agent is familiar (e.g., human hand) compared to unfamiliar (e.g., mechanical claw). These findings implicate a crucial role of the developing agentive self for infants’ processing of others’ action goals. Recent theoretical accounts suggest that predictive gaze behavior relies on an interplay between infants’ agentive experience (top-down processes) and perceptual information about the agent and the action-event (bottom-up information; e.g., agency cues). The present study examined 7-, 11-, and 18-month-old infants’ predictive gaze behavior for a grasping action performed by an unfamiliar tool, depending on infants’ age-related action knowledge about tool-use and the display of the agency cue of producing a salient action effect. The results are in line with the notion of a systematic interplay between experience-based top-down processes and cue-based bottom-up information: Regardless of the salient action effect, predictive gaze shifts did not occur in the 7-month-olds (least experienced age group), but did occur in the 18-month-olds (most experienced age group). In the 11-month-olds, however, predictive gaze shifts occurred only when a salient action effect was presented. This sheds new light on how the developing agentive self, in interplay with available agency cues, supports infants’ action-goal prediction also for observed tool-use actions.

Emergent Goal-Anticipatory Gaze in Infants via Event-Predictive Learning and Inference

From about 7 months of age onward, infants start to reliably fixate the goal of an observed action, such as a grasp, before the action is complete. The available research has identified a variety of factors that influence such goal-anticipatory gaze shifts, including the experience with the shown action events and familiarity with the observed agents. However, the underlying cognitive processes are still heavily debated. We propose that our minds (i) tend to structure sensorimotor dynamics into probabilistic, generative event-predictive, and event boundary predictive models, and, meanwhile, (ii) choose actions with the objective to minimize predicted uncertainty. We implement this proposition by means of event-predictive learning and active inference. The implemented learning mechanism induces an inductive, event-predictive bias, thus developing schematic encodings of experienced events and event boundaries. The implemented active inference principle chooses actions by aiming at minimizing expected future uncertainty. We train our system on multiple object-manipulation events. As a result, the generation of goal-anticipatory gaze shifts emerges while learning about object manipulations: the model starts fixating the inferred goal already at the start of an observed event after having sampled some experience with possible events and when a familiar agent (i.e., a hand) is involved. Meanwhile, the model keeps reactively tracking an unfamiliar agent (i.e., a mechanical claw) that is performing the same movement. We qualitatively compare these modeling results to behavioral data of infants and conclude that event-predictive learning combined with active inference may be critical for eliciting goal-anticipatory gaze behavior in infants.

Event-Predictive Cognition: A Root for Conceptual Human Thought

Our minds navigate a continuous stream of sensorimotor experiences, selectively compressing them into events. Event-predictive encodings and processing abilities have evolved because they mirror interactions between agents and objects-and the pursuance or avoidance of critical interactions lies at the heart of survival and reproduction. However, it appears that these abilities have evolved not only to pursue live-enhancing events and to avoid threatening events, but also to distinguish food sources, to produce and to use tools, to cooperate, and to communicate. They may have even set the stage for the formation of larger societies and the development of cultural identities. Research on event-predictive cognition investigates how events and conceptualizations thereof are learned, structured, and processed dynamically. It suggests that event-predictive encodings and processes optimally mediate between sensorimotor processes and language. On the one hand, they enable us to perceive and control physical interactions with our world in a highly adaptive, versatile, goal-directed manner. On the other hand, they allow us to coordinate complex social interactions and, in particular, to comprehend and produce language. Event-predictive learning segments sensorimotor experiences into event-predictive encodings. Once first encodings are formed, the mind learns progressively higher order compositional structures, which allow reflecting on the past, reasoning, and planning on multiple levels of abstraction. We conclude that human conceptual thought may be grounded in the principles of event-predictive cognition constituting its root.

The impact of salient action effects on 6-, 7-, and 11-month-olds' goal-predictive gaze shifts for a human grasping action

When infants observe a human grasping action, experience-based accounts predict that all infants familiar with grasping actions should be able to predict the goal regardless of additional agency cues such as an action effect. Cue-based accounts, however, suggest that infants use agency cues to identify and predict action goals when the action or the agent is not familiar. From these accounts, we hypothesized that younger infants would need additional agency cues such as a salient action effect to predict the goal of a human grasping action, whereas older infants should be able to predict the goal regardless of agency cues. In three experiments, we presented 6-, 7-, and 11-month-olds with videos of a manual grasping action presented either with or without an additional salient action effect (Exp. 1 and 2), or we presented 7-month-olds with videos of a mechanical claw performing a grasping action presented with a salient action effect (Exp. 3). The 6-month-olds showed tracking gaze behavior, and the 11-month-olds showed predictive gaze behavior, regardless of the action effect. However, the 7-month-olds showed predictive gaze behavior in the action-effect condition, but tracking gaze behavior in the no-action-effect condition and in the action-effect condition with a mechanical claw. The results therefore support the idea that salient action effects are especially important for infants' goal predictions from 7 months on, and that this facilitating influence of action effects is selective for the observation of human hands.