Infants use social context to bind actions into a collaborative sequence

[Machine learning algorithms for identifying autism spectrum disorder through eye-tracking in different intention videos]

OBJECTIVES

To investigate the differences in visual perception between children with autism spectrum disorder (ASD) and typically developing (TD) children when watching different intention videos, and to explore the feasibility of machine learning algorithms in objectively distinguishing between ASD children and TD children.

METHODS

A total of 58 children with ASD and 50 TD children were enrolled and were asked to watch the videos containing joint intention and non-joint intention, and the gaze duration and frequency in different areas of interest were used as original indicators to construct classifier-based models. The models were evaluated in terms of the indicators such as accuracy, sensitivity, and specificity.

RESULTS

When using eight common classifiers, including support vector machine, linear discriminant analysis, decision tree, random forest, and K-nearest neighbors (with K values of 1, 3, 5, and 7), based on the original feature indicators, the highest classification accuracy achieved was 81.90%. A feature reconstruction approach with a decision tree classifier was used to further improve the accuracy of classification, and then the model showed the accuracy of 91.43%, the specificity of 89.80%, and the sensitivity of 92.86%, with an area under the receiver operating characteristic curve of 0.909 (P<0.001).

CONCLUSIONS

The machine learning model based on eye-tracking data can accurately distinguish ASD children from TD children, which provides a scientific basis for developing rapid and objective ASD screening tools.

Seeing social interactions

Seeing the interactions between other people is a critical part of our everyday visual experience, but recognizing the social interactions of others is often considered outside the scope of vision and grouped with higher-level social cognition like theory of mind. Recent work, however, has revealed that recognition of social interactions is efficient and automatic, is well modeled by bottom-up computational algorithms, and occurs in visually-selective regions of the brain. We review recent evidence from these three methodologies (behavioral, computational, and neural) that converge to suggest the core of social interaction perception is visual. We propose a computational framework for how this process is carried out in the brain and offer directions for future interdisciplinary investigations of social perception.

Joint intention understanding in children with autism spectrum disorder

This study examined the ability of children with autism spectrum disorder (ASD) to generate joint intention-based action prediction in a joint action task. Children were presented with a series of videos in which two actors either played with blocks based on joint intention (social condition) or played with blocks independently (nonsocial condition). In the familiarization stage, two actors demonstrated how they played with blocks three times. In the test stage, one actor left the scene, and another actor grasped a block and asked where she should place it. Children's gaze behavior was assessed by an eye tracker. After watching videos, children were asked to answer two questions: an action prediction question and an intention understanding question. The results showed that in the implicit eye movement task, children with ASD and typically developing (TD) children exhibited location-based anticipatory gaze under both conditions. However, in terms of explicit behavioral responses, TD children showed higher accuracy in response to action prediction questions and intention understanding questions than children with ASD in the social condition, while no significant group difference was found in the nonsocial condition. These results indicate that children with ASD have difficulty understanding joint intention and that their action prediction is primarily driven by bottom-up sensory inputs.

Infants expect agents to minimize the collective cost of collaborative actions

This paper argues that human infants address the challenges of optimizing, recognizing, and interpreting collaborative behaviors by assessing their collective efficiency. This hypothesis was tested by using a looking-time study. Fourteen-month-olds (N = 32) were familiarized with agents performing a collaborative action in computer animations. During the test phase, the looking times were measured while the agents acted with various efficiency parameters. In the critical condition, the agents' actions were individually efficient, but their combination was either collectively efficient or inefficient. Infants looked longer at test events that violated expectations of collective efficiency (p = .006, d = 0.79). Thus, preverbal infants apply expectations of collective efficiency to actions involving multiple agents.

Sequential learning of emotional faces is statistical at 12 months of age

Infants are capable of extracting statistical regularities from continuous streams of elements, which helps them structuring their surrounding environment. The current study examines 12-month-olds' capacity to extract statistical information from a sequence of emotional faces. Using a familiarization procedure, infants were presented with videos of two actresses expressing the same facial emotion, and subsequently turning toward or away from each other. Videos displayed different emotions (i.e., anger, happiness, fear, sadness, surprise, amusement, disgust, and exasperation) and were organized sequentially, so that the transitional probabilities between videos were highly predictable in some cases, and less predictable in others. At test, infants discriminated highly predictable from low predictable transitional probabilities, suggesting that they extracted statistical regularities from the sequence of emotional faces. However, when examining the looking toward and the looking away conditions separately, infants showed evidence of statistical learning in the looking toward condition only. Together, these findings suggest that 12-month-old infants rely on statistical learning to segment a continuous sequence of emotional faces, although this ability can be modulated by the nature of the stimuli. The contribution of statistical learning to structure infants' social environment is discussed.

Cultural Similarities and Differences in the Development of Sociomoral Judgments: An Eye-Tracking Study

People integrate the valence of behavior and that of outcome when making moral judgments. However, the role of culture in the development of this integration among young children remains unclear. We investigated cultural similarities and differences in moral judgments by measuring both visual attention and verbal evaluations. Three- and four-year-olds from Japan and the U.S. (N = 141) were shown sociomoral scenarios that varied in agents' behavior which reflected prosocial or antisocial intention and recipients' emotional outcome (happy, neutral, or sad); then, they were asked to evaluate agents' moral trait. Their eye fixations while observing moral scenarios were measured using an eye-tracker. We found culturally similar tendencies in the integration of behavior and outcome; however, a cultural difference was shown in their verbal evaluation. The link between implicit attention and explicit verbal evaluation was negligible. Both culturally shared and specific aspects of sociomoral development are discussed.

Let's get it together: Infants generate visual predictions based on collaborative goals

Infants engage in social interactions that include multiple partners from very early in development. A growing body of research shows that infants visually predict the outcomes of an individual's intentional actions, such as a person reaching towards an object (e.g., Krogh-Jespersen & Woodward, 2014), and even show sophistication in their predictions regarding failed actions (e.g., Brandone, Horwitz, Aslin, & Wellman, 2014). Less is known about infants' understanding of actions involving more than one individual (e.g., collaborative actions), which require representing each partners' actions in light of the shared goal. Using eye-tracking, Study 1 examined whether 14-month-old infants visually predict the actions of an individual based on her previously shared goal. Infants viewed videos of two women engaged in either a collaborative or noncollaborative interaction. At test, only one woman was present and infants' visual predictions regarding her future actions were measured. Fourteen-month-olds anticipated an individual's future actions based on her past collaborative behavior. Study 2 revealed that 11-month-old infants only visually predict higher-order shared goals after engaging in a collaborative intervention. Together, our results indicate that by the second year after birth, infants perceive others' collaborative actions as structured by shared goals and that active engagement in collaboration strengthens this understanding in young infants.

Developmental changes in visual responses to social interactions

•

Children show less interaction selectivity in the pSTS than adults.

•

Adults show bilateral pSTS selectivity, while children are more right-lateralized.

•

Exploratory findings suggest interaction selectivity in pSTS is more focally tuned in adults.

Recent evidence demonstrates that a region of the posterior superior temporal sulcus (pSTS) is selective to visually observed social interactions in adults. In contrast, little is known about neural responses to social interactions in children. Here, we used fMRI to ask whether the pSTS is ‘tuned’ to social interactions in children at all, and if so, how selectivity might differ from adults. This was investigated in the pSTS, along with several other socially-tuned regions in neighbouring temporal cortex: extrastriate body area, face selective STS, fusiform face area, and mentalizing selective temporo-parietal junction.

Both children and adults showed selectivity to social interaction within right pSTS, while only adults showed selectivity on the left. Adults also showed both more focal and greater selectivity than children (6–12 years) bilaterally. Exploratory sub-group analyses showed that younger children (6–8), but not older children (9–12), are less selective than adults on the right, while there was a continuous developmental trend (adults > older > younger) in left pSTS. These results suggest that, over development, the neural response to social interactions is characterized by increasingly more selective, focal, and bilateral pSTS responses, a process that likely continues into adolescence.

Six-Month-Old Infants Predict Agents' Goal-Directed Actions on Occluded Objects

Infants can infer agents' goals after observing agents' goal-directed actions on objects and can subsequently make predictions about how agents will act on objects in the future. We investigated the representations supporting these predictions. We familiarized 6-month-old infants to an agent who preferentially reached for one of two featurally distinct objects following a cue. At test, the objects were sequentially occluded from the infant in the agent's presence. We asked whether infants could generate action predictions without visual access to the relevant objects by measuring whether infants shifted their gaze to the location of the agent's hidden goal object following the cue. We also examined what infants represented about the hidden objects by removing one of the occluders to reveal either the original hidden object or the unexpected other object and measuring infants' looking time. We found that, even without visual access to the objects, infants made predictive gazes to the location of the agent's occluded goal object, but failed to represent the features of either hidden object. These results suggest that infants make goal-based action predictions when the relevant objects in the scene are occluded, but doing so may come at the expense of maintaining representations of the objects.

How do infants and adults process communicative events in real time?

Speech allows humans to communicate and to navigate the social world. By 12 months, infants recognize that speech elicits appropriate responses from others. However, it is unclear how infants process dynamic communicative scenes and how their processing abilities compare with those of adults. Do infants, like adults, process communicative events while the event is occurring or only after being presented with the outcome? We examined 12-month-olds' and adults' eye movements as they watched a Communicator grasp one (target) of two objects. During the test event, the Communicator could no longer reach the objects, so she spoke or coughed to a Listener, who selected either object. Infants' and adults' patterns of looking to the actors and objects revealed that both groups immediately evaluated the Communicator's speech, but not her cough, as communicative and recognized that the Listener should select the target object only when the Communicator spoke. Furthermore, infants and adults shifted their attention between the actors and the objects in very similar ways. This suggests that 12-month-olds can quickly process communicative events as they occur with adult-like accuracy. However, differences in looking reveal that 12-month-olds process slower than adults. This early developing processing ability may allow infants to learn language and acquire knowledge from communicative interactions.

Action Prediction Allows Hypothesis Testing via Internal Forward Models at 6 Months of Age

We propose that action prediction provides a cornerstone in a learning process known as internal forward models. According to this suggestion infants' predictions (looking to the mouth of someone moving a spoon upward) will moments later be validated or proven false (spoon was in fact directed toward a bowl), information that is directly perceived as the distance between the predicted and actual goal. Using an individual difference approach we demonstrate that action prediction correlates with the tendency to react with surprise when social interactions are not acted out as expected (action evaluation). This association is demonstrated across tasks and in a large sample (n = 118) at 6 months of age. These results provide the first indication that infants might rely on internal forward models to structure their social world. Additional analysis, consistent with prior work and assumptions from embodied cognition, demonstrates that the latency of infants' action predictions correlate with the infant's own manual proficiency.

Statistical learning in social action contexts

Sensitivity to the regularities and structure contained within sequential, goal-directed actions is an important building block for generating expectations about the actions we observe. Until now, research on statistical learning for actions has solely focused on individual action sequences, but many actions in daily life involve multiple actors in various interaction contexts. The current study is the first to investigate the role of statistical learning in tracking regularities between actions performed by different actors, and whether the social context characterizing their interaction influences learning. That is, are observers more likely to track regularities across actors if they are perceived as acting jointly as opposed to in parallel? We tested adults and toddlers to explore whether social context guides statistical learning and-if so-whether it does so from early in development. In a between-subjects eye-tracking experiment, participants were primed with a social context cue between two actors who either shared a goal of playing together ('Joint' condition) or stated the intention to act alone ('Parallel' condition). In subsequent videos, the actors performed sequential actions in which, for certain action pairs, the first actor's action reliably predicted the second actor's action. We analyzed predictive eye movements to upcoming actions as a measure of learning, and found that both adults and toddlers learned the statistical regularities across actors when their actions caused an effect. Further, adults with high statistical learning performance were sensitive to social context: those who observed actors with a shared goal were more likely to correctly predict upcoming actions. In contrast, there was no effect of social context in the toddler group, regardless of learning performance. These findings shed light on how adults and toddlers perceive statistical regularities across actors depending on the nature of the observed social situation and the resulting effects.

Callous-unemotional traits affect adolescents' perception of collaboration

BACKGROUND

How is the perception of collaboration influenced by individual characteristics, in particular high levels of callous-unemotional (CU) traits? CU traits are associated with low empathy and endorsement of negative social goals such as dominance and forced respect. Thus, it is possible that they could relate to difficulties in interpreting that others are collaborating based on a shared goal.

METHODS

In the current study, a community sample of 15- to 16-year olds participated in an eye tracking task measuring whether they expect that others engaged in an action sequence are collaborating, depending on the emotion they display toward each other. Positive emotion would indicate that they share a goal, while negative emotion would indicate that they hold individual goals.

RESULTS

When the actors showed positive emotion toward each other, expectations of collaboration varied with CU traits. The higher adolescents were on CU traits, the less likely they were to expect collaboration. When the actors showed negative emotion toward each other, CU traits did not influence expectations of collaboration.

CONCLUSIONS

The findings suggest that CU traits are associated with difficulty in perceiving positive social interactions, which could further contribute to the behavioral and emotional problems common to those with high CU traits.

Eye Movements During Action Observation

An important element in social interactions is predicting the goals of others, including the goals of others' manual actions. Over a decade ago, Flanagan and Johansson demonstrated that, when observing other people reaching for objects, the observer's gaze arrives at the goal before the action is completed. Moreover, those authors proposed that this behavior was mediated by an embodied process, which takes advantage of the observer's motor knowledge. Here, we scrutinize work that has followed that seminal article. We include studies on adults that have used combined eye tracking and transcranial magnetic stimulation technologies to test causal hypotheses about underlying brain circuits. We also include developmental studies on human infants. We conclude that, although several aspects of the embodied process of predictive eye movements remain to be clarified, current evidence strongly suggests that the motor system plays a causal role in guiding predictive gaze shifts that focus on another person's future goal. The early emergence of the predictive gaze in infant development underlines its importance for social cognition and interaction.

Novel paradigms to measure variability of behavior in early childhood: posture, gaze, and pupil dilation

A central challenge of investigating the underlying mechanisms of and the individual differences in young children's behavior is the measurement of the internal physiological mechanism and the involved expressive emotions. Here, we illustrate two paradigms that assess concurrent indicators of both children's social perception as well as their emotional expression. In one set of studies, children view situations while their eye movements are mapped onto a live scene. In these studies, children's internal arousal is measured via changes in their pupil dilation by using eye tracking technology. In another set of studies, we measured children's emotional expression via changes in their upper-body posture by using depth sensor imaging technology. Together, these paradigms can provide new insights into the internal mechanism and outward emotional expression involved in young children's behavior.

The Microstructure of Action Perception in Infancy: Decomposing the Temporal Structure of Social Information Processing

In this article, we review recent evidence of infants' early competence in perceiving and interpreting the actions of others. We present a theoretical model that decomposes the timeline of action perception into a series of distinct processes that occur in a particular order. Once an agent is detected, covert attention can be allocated to the future state of the agent (priming), which may lead to overt gaze shifts that predict goals (prediction). Once these goals are achieved, the consequence of the agents' actions and the manner in which the actions were performed can be evaluated (evaluation). We propose that all of these processes have unique requirements, both in terms of timing and cognitive resources. To understand more fully the rich social world of infants, we need to pay more attention to the temporal structure of social perception and ask what information is available to infants and how this changes over time.

Eighteen-Month-Olds, but not 14-Month-Olds, Use Social Context to Bind Action Sequences

We demonstrate that 18-month-olds, but not 14-month-olds, can anticipate others' actions based on an interpretation of shared goals that bind together individual actions into a collaborative sequence. After viewing a sequence of actions performed by two people who socially interact, 18-month-olds bound together the socially engaged actors' actions such that they later expected the actors to share the same final goal. Eighteen-month-olds who saw nonsocially engaged actors did not have this expectation and neither did 14-month-olds when viewing either socially or nonsocially engaged actors. The results are discussed in light of the possibility that experience in collaborations could be necessary for understanding collaboration from a third-person perspective.

Children use nonverbal cues to make inferences about social power

Four studies (N = 192) tested whether young children use nonverbal information to make inferences about differences in social power. Five- and six-year-old children were able to determine which of two adults was "in charge" in dynamic videotaped conversations (Study 1) and in static photographs (Study 4) using only nonverbal cues. Younger children (3-4 years) were not successful in Study 1 or Study 4. Removing irrelevant linguistic information from conversations did not improve the performance of 3- to 4-year-old children (Study 3), but including relevant linguistic cues did (Study 2). Thus, at least by 5 years of age, children show sensitivity to some of the same nonverbal cues adults use to determine other people's social roles.

Human infants detect other people's interactions based on complex patterns of kinematic information

Do infants perceive other people's interactions by means of a mechanism that integrates biological motion information across the observed individuals? In support of this view, the present study demonstrates that infants (N = 28, Age  = 14 months) discriminate between point light displays representing disrupted and non-disrupted interactions between people, even though the two interaction types are identical at the level of individual point light agents. Moreover, a second experiment (sample 2: N = 28, Age  = 14 months) indicated that visual preference in this context is influenced by an audiovisual integration processes that takes into account the presence of an interaction between people. All these results were found exclusively for upright displays--when stimuli were shown upside-down (disrupting biological motion processing), performance was random. Collectively, these findings point to an important role for biological motion in social perception in human infants.

Infants' online perception of give-and-take interactions

This research investigated infants' online perception of give-me gestures during observation of a social interaction. In the first experiment, goal-directed eye movements of 12-month-olds were recorded as they observed a give-and-take interaction in which an object is passed from one individual to another. Infants' gaze shifts from the passing hand to the receiving hand were significantly faster when the receiving hand formed a give-me gesture relative to when it was presented as an inverted hand shape. Experiment 2 revealed that infants' goal-directed gaze shifts were not based on different affordances of the two receiving hands. Two additional control experiments further demonstrated that differences in infants' online gaze behavior were not mediated by an attentional preference for the give-me gesture. Together, our findings provide evidence that properties of social action goals influence infants' online gaze during action observation. The current studies demonstrate that infants have expectations about well-formed object transfer actions between social agents. We suggest that 12-month-olds are sensitive to social goals within the context of give-and-take interactions while observing from a third-party perspective.

Perception of individual and joint action in infants and adults

Infants and adults frequently observe actions performed jointly by more than one person. Research in action perception, however, has focused largely on actions performed by an individual person. Here, we explore how 9- and 12-month-old infants and adults perceive a block-stacking action performed by either one agent (individual condition) or two agents (joint condition). We used eye tracking to measure the latency of participants' gaze shifts towards action goals. Adults anticipated goals in both conditions significantly faster than infants, and their gaze latencies did not differ between conditions. By contrast, infants showed faster anticipation of goals in the individual condition than in the joint condition. This difference was more pronounced in 9-month-olds. Further analyses of fixations examined the role of visual attention in action perception. These findings are cautiously interpreted in terms of low-level processing in infants and higher-level processing in adults. More precisely, our results suggest that adults are able to infer the overarching joint goal of two agents, whereas infants are not yet able to do so and might rely primarily on visual cues to infer the respective sub-goals. In conclusion, our findings indicate that the perception of joint action in infants develops differentially from that of individual action.

The early origins of human charity: developmental changes in preschoolers' sharing with poor and wealthy individuals

Recent studies have provided evidence that young children already engage in sharing behavior. The underlying social-cognitive mechanisms, however, are still under debate. In particular, it is unclear whether or not young children's sharing is motivated by an appreciation of others' wealth. Manipulating the material needs of recipients in a sharing task (Experiment 1) and a resource allocation task (Experiment 2), we show that 5- but not 3-year-old children share more with poor than wealthy individuals. The 3-year-old children even showed a tendency to behave less selfishly towards the rich, yet not the poor recipient. This suggests that very early instances of sharing behavior are not motivated by a consideration of others' material needs. Moreover, the results show that 5-year-old children were rather inclined to give more to the poor individual than distributing the resources equally, demonstrating that their wish to support the poor overruled the otherwise very prominent inclination to share resources equally. This indicates that charity has strong developmental roots in preschool children.