Social is special: A normative framework for teaching with and learning from evaluative feedback

Inside the funhouse mirror factory: How social media distorts perceptions of norms

The current paper explains how modern technology interacts with human psychology to create a funhouse mirror version of social norms. We argue that norms generated on social media often tend to be more extreme than offline norms which can create false perceptions of norms-known as pluralistic ignorance. We integrate research from political science, psychology, and cognitive science to explain how online environments become saturated with false norms, who is misrepresented online, what happens when online norms deviate from offline norms, where people are affected online, and why expressions are more extreme online. We provide a framework for understanding and correcting for the distortions in our perceptions of social norms that are created by social media platforms. We argue the funhouse mirror nature of social media can be pernicious for individuals and society by increasing pluralistic ignorance and false polarization.

Social satisficing: Multi-agent reinforcement learning with satisficing agents

For a reinforcement learning agent to finish trial-and-error in a realistic time duration, it is necessary to limit the scope of exploration during the learning process. However, limiting the exploration scope means limitation in optimality: the agent could fall into a suboptimal solution. This is the nature of local, bottom-up way of learning. An alternative way is to set a goal to be achieved, which is a more global, top-down way. The risk-sensitive satisficing (RS) value function incorporate, as a method of the latter way, the satisficing principle into reinforcement learning and enables agents to quickly converge to exploiting the optimal solution without falling into a suboptimal one, when an appropriate goal (aspiration level) is given. However, how best to determine the aspiration level is still an open problem. This study proposes social satisficing, a framework for multi-agent reinforcement learning which determines the aspiration level through information sharing among multiple agents. In order to verify the effectiveness of this novel method, we conducted simulations in a learning environment with many suboptimal goals (SuboptimaWorld). The results show that the proposed method, which converts the aspiration level at the episodic level into local (state-wise) aspiration levels, possesses a higher learning efficiency than any of the compared methods, and that the novel method has the ability to autonomously adjust exploration scope, while keeping the shared information minimal. This study provides a glimpse into an aspect of human and biological sociality which has been mentioned little in the context of artificial intelligence and machine learning.

The influence of social feedback on reward learning in the Iowa gambling task

Learning, an important activity for both human and animals, has long been a focal point of research. During the learning process, subjects assimilate not only their own information but also information from others, a phenomenon known as social learning. While numerous studies have explored the impact of social feedback as a reward/punishment during learning, few studies have investigated whether social feedback facilitates or inhibits the learning of environmental rewards/punishments. This study aims to test the effects of social feedback on economic feedback and its cognitive processes by using the Iowa Gambling Task (IGT). One hundred ninety-two participants were recruited and categorized into one non-social feedback group and four social feedback groups. Participants in the social feedback groups were informed that after the outcome of each choice, they would also receive feedback from an online peer. This peer was a fictitious entity, with variations in identity (novice or expert) and feedback type (random or effective). The Outcome-Representation Learning model (ORL model) was used to quantify the cognitive components of learning. Behavioral results showed that both the identity of the peer and the type of feedback provided significantly influenced the deck selection, with effective social feedback increasing the ratio of chosen good decks. Results in the ORL model showed that the four social feedback groups exhibited lower learning rates for gain and loss compared to the nonsocial feedback group, which suggested, in the social feedback groups, the impact of the recent outcome on the update of value decreased. Parameters such as forgetfulness, win frequency, and deck perseverance in the expert-effective feedback group were significantly higher than those in the non-social feedback and expert-random feedback groups. These findings suggest that individuals proactively evaluate feedback providers and selectively adopt effective feedback to enhance learning.

Human and nonhuman norms: a dimensional framework

Human communities teem with a variety of social norms. In order to change unjust and harmful social norms, it is crucial to identify the psychological processes that give rise to them. Most researchers take it for granted that social norms are uniquely human. By contrast, we approach this matter from a comparative perspective, leveraging recent research on animal social behaviour. While there is currently only suggestive evidence for norms in nonhuman communities, we argue that human social norms are likely produced by a wide range of mechanisms, many of which we share with nonhuman animals. Approaching this variability from a comparative perspective can help norm researchers expand and reframe the range of hypotheses they test when attempting to understand the causes of socially normative behaviours in humans. First, we diagnose some of the theoretical obstacles to developing a comparative science of social norms, and offer a few basic constructs and distinctions to help norm researchers overcome these obstacles. Then we develop a six-dimensional model of the psychological and social factors that contribute to variability in both human and potential nonhuman norms. This article is part of the theme issue 'Social norm change: drivers and consequences'.

Computational Social Psychology

Social psychologists attempt to explain how we interact by appealing to basic principles of how we think. To make good on this ambition, they are increasingly relying on an interconnected set of formal tools that model inference, attribution, value-guided decision making, and multi-agent interactions. By reviewing progress in each of these areas and highlighting the connections between them, we can better appreciate the structure of social thought and behavior, while also coming to understand when, why, and how formal tools can be useful for social psychologists.

Rethinking Norm Psychology

Norms permeate human life. Most of people's activities can be characterized by rules about what is appropriate, allowed, required, or forbidden-rules that are crucial in making people hyper-cooperative animals. In this article, I examine the current cognitive-evolutionary account of "norm psychology" and propose an alternative that is better supported by evidence and better placed to promote interdisciplinary dialogue. The incumbent theory focuses on rules and claims that humans genetically inherit cognitive and motivational mechanisms specialized for processing these rules. The cultural-evolutionary alternative defines normativity in relation to behavior-compliance, enforcement, and commentary-and suggests that it depends on implicit and explicit processes. The implicit processes are genetically inherited and domain-general; rather than being specialized for normativity, they do many jobs in many species. The explicit processes are culturally inherited and domain-specific; they are constructed from mentalizing and reasoning by social interaction in childhood. The cultural-evolutionary, or "cognitive gadget," perspective suggests that people alive today-parents, educators, elders, politicians, lawyers-have more responsibility for sustaining normativity than the nativist view implies. People's actions not only shape and transmit the rules, but they also create in each new generation mental processes that can grasp the rules and put them into action.

The utilitarian brain: Moving beyond the Free Energy Principle

The Free Energy Principle (FEP) is a normative computational framework for iterative reduction of prediction error and uncertainty through perception-intervention cycles that has been presented as a potential unifying theory of all brain functions (Friston, 2006). Any theory hoping to unify the brain sciences must be able to explain the mechanisms of decision-making, an important cognitive faculty, without the addition of independent, irreducible notions. This challenge has been accepted by several proponents of the FEP (Friston, 2010; Gershman, 2019). We evaluate attempts to reduce decision-making to the FEP, using Lucas' (2005) meta-theory of the brain's contextual constraints as a guidepost. We find reductive variants of the FEP for decision-making unable to explain behavior in certain types of diagnostic, predictive, and multi-armed bandit tasks. We trace the shortcomings to the core theory's lack of an adequate notion of subjective preference or "utility", a concept central to decision-making and grounded in the brain's biological reality. We argue that any attempts to fully reduce utility to the FEP would require unrealistic assumptions, making the principle an unlikely candidate for unifying brain science. We suggest that researchers instead attempt to identify contexts in which either informational or independent reward constraints predominate, delimiting the FEP's area of applicability. To encourage this type of research, we propose a two-factor formal framework that can subsume any FEP model and allows experimenters to compare the contributions of informational versus reward constraints to behavior.

Algorithm-mediated social learning in online social networks

Human social learning is increasingly occurring on online social platforms, such as Twitter, Facebook, and TikTok. On these platforms, algorithms exploit existing social-learning biases (i.e., towards prestigious, ingroup, moral, and emotional information, or 'PRIME' information) to sustain users' attention and maximize engagement. Here, we synthesize emerging insights into 'algorithm-mediated social learning' and propose a framework that examines its consequences in terms of functional misalignment. We suggest that, when social-learning biases are exploited by algorithms, PRIME information becomes amplified via human-algorithm interactions in the digital social environment in ways that cause social misperceptions and conflict, and spread misinformation. We discuss solutions for reducing functional misalignment, including algorithms promoting bounded diversification and increasing transparency of algorithmic amplification.

The cultural evolution of teaching

Teaching is an important process of cultural transmission. Some have argued that human teaching is a cognitive instinct - a form of 'natural cognition' centred on mindreading, shaped by genetic evolution for the education of juveniles, and with a normative developmental trajectory driven by the unfolding of a genetically inherited predisposition to teach. Here, we argue instead that human teaching is a culturally evolved trait that exhibits characteristics of a cognitive gadget. Children learn to teach by participating in teaching interactions with socialising agents, which shape their own teaching practices. This process hijacks psychological mechanisms involved in prosociality and a range of domain-general cognitive abilities, such as reinforcement learning and executive function, but not a suite of cognitive adaptations specifically for teaching. Four lines of evidence converge on this hypothesis. The first, based on psychological experiments in industrialised societies, indicates that domain-general cognitive processes are important for teaching. The second and third lines, based on naturalistic and experimental research in small-scale societies, indicate marked cross-cultural variation in mature teaching practice and in the ontogeny of teaching among children. The fourth line indicates that teaching has been subject to cumulative cultural evolution, i.e. the gradual accumulation of functional changes across generations.

Entering into a self-regulated learning mode prevents detrimental effects of feedback removal on memory

Incentives can decrease performance by undermining intrinsic motivation. How such an interplay of external reinforcers and internal self-regulation influences memory processes, however, is less known. Here, we investigated their interaction on memory performance while learning the meaning of new-words from their context. Specifically, participants inferred congruent meanings of new-words from semantic context (congruent trials) or lack of congruence (incongruent trials), while receiving external feedback in the first or second half of trials only. Removing feedback during learning of congruent word meanings lowered subsequent recognition rates a day later, whereas recognition remained high in the group, which received feedback only in the second half. In contrast, feedback did not substantially alter recognition rates for learning that new-words had no congruent meanings. Our findings suggest that external reinforcers can selectively impair memories if internal self-regulated processes are not already established, but whether they do so depends on what is being learned (specific word-meanings vs. unspecific incongruence). This highlights the relevance of self-regulated learning in education to support stable memory formation.

Leveraging artificial intelligence to improve people's planning strategies

SignificanceMany bad decisions and their devastating consequences could be avoided if people used optimal decision strategies. Here, we introduce a principled computational approach to improving human decision making. The basic idea is to give people feedback on how they reach their decisions. We develop a method that leverages artificial intelligence to generate this feedback in such a way that people quickly discover the best possible decision strategies. Our empirical findings suggest that a principled computational approach leads to improvements in decision-making competence that transfer to more difficult decisions in more complex environments. In the long run, this line of work might lead to apps that teach people clever strategies for decision making, reasoning, goal setting, planning, and goal achievement.

The computational challenge of social learning

The complex reward structure of the social world and the uncertainty endemic to social contexts poses a challenge for modeling. For example, during social interactions, the actions of one person influence the internal states of another. These social dependencies make it difficult to formalize social learning problems in a mathematically tractable way. While it is tempting to dispense with these complexities, they are a defining feature of social life. Because the structure of social interactions challenges the simplifying assumptions often made in models, they make an ideal testbed for computational models of cognition. By adopting a framework that embeds existing social knowledge into the model, we can go beyond explaining behaviors in laboratory tasks to explaining those observed in the wild.

Trusting and learning from others: immediate and long-term effects of learning from observation and advice

Social learning underpins our species's extraordinary success. Learning through observation has been investigated in several species, but learning from advice-where information is intentionally broadcast-is less understood. We used a pre-registered, online experiment (n = 1492) combined with computational modelling to examine learning through observation and advice. Participants were more likely to immediately follow advice than to copy an observed choice, but this was dependent upon trust in the adviser: highly paranoid participants were less likely to follow advice in the short term. Reinforcement learning modelling revealed two distinct patterns regarding the long-term effects of social information: some individuals relied fully on social information, whereas others reverted to trial-and-error learning. This variation may affect the prevalence and fidelity of socially transmitted information. Our results highlight the privileged status of advice relative to observation and how the assimilation of intentionally broadcast information is affected by trust in others.

Emotion prediction errors guide socially adaptive behaviour

People make decisions based on deviations from expected outcomes, known as prediction errors. Past work has focused on reward prediction errors, largely ignoring violations of expected emotional experiences—emotion prediction errors. We leverage a method to measure real-time fluctuations in emotion as people decide to punish or forgive others. Across four studies (N=1,016), we reveal that emotion and reward prediction errors have distinguishable contributions to choice, such that emotion prediction errors exert the strongest impact during decision-making. We additionally find that a choice to punish or forgive can be decoded in less than a second from an evolving emotional response, suggesting emotions swiftly influence choice. Finally, individuals reporting significant levels of depression exhibit selective impairments in using emotion—but not reward—prediction errors. Evidence for emotion prediction errors potently guiding social behaviors challenge standard decision-making models that have focused solely on reward.

How social learning amplifies moral outrage expression in online social networks

Moral outrage shapes fundamental aspects of social life and is now widespread in online social networks. Here, we show how social learning processes amplify online moral outrage expressions over time. In two preregistered observational studies on Twitter (7331 users and 12.7 million total tweets) and two preregistered behavioral experiments (N = 240), we find that positive social feedback for outrage expressions increases the likelihood of future outrage expressions, consistent with principles of reinforcement learning. In addition, users conform their outrage expressions to the expressive norms of their social networks, suggesting norm learning also guides online outrage expressions. Norm learning overshadows reinforcement learning when normative information is readily observable: in ideologically extreme networks, where outrage expression is more common, users are less sensitive to social feedback when deciding whether to express outrage. Our findings highlight how platform design interacts with human learning mechanisms to affect moral discourse in digital public spaces.

Adaptive Empathy: Empathic Response Selection as a Dynamic, Feedback-Based Learning Process

Empathy allows us to respond to the emotional state of another person. Considering that an empathic interaction may last beyond the initial response, learning mechanisms may be involved in dynamic adaptation of the reaction to the changing emotional state of the other person. However, traditionally, empathy is assessed through sets of isolated reactions to another's distress. Here we address this gap by focusing on adaptive empathy, defined as the ability to learn and adjust one's empathic responses based on feedback. For this purpose, we designed a novel paradigm of associative learning in which participants chose one of two empathic strategies (reappraisal or distraction) to attenuate the distress of a target person, where one strategy had a higher probability of relieving distress. After each choice, participants received feedback about the success of their chosen strategy in relieving the target person's distress, which they could use to inform their future decisions. The results show that the participants made more accurate choices in the adaptive empathy condition than in a non-social control condition, pointing to an advantage for learning from social feedback. We found a correlation between adaptive empathy and a trait measure of cognitive empathy. These findings indicate that the ability to learn about the effectiveness of empathic responses may benefit from incorporating mentalizing abilities. Our findings provide a lab-based model for studying adaptive empathy and point to the potential contribution of learning theory to enhancing our understanding of the dynamic nature of empathy.

Reinforcement Learning With Human Advice: A Survey

In this paper, we provide an overview of the existing methods for integrating human advice into a reinforcement learning process. We first propose a taxonomy of the different forms of advice that can be provided to a learning agent. We then describe the methods that can be used for interpreting advice when its meaning is not determined beforehand. Finally, we review different approaches for integrating advice into the learning process.

Human biases limit cumulative innovation

Is technological advancement constrained by biases in human cognition? People in all societies build on discoveries inherited from previous generations, leading to cumulative innovation. However, biases in human learning and memory may influence the process of knowledge transmission, potentially limiting this process. Here, we show that cumulative innovation in a continuous optimization problem is systematically constrained by human biases. In a large (n = 1250) behavioural study using a transmission chain design, participants searched for virtual technologies in one of four environments after inheriting a solution from previous generations. Participants converged on worse solutions in environments misaligned with their biases. These results substantiate a mathematical model of cumulative innovation in Bayesian agents, highlighting formal relationships between cultural evolution and distributed stochastic optimization. Our findings provide experimental evidence that human biases can limit the advancement of knowledge in a controlled laboratory setting, reinforcing concerns about bias in creative, scientific and educational contexts.

Machiavellian strategist or cultural learner? Mentalizing and learning over development in a resource-sharing game

Theorists have sought to identify the key selection pressures that drove the evolution of our species' cognitive abilities, life histories and cooperative inclinations. Focusing on two leading theories, each capable of accounting for many of the rapid changes in our lineage, we present a simple experiment designed to assess the explanatory power of both the Machiavellian Intelligence and the Cultural Brain/Intelligence Hypotheses. Children (aged 3-7 years) observed a novel social interaction that provided them with behavioural information that could either be used to outmanoeuvre a partner in subsequent interactions or for cultural learning. The results show that, even after four rounds of repeated interaction and sometimes lower pay-offs, children continued to rely on copying the observed behaviour instead of harnessing the available social information to strategically extract pay-offs (stickers) from their partners. Analyses further reveal that superior mentalizing abilities are associated with more targeted cultural learning - the selective copying of fewer irrelevant actions - while superior generalized cognitive abilities are associated with greater imitation of irrelevant actions. Neither mentalizing capacities nor more general measures of cognition explain children's ability to strategically use social information to maximize pay-offs. These results provide developmental evidence favouring the Cultural Brain/Intelligence Hypothesis over the Machiavellian Intelligence Hypothesis.

The actions of others act as a pseudo-reward to drive imitation in the context of social reinforcement learning

While there is no doubt that social signals affect human reinforcement learning, there is still no consensus about how this process is computationally implemented. To address this issue, we compared three psychologically plausible hypotheses about the algorithmic implementation of imitation in reinforcement learning. The first hypothesis, decision biasing (DB), postulates that imitation consists in transiently biasing the learner's action selection without affecting their value function. According to the second hypothesis, model-based imitation (MB), the learner infers the demonstrator's value function through inverse reinforcement learning and uses it to bias action selection. Finally, according to the third hypothesis, value shaping (VS), the demonstrator's actions directly affect the learner's value function. We tested these three hypotheses in 2 experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task. We show through model comparison and model simulation that VS provides the best explanation of learner's behavior. Results replicated in a third independent experiment featuring a larger cohort and a different design (N = 302). In our experiments, we also manipulated the quality of the demonstrators' choices and found that learners were able to adapt their imitation rate, so that only skilled demonstrators were imitated. We proposed and tested an efficient meta-learning process to account for this effect, where imitation is regulated by the agreement between the learner and the demonstrator. In sum, our findings provide new insights and perspectives on the computational mechanisms underlying adaptive imitation in human reinforcement learning.

The MAD Model of Moral Contagion: The Role of Motivation, Attention, and Design in the Spread of Moralized Content Online

With more than 3 billion users, online social networks represent an important venue for moral and political discourse and have been used to organize political revolutions, influence elections, and raise awareness of social issues. These examples rely on a common process to be effective: the ability to engage users and spread moralized content through online networks. Here, we review evidence that expressions of moral emotion play an important role in the spread of moralized content (a phenomenon we call moral contagion). Next, we propose a psychological model called the motivation, attention, and design (MAD) model to explain moral contagion. The MAD model posits that people have group-identity-based motivations to share moral-emotional content, that such content is especially likely to capture our attention, and that the design of social-media platforms amplifies our natural motivational and cognitive tendencies to spread such content. We review each component of the model (as well as interactions between components) and raise several novel, testable hypotheses that can spark progress on the scientific investigation of civic engagement and activism, political polarization, propaganda and disinformation, and other moralized behaviors in the digital age.

Knowing Ourselves Together: The Cultural Origins of Metacognition

Metacognition - the ability to represent, monitor and control ongoing cognitive processes - helps us perform many tasks, both when acting alone and when working with others. While metacognition is adaptive, and found in other animals, we should not assume that all human forms of metacognition are gene-based adaptations. Instead, some forms may have a social origin, including the discrimination, interpretation, and broadcasting of metacognitive representations. There is evidence that each of these abilities depends on cultural learning and therefore that cultural selection might shape human metacognition. The cultural origins hypothesis is a plausible and testable alternative that directs us towards a substantial new programme of research.

Norms and the Flexibility of Moral Action

A complex web of social and moral norms governs many everyday human behaviors, acting as the glue for social harmony. The existence of moral norms helps elucidate the psychological motivations underlying a wide variety of seemingly puzzling behavior, including why humans help or trust total strangers. In this review, we examine four widespread moral norms: Fairness, altruism, trust, and cooperation, and consider how a single social instrument-reciprocity-underpins compliance to these norms. Using a game theoretic framework, we examine how both context and emotions moderate moral standards, and by extension, moral behavior. We additionally discuss how a mechanism of reciprocity facilitates the adherence to, and enforcement of, these moral norms through a core network of brain regions involved in processing reward. In contrast, violating this set of moral norms elicits neural activation in regions involved in resolving decision conflict and exerting cognitive control. Finally, we review how a reinforcement mechanism likely governs learning about morally normative behavior. Together, this review aims to explain how moral norms are deployed in ways that facilitate flexible moral choices.

A social insect perspective on the evolution of social learning mechanisms

The social world offers a wealth of opportunities to learn from others, and across the animal kingdom individuals capitalize on those opportunities. Here, we explore the role of natural selection in shaping the processes that underlie social information use, using a suite of experiments on social insects as case studies. We illustrate how an associative framework can encompass complex, context-specific social learning in the insect world and beyond, and based on the hypothesis that evolution acts to modify the associative process, suggest potential pathways by which social information use could evolve to become more efficient and effective. Social insects are distant relatives of vertebrate social learners, but the research we describe highlights routes by which natural selection could coopt similar cognitive raw material across the animal kingdom.

Moral learning: Psychological and philosophical perspectives

The past 15years occasioned an extraordinary blossoming of research into the cognitive and affective mechanisms that support moral judgment and behavior. This growth in our understanding of moral mechanisms overshadowed a crucial and complementary question, however: How are they learned? As this special issue of the journal Cognition attests, a new crop of research into moral learning has now firmly taken root. This new literature draws on recent advances in formal methods developed in other domains, such as Bayesian inference, reinforcement learning and other machine learning techniques. Meanwhile, it also demonstrates how learning and deciding in a social domain-and especially in the moral domain-sometimes involves specialized cognitive systems. We review the contributions to this special issue and situate them within the broader contemporary literature. Our review focuses on how we learn moral values and moral rules, how we learn about personal moral character and relationships, and the philosophical implications of these emerging models.