Measuring self-regulation in everyday life: Reliability and validity of smartphone-based experiments in alcohol use disorder

Self-regulation, the ability to guide behavior according to one's goals, plays an integral role in understanding loss of control over unwanted behaviors, for example in alcohol use disorder (AUD). Yet, experimental tasks that measure processes underlying self-regulation are not easy to deploy in contexts where such behaviors usually occur, namely outside the laboratory, and in clinical populations such as people with AUD. Moreover, lab-based tasks have been criticized for poor test-retest reliability and lack of construct validity. Smartphones can be used to deploy tasks in the field, but often require shorter versions of tasks, which may further decrease reliability. Here, we show that combining smartphone-based tasks with joint hierarchical modeling of longitudinal data can overcome at least some of these shortcomings. We test four short smartphone-based tasks outside the laboratory in a large sample (N = 488) of participants with AUD. Although task measures indeed have low reliability when data are analyzed traditionally by modeling each session separately, joint modeling of longitudinal data increases reliability to good and oftentimes excellent levels. We next test the measures' construct validity and show that extracted latent factors are indeed in line with theoretical accounts of cognitive control and decision-making. Finally, we demonstrate that a resulting cognitive control factor relates to a real-life measure of drinking behavior and yields stronger correlations than single measures based on traditional analyses. Our findings demonstrate how short, smartphone-based task measures, when analyzed with joint hierarchical modeling and latent factor analysis, can overcome frequently reported shortcomings of experimental tasks.

Risk taking for potential losses but not gains increases with time of day

Humans exhibit distinct risk preferences when facing choices involving potential gains and losses. These preferences are believed to be subject to neuromodulatory influence, particularly from dopamine and serotonin. As neuromodulators manifest circadian rhythms, this suggests decision making under risk might be affected by time of day. Here, in a large subject sample collected using a smartphone application, we found that risky options with potential losses were increasingly chosen over the course of the day. We observed this result in both a within-subjects design (N = 2599) comparing risky options chosen earlier and later in the day in the same individuals, and in a between-subjects design (N = 26,720) showing our effect generalizes across ages and genders. Using computational modelling, we show this diurnal change in risk preference reflects a decrease in sensitivity to increasing losses, but no change was observed in the relative impacts of gains and losses on choice (i.e., loss aversion). Thus, our findings reveal a striking diurnal modulation in human decision making, a pattern with potential importance for real-life decisions that include voting, medical decisions, and financial investments.

A highly replicable decline in mood during rest and simple tasks

Does our mood change as time passes? This question is central to behavioural and affective science, yet it remains largely unexamined. To investigate, we intermixed subjective momentary mood ratings into repetitive psychology paradigms. Here we demonstrate that task and rest periods lowered participants' mood, an effect we call 'Mood Drift Over Time'. This finding was replicated in 19 cohorts totalling 28,482 adult and adolescent participants. The drift was relatively large (-13.8% after 7.3 min of rest, Cohen's d = 0.574) and was consistent across cohorts. Behaviour was also impacted: participants were less likely to gamble in a task that followed a rest period. Importantly, the drift slope was inversely related to reward sensitivity. We show that accounting for time using a linear term significantly improves the fit of a computational model of mood. Our work provides conceptual and methodological reasons for researchers to account for time's effects when studying mood and behaviour.

Insights into the accuracy of social scientists' forecasts of societal change

How well can social scientists predict societal change, and what processes underlie their predictions? To answer these questions, we ran two forecasting tournaments testing the accuracy of predictions of societal change in domains commonly studied in the social sciences: ideological preferences, political polarization, life satisfaction, sentiment on social media, and gender-career and racial bias. After we provided them with historical trend data on the relevant domain, social scientists submitted pre-registered monthly forecasts for a year (Tournament 1; N = 86 teams and 359 forecasts), with an opportunity to update forecasts on the basis of new data six months later (Tournament 2; N = 120 teams and 546 forecasts). Benchmarking forecasting accuracy revealed that social scientists' forecasts were on average no more accurate than those of simple statistical models (historical means, random walks or linear regressions) or the aggregate forecasts of a sample from the general public (N = 802). However, scientists were more accurate if they had scientific expertise in a prediction domain, were interdisciplinary, used simpler models and based predictions on prior data.

Opportunity cost determines free-operant action initiation latency and predicts apathy

BACKGROUND

Apathy, a disabling and poorly understood neuropsychiatric symptom, is characterised by impaired self-initiated behaviour. It has been hypothesised that the opportunity cost of time (OCT) may be a key computational variable linking self-initiated behaviour with motivational status. OCT represents the amount of reward which is foregone per second if no action is taken. Using a novel behavioural task and computational modelling, we investigated the relationship between OCT, self-initiation and apathy. We predicted that higher OCT would engender shorter action latencies, and that individuals with greater sensitivity to OCT would have higher behavioural apathy.

METHODS

We modulated the OCT in a novel task called the 'Fisherman Game', Participants freely chose when to self-initiate actions to either collect rewards, or on occasion, to complete non-rewarding actions. We measured the relationship between action latencies, OCT and apathy for each participant across two independent non-clinical studies, one under laboratory conditions (n = 21) and one online (n = 90). 'Average-reward' reinforcement learning was used to model our data. We replicated our findings across both studies.

RESULTS

We show that the latency of self-initiation is driven by changes in the OCT. Furthermore, we demonstrate, for the first time, that participants with higher apathy showed greater sensitivity to changes in OCT in younger adults. Our model shows that apathetic individuals experienced greatest change in subjective OCT during our task as a consequence of being more sensitive to rewards.

CONCLUSIONS

Our results suggest that OCT is an important variable for determining free-operant action initiation and understanding apathy.

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

Aberrant Striatal Value Representation in Huntington's Disease Gene Carriers 25 Years Before Onset

BACKGROUND

In this study, we asked whether differences in striatal activity during a reinforcement learning (RL) task with gain and loss domains could be one of the earliest functional imaging features associated with carrying the Huntington's disease (HD) gene. Based on previous work, we hypothesized that HD gene carriers would show either neural or behavioral asymmetry between gain and loss learning.

METHODS

We recruited 35 HD gene carriers, expected to demonstrate onset of motor symptoms in an average of 26 years, and 35 well-matched gene-negative control subjects. Participants were placed in a functional magnetic resonance imaging scanner, where they completed an RL task in which they were required to learn to choose between abstract stimuli with the aim of gaining rewards and avoiding losses. Task behavior was modeled using an RL model, and variables from this model were used to probe functional magnetic resonance imaging data.

RESULTS

In comparison with well-matched control subjects, gene carriers more than 25 years from motor onset showed exaggerated striatal responses to gain-predicting stimuli compared with loss-predicting stimuli (p = .002) in our RL task. Using computational analysis, we also found group differences in striatal representation of stimulus value (p = .0004). We found no group differences in behavior, cognitive scores, or caudate volumes.

CONCLUSIONS

Behaviorally, gene carriers 9 years from predicted onset have been shown to learn better from gains than from losses. Our data suggest that a window exists in which HD-related functional neural changes are detectable long before associated behavioral change and 25 years before predicted motor onset. These represent the earliest functional imaging differences between HD gene carriers and control subjects.

Smartphones and the Neuroscience of Mental Health

Improvements in understanding the neurobiological basis of mental illness have unfortunately not translated into major advances in treatment. At this point, it is clear that psychiatric disorders are exceedingly complex and that, in order to account for and leverage this complexity, we need to collect longitudinal data sets from much larger and more diverse samples than is practical using traditional methods. We discuss how smartphone-based research methods have the potential to dramatically advance our understanding of the neuroscience of mental health. This, we expect, will take the form of complementing lab-based hard neuroscience research with dense sampling of cognitive tests, clinical questionnaires, passive data from smartphone sensors, and experience-sampling data as people go about their daily lives. Theory- and data-driven approaches can help make sense of these rich data sets, and the combination of computational tools and the big data that smartphones make possible has great potential value for researchers wishing to understand how aspects of brain function give rise to, or emerge from, states of mental health and illness.

The temporal representation of experience in subjective mood

Humans refer to their mood state regularly in day-to-day as well as clinical interactions. Theoretical accounts suggest that when reporting on our mood we integrate over the history of our experiences; yet, the temporal structure of this integration remains unexamined. Here, we use a computational approach to quantitatively answer this question and show that early events exert a stronger influence on reported mood (a primacy weighting) compared to recent events. We show that a Primacy model accounts better for mood reports compared to a range of alternative temporal representations across random, consistent, or dynamic reward environments, different age groups, and in both healthy and depressed participants. Moreover, we find evidence for neural encoding of the Primacy, but not the Recency, model in frontal brain regions related to mood regulation. These findings hold implications for the timing of events in experimental or clinical settings and suggest new directions for individualized mood interventions.

Momentary subjective well-being depends on learning and not reward

Subjective well-being or happiness is often associated with wealth. Recent studies suggest that momentary happiness is associated with reward prediction error, the difference between experienced and predicted reward, a key component of adaptive behaviour. We tested subjects in a reinforcement learning task in which reward size and probability were uncorrelated, allowing us to dissociate between the contributions of reward and learning to happiness. Using computational modelling, we found convergent evidence across stable and volatile learning tasks that happiness, like behaviour, is sensitive to learning-relevant variables (i.e. probability prediction error). Unlike behaviour, happiness is not sensitive to learning-irrelevant variables (i.e. reward prediction error). Increasing volatility reduces how many past trials influence behaviour but not happiness. Finally, depressive symptoms reduce happiness more in volatile than stable environments. Our results suggest that how we learn about our world may be more important for how we feel than the rewards we actually receive.

Many people believe they would be happier if only they had more money. And events such as winning the lottery or receiving a large pay rise do make people happy, at least temporarily. But recent studies suggest that the main factor driving happiness on such occasions is not the size of the reward received. Instead, it is how well that reward matches up with expectations. Receiving a 10% pay rise when you were expecting 1% will make you feel happier than receiving 10% when you had been expecting 20%.

This difference between an expected and an actual reward is referred to as a reward prediction error. Reward prediction errors have a key role in learning. They motivate people to repeat behaviours that led to unexpectedly large rewards. But they also enable people to update their beliefs about the world, which is rewarding in itself. Could it be that reward prediction errors are associated with happiness mainly because they help us understand the world a little better than before?

To test this idea, Blain and Rutledge designed a task in which the likelihood of receiving a reward was unrelated to the size of the reward. This study design makes it possible to separate out the contributions of learning versus reward to moment-by-moment happiness.

In the task, volunteers had to decide which of two cars would win a race. In the ‘stable’ condition, one of the cars always had an 80% chance of winning. In the ‘volatile’ condition, one car had an 80% chance of winning for the first 20 trials. The other car then had an 80% chance of winning for the next 20 trials. The volunteers were not told these probabilities in advance, but had to work them out by playing the game. However, on every trial, the volunteers were shown the reward they would receive if they chose either of the cars and that car went on to win. The size of the rewards varied at random and was unrelated to the likelihood of a car winning.

Every few trials, the volunteers were asked to indicate their current level of happiness on a scale. The results showed that volunteers were happier after winning than after losing. On average they were also happier in the stable condition than in the volatile condition. This was especially true for volunteers with pre-existing symptoms of depression. Moreover, happiness after wins did not depend on how large the reward they got was, but instead simply on how surprised they were to win.

These results suggest that how we learn about the world around us can be more important for how we feel than rewards we receive directly. Measuring happiness in various types of environment could help us understand factors affecting mental health. The current results suggest, for example, that uncertain environments may be especially unpleasant for people with depression. Further research is needed to understand why this might be the case. In the real world, rewards are often uncertain and infrequent, but learning may nevertheless have the potential to boost happiness.

9 Aberrant striatal value representation in Huntington’s disease gene carriers 25 years before onset

Aims

Huntington’s disease (HD) is a devastating genetic neurodegenerative condition typically manifesting clinically in the fourth or fifth decade. With the advent of genetic therapies there is increased need to identify the earliest changes associated with carrying the HD gene. In this study we sought to determine the earliest functional imaging differences between HD gene carriers and matched controls. Based on previous work, we hypothesised that as compared to controls, HD gene carriers decades from onset would show a neural ‘reward bias’ – an exaggerated striatal response to gains as compared to losses.

Methods

We recruited 35 HD gene carriers, estimated to be on average 26 years from motor onset, and 35 controls. Groups were well matched for age, gender and education level.

Participants completed a reinforcement learning task in a fMRI scanner using a sequence optimised for orbitofrontal and striatal signal. In this task participants were required to learn to choose between stimuli with the aim of maximise rewards and avoiding losses. Task behaviour was modelled using a computational model and computational variables from the best fitting model was used to probe fMRI data.

Results

As hypothesised, we found that, in comparison to matched controls, gene carriers over 25 years from motor onset showed exaggerated striatal responses to gain as compared to loss predicting stimuli (p=0.003) in a reinforcement learning task. Using computational analysis, we also found group differences in striatal representation of stimulus value (p=0.0007).

Conclusion

These represent the earliest functional imaging differences between HD gene carriers and controls. Behaviourally gene carriers, 9 years from predicted onset, have shown enhanced learning from gains as compared to losses. Importantly, we found no group differences in behaviour, or caudate volumes. Our data suggests a therapeutic window exists whereby HD- related functional neural changes are detectable 25 years before predicted onset.

Under the Hood: Using Computational Psychiatry to Make Psychological Therapies More Mechanism-Focused

Psychological therapies, such as CBT, are an important part of the treatment of a range of psychiatric disorders such as depression and anxiety. There is a growing desire to understand the mechanisms by which such therapies effect change so as to improve treatment outcomes. Here we argue that adopting a computational framework may be one such approach. Computational psychiatry aims to provide a theoretical framework for moving between higher-level psychological states (like emotions, decisions and beliefs) to neural circuits, by modeling these constructs mathematically. These models are explicit hypotheses that contain quantifiable variables and parameters derived from each individual's behavior. This approach has two advantages. Firstly, some of the variables described by these models appears to reflect the neural activity of specific brain regions. Secondly, the parameters estimated by these models may offer a unique description of a patient's symptoms which can be used to both tailor therapy and track its effect. In doing so this approach may offer some additional granularity in understanding how psychological therapies, such as CBT, are working. Although this field shows significant promise, we also highlight several of the key hurdles that must first be overcome before clinical translation of computational insights can be realized.

Neurocomputational mechanisms underpinning aberrant social learning in young adults with low self-esteem

Low self-esteem is a risk factor for a range of psychiatric disorders. From a cognitive perspective a negative self-image can be maintained through aberrant learning about self-worth derived from social feedback. We previously showed that neural teaching signals that represent the difference between expected and actual social feedback (i.e., social prediction errors) drive fluctuations in self-worth. Here, we used model-based functional magnetic resonance imaging (fMRI) to characterize learning from social prediction errors in 61 participants drawn from a population-based sample (n = 2402) who were recruited on the basis of being in the bottom or top 10% of self-esteem scores. Participants performed a social evaluation task during fMRI scanning, which entailed predicting whether other people liked them as well as the repeated provision of reported feelings of self-worth. Computational modeling results showed that low self-esteem participants had persistent expectations that others would dislike them, and a reduced propensity to update these expectations in response to social prediction errors. Low self-esteem subjects also displayed an enhanced volatility in reported feelings of self-worth, and this was linked to an increased tendency for social prediction errors to determine momentary self-worth. Canonical correlation analysis revealed that individual differences in self-esteem related to several interconnected psychiatric symptoms organized around a single dimension of interpersonal vulnerability. Such interpersonal vulnerability was associated with an attenuated social value signal in ventromedial prefrontal cortex when making predictions about being liked, and enhanced dorsal prefrontal cortex activity upon receipt of social feedback. We suggest these computational signatures of low self-esteem and their associated neural underpinnings might represent vulnerability for development of psychiatric disorder.

Distinct Processing of Aversive Experience in Amygdala Subregions

Background

The amygdala is an anatomically complex medial temporal brain structure whose subregions are considered to serve distinct functions. However, their precise role in mediating human aversive experience remains ill understood.

Methods

We used functional magnetic resonance imaging in 39 healthy volunteers with varying levels of trait anxiety to assess distinct contributions of the basolateral amygdala (BLA) and centromedial amygdala to anticipation and experience of aversive events. Additionally, we examined the relationship between any identified functional subspecialization and measures of subjective reported aversion and trait anxiety.

Results

Our results show that the centromedial amygdala is responsive to aversive outcomes but insensitive to predictive aversive cues. In contrast, the BLA encodes an aversive prediction error that quantifies whether cues and outcomes are worse than expected. A neural representation within the BLA for distinct threat levels was mirrored in self-reported subjective anxiety across individuals. Furthermore, high trait-anxious individuals were characterized by indiscriminately heightened BLA activity in response to aversive cues, regardless of actual threat level.

Conclusions

Our results demonstrate that amygdala subregions are distinctly engaged in processing of aversive experience, with elevated and undifferentiated BLA responses to threat emerging as a potential neurobiological mediator of vulnerability to anxiety disorders.

Machine learning and big data in psychiatry: toward clinical applications

Psychiatry is a medical field concerned with the treatment of mental illness. Psychiatric disorders broadly relate to higher functions of the brain, and as such are richly intertwined with social, cultural, and experiential factors. This makes them exquisitely complex phenomena that depend on and interact with a large number of variables. Computational psychiatry provides two ways of approaching this complexity. Theory-driven computational approaches employ mechanistic models to make explicit hypotheses at multiple levels of analysis. Data-driven machine-learning approaches can make predictions from high-dimensional data and are generally agnostic as to the underlying mechanisms. Here, we review recent advances in the use of big data and machine-learning approaches toward the aim of alleviating the suffering that arises from psychiatric disorders.

Oxytocin modulates social value representations in the amygdala

Humans exhibit considerable variation in how they value their own interest relative to the interests of others. Deciphering the neural codes representing potential rewards for self and others is crucial for understanding social decision-making. Here we integrate computational modeling with functional magnetic resonance imaging to investigate the neural representation of social value and the modulation by oxytocin, a nine-amino acid neuropeptide, in participants evaluating monetary allocations to self and other (self-other allocations). We found that an individual's preferred self-other allocation serves as a reference point for computing the value of potential self-other allocations. In more prosocial participants, amygdala activity encoded a social-value-distance signal; that is, the value dissimilarity between potential and preferred allocations. Intranasal oxytocin administration amplified this amygdala representation and increased prosocial behavior in more individualistic participants but not in more prosocial ones. Our results reveal a neurocomputational mechanism underlying social-value representations and suggest that oxytocin may promote prosociality by modulating social-value representations in the amygdala.

The Psychological and Neural Basis of Loss Aversion

Loss aversion is a central element of prospect theory, the dominant theory of decision making under uncertainty for the past four decades, and refers to the overweighting of potential losses relative to equivalent gains, a critical determinant of risky decision making. Recent advances in affective and decision neuroscience have shed new light on the psychological and neurobiological mechanisms underlying loss aversion. Here, integrating disparate literatures from the level of neurotransmitters to subjective reports of emotion, we propose a novel neural and computational framework that links norepinephrine to loss aversion and identifies a distinct role for dopamine in risk taking for rewards. We also propose that loss aversion specifically relates to anticipated emotions and aspects of the immediate experience of realized gains and losses but not their long-term emotional consequences, highlighting an underappreciated temporal structure. Finally, we discuss challenges to loss aversion and the relevance of loss aversion to understanding psychiatric disorders. Refining models of loss aversion will have broad consequences for the science of decision making and for how we understand individual variation in economic preferences and psychological well-being across both healthy and psychiatric populations.

Beliefs about bad people are volatile

People form moral impressions rapidly, effortlessly and from a remarkably young age^1-5. Putatively 'bad' agents command more attention and are identified more quickly and accurately than benign or friendly agents^5-12. Such vigilance is adaptive, but can also be costly in environments where people sometimes make mistakes, because incorrectly attributing bad character to good people damages existing relationships and discourages forming new relationships^13-16. The ability to accurately infer the moral character of others is critical for healthy social functioning, but the computational processes that support this ability are not well understood. Here, we show that moral inference is explained by an asymmetric Bayesian updating mechanism in which beliefs about the morality of bad agents are more uncertain (and therefore more volatile) than beliefs about the morality of good agents. This asymmetry seems to be a property of learning about immoral agents in general, as we also find greater uncertainty for beliefs about the non-moral traits of bad agents. Our model and data reveal a cognitive mechanism that permits flexible updating of beliefs about potentially threatening others, a mechanism that could facilitate forgiveness when initial bad impressions turn out to be inaccurate. Our findings suggest that negative moral impressions destabilize beliefs about others, promoting cognitive flexibility in the service of cooperative but cautious behaviour.

Neural activity and fundamental learning, motivated by monetary loss and reward, are intact in mild to moderate major depressive disorder

INTRODUCTION

Reduced motivation is an important symptom of major depression, thought to impair recovery by reducing opportunities for rewarding experiences. We characterized motivation for monetary outcomes in depressed outpatients (N = 39, 22 female) and controls (N = 22, 11 female) in terms of their effectiveness in seeking rewards and avoiding losses. We assessed motivational function during learning of associations between stimuli and actions, as well as when learning was complete. We compared the activity within neural circuits underpinning these behaviors between depressed patients and controls.

METHODS

We used a Go/No-Go task that assessed subjects' abilities in learning to emit or withhold actions to obtain monetary rewards or avoid losses. We derived motivation-relevant parameters of behavior (learning rate, Pavlovian bias, and motivational influence of gains and losses). After learning, participants performed the task during functional magnetic resonance imaging (fMRI). We compared neural activation during anticipation of action emission vs. action inhibition, and for actions performed to obtain rewards compared to actions that avoid losses.

RESULTS

Depressed patients showed a similar Pavlovian bias to controls and were equivalent in terms of withholding action to gain rewards and emitting action to avoid losses, behaviors that conflict with well-described Pavlovian tendencies to approach rewards and avoid losses. Patients were not impaired in overall performance or learning and showed no abnormal neural responses, for example in bilateral midbrain or striatum. We conclude that basic mechanisms subserving motivated learning are thus intact in moderate depression.

IMPLICATIONS

Therapeutically, the intact mechanisms identified here suggest that learning-based interventions may be particularly effective in encouraging recovery. Etiologically, our results suggest that the severe motivational deficits clinically observed in depression are likely to have complex origins, possibly related to an impairment in the representation of future states necessary for long-term planning.

Age-dependent Pavlovian biases influence motor decision-making

Motor decision-making is an essential component of everyday life which requires weighing potential rewards and punishments against the probability of successfully executing an action. To achieve this, humans rely on two key mechanisms; a flexible, instrumental, value-dependent process and a hardwired, Pavlovian, value-independent process. In economic decision-making, age-related decline in risk taking is explained by reduced Pavlovian biases that promote action toward reward. Although healthy ageing has also been associated with decreased risk-taking in motor decision-making, it is currently unknown whether this is a result of changes in Pavlovian biases, instrumental processes or a combination of both. Using a newly established approach-avoidance computational model together with a novel app-based motor decision-making task, we measured sensitivity to reward and punishment when participants (n = 26,532) made a ‘go/no-go’ motor gamble based on their perceived ability to execute a complex action. We show that motor decision-making can be better explained by a model with both instrumental and Pavlovian parameters, and reveal age-related changes across punishment- and reward-based instrumental and Pavlovian processes. However, the most striking effect of ageing was a decrease in Pavlovian attraction towards rewards, which was associated with a reduction in optimality of choice behaviour. In a subset of participants who also played an independent economic decision-making task (n = 17,220), we found similar decision-making tendencies for motor and economic domains across a majority of age groups. Pavlovian biases, therefore, play an important role in not only explaining motor decision-making behaviour but also the changes which occur through normal ageing. This provides a deeper understanding of the mechanisms which shape motor decision-making across the lifespan.

Decisions in everyday life often require weighing the probability of successfully executing an action (e.g., successfully crossing a street) against potential rewards and punishments. Although older individuals take fewer risks during such motor decision-making scenarios, the underlying mechanism remains unclear. Similar age-related changes in economic decision-making are explained by a decrease in Pavlovian attraction toward reward. However, despite the role of Pavlovian biases in linking action with reward and avoidance with punishment, their impact on motor decision-making is unclear. To address this, we developed a novel app-based motor decision-making task (n = 26,532). We found that motor decision-making was subject to Pavlovian influences. Although we found age-related changes for both punishment and reward-based decision-making processes, the most striking effect of ageing was a decrease in the facilitatory effect of Pavlovian attraction on action in pursuit of reward. Using data from an independent economic decision task in the same individuals (n = 17,220), we demonstrate similar decision-making tendencies for motor and economic domains across a majority of age groups. Hence, Pavlovian biases play an essential role in not only explaining motor decision-making behaviour but also the changes which occur through normal ageing.

Neural and computational processes underlying dynamic changes in self-esteem

Self-esteem is shaped by the appraisals we receive from others. Here, we characterize neural and computational mechanisms underlying this form of social influence. We introduce a computational model that captures fluctuations in self-esteem engendered by prediction errors that quantify the difference between expected and received social feedback. Using functional MRI, we show these social prediction errors correlate with activity in ventral striatum/subgenual anterior cingulate cortex, while updates in self-esteem resulting from these errors co-varied with activity in ventromedial prefrontal cortex (vmPFC). We linked computational parameters to psychiatric symptoms using canonical correlation analysis to identify an 'interpersonal vulnerability' dimension. Vulnerability modulated the expression of prediction error responses in anterior insula and insula-vmPFC connectivity during self-esteem updates. Our findings indicate that updating of self-evaluative beliefs relies on learning mechanisms akin to those used in learning about others. Enhanced insula-vmPFC connectivity during updating of those beliefs may represent a marker for psychiatric vulnerability.

Association of Neural and Emotional Impacts of Reward Prediction Errors With Major Depression

IMPORTANCE

Major depressive disorder (MDD) is associated with deficits in representing reward prediction errors (RPEs), which are the difference between experienced and predicted reward. Reward prediction errors underlie learning of values in reinforcement learning models, are represented by phasic dopamine release, and are known to affect momentary mood.

OBJECTIVE

To combine functional neuroimaging, computational modeling, and smartphone-based large-scale data collection to test, in the absence of learning-related concerns, the hypothesis that depression attenuates the impact of RPEs.

DESIGN, SETTING, AND PARTICIPANTS

Functional magnetic resonance imaging (fMRI) data were collected on 32 individuals with moderate MDD and 20 control participants who performed a probabilistic reward task. A risky decision task with repeated happiness ratings as a measure of momentary mood was also tested in the laboratory in 74 participants and with a smartphone-based platform in 1833 participants. The study was conducted from November 20, 2012, to February 17, 2015.

MAIN OUTCOMES AND MEASURES

Blood oxygen level-dependent activity was measured in ventral striatum, a dopamine target area known to represent RPEs. Momentary mood was measured during risky decision making.

RESULTS

Of the 52 fMRI participants (mean [SD] age, 34.0 [9.1] years), 30 (58%) were women and 32 had MDD. Of the 74 participants in the laboratory risky decision task (mean age, 34.2 [10.3] years), 44 (59%) were women and 54 had MDD. Of the smartphone group, 543 (30%) had a depression history and 1290 (70%) had no depression history; 918 (50%) were women, and 593 (32%) were younger than 30 years. Contrary to previous results in reinforcement learning tasks, individuals with moderate depression showed intact RPE signals in ventral striatum (z = 3.16; P = .002) that did not differ significantly from controls (z = 0.91; P = .36). Symptom severity correlated with baseline mood parameters in laboratory (ρ = -0.54; P < 1 × 10-6) and smartphone (ρ = -0.30; P < 1 × 10-39) data. However, participants with depression showed an intact association between RPEs and happiness in a computational model of momentary mood dynamics (z = 4.55; P < .001) that was not attenuated compared with controls (z = -0.42; P = .67).

CONCLUSIONS AND RELEVANCE

The neural and emotional impact of RPEs is intact in major depression. These results suggest that depression does not affect the expression of dopaminergic RPEs and that attenuated RPEs in previous reports may reflect downstream effects more closely related to aberrant behavior. The correlation between symptom severity and baseline mood parameters supports an association between depression and momentary mood fluctuations during cognitive tasks. These results demonstrate a potential for smartphones in large-scale computational phenotyping, which is a goal for computational psychiatry.

Approach-Induced Biases in Human Information Sampling

Information sampling is often biased towards seeking evidence that confirms one's prior beliefs. Despite such biases being a pervasive feature of human behavior, their underlying causes remain unclear. Many accounts of these biases appeal to limitations of human hypothesis testing and cognition, de facto evoking notions of bounded rationality, but neglect more basic aspects of behavioral control. Here, we investigated a potential role for Pavlovian approach in biasing which information humans will choose to sample. We collected a large novel dataset from 32,445 human subjects, making over 3 million decisions, who played a gambling task designed to measure the latent causes and extent of information-sampling biases. We identified three novel approach-related biases, formalized by comparing subject behavior to a dynamic programming model of optimal information gathering. These biases reflected the amount of information sampled ("positive evidence approach"), the selection of which information to sample ("sampling the favorite"), and the interaction between information sampling and subsequent choices ("rejecting unsampled options"). The prevalence of all three biases was related to a Pavlovian approach-avoid parameter quantified within an entirely independent economic decision task. Our large dataset also revealed that individual differences in the amount of information gathered are a stable trait across multiple gameplays and can be related to demographic measures, including age and educational attainment. As well as revealing limitations in cognitive processing, our findings suggest information sampling biases reflect the expression of primitive, yet potentially ecologically adaptive, behavioral repertoires. One such behavior is sampling from options that will eventually be chosen, even when other sources of information are more pertinent for guiding future action.

Acute stress selectively impairs learning to act

Stress interferes with instrumental learning. However, choice is also influenced by non-instrumental factors, most strikingly by biases arising from Pavlovian associations that facilitate action in pursuit of rewards and inaction in the face of punishment. Whether stress impacts on instrumental learning via these Pavlovian associations is unknown. Here, in a task where valence (reward or punishment) and action (go or no-go) were orthogonalised, we asked whether the impact of stress on learning was action or valence specific. We exposed 60 human participants either to stress (socially-evaluated cold pressor test) or a control condition (room temperature water). We contrasted two hypotheses: that stress would lead to a non-selective increase in the expression of Pavlovian biases; or that stress, as an aversive state, might specifically impact action production due to the Pavlovian linkage between inaction and aversive states. We found support for the second of these hypotheses. Stress specifically impaired learning to produce an action, irrespective of the valence of the outcome, an effect consistent with a Pavlovian linkage between punishment and inaction. This deficit in action-learning was also reflected in pupillary responses; stressed individuals showed attenuated pupillary responses to action, hinting at a noradrenergic contribution to impaired action-learning under stress.

Perimovement decrease of alpha/beta oscillations in the human nucleus accumbens

The present work clarifies how the nucleus accumbens contributes to action. This region is often assumed to influence behavior “off-line” by evaluating outcomes. Studying rare recordings of local field potentials from the human nucleus accumbens, we observe a perimovement decrease of alpha and beta oscillations in seven of eight individuals, a signal that, in the motor system, is directly related to action preparation. Our results support the idea of an online role of this region for imminent action.

The human nucleus accumbens is thought to play an important role in guiding future action selection via an evaluation of current action outcomes. Here we provide electrophysiological evidence for a more direct, i.e., online, role during action preparation. We recorded local field potentials from the nucleus accumbens in patients with epilepsy undergoing surgery for deep brain stimulation. We found a consistent decrease in the power of alpha/beta oscillations (10–30 Hz) before and around the time of movements. This perimovement alpha/beta desynchronization was observed in seven of eight patients and was present both before instructed movements in a serial reaction time task as well as before self-paced, deliberate choices in a decision making task. A similar beta decrease over sensorimotor cortex and in the subthalamic nucleus has been directly related to movement preparation and execution. Our results support the idea of a direct role of the human nucleus accumbens in action preparation and execution.

The social contingency of momentary subjective well-being

Although social comparison is a known determinant of overall life satisfaction, it is not clear how it affects moment-to-moment variation in subjective emotional state. Using a novel social decision task combined with computational modelling, we show that a participant's subjective emotional state reflects not only the impact of rewards they themselves receive, but also the rewards received by a social partner. Unequal outcomes, whether advantageous or disadvantageous, reduce average momentary happiness. Furthermore, the relative impacts of advantageous and disadvantageous inequality on momentary happiness at the individual level predict a subject's generosity in a separate dictator game. These findings demonstrate a powerful social influence upon subjective emotional state, where emotional reactivity to inequality is strongly predictive of altruism in an independent task domain.

Risk Taking for Potential Reward Decreases across the Lifespan

The extent to which aging affects decision-making is controversial. Given the critical financial decisions that older adults face (e.g., managing retirement funds), changes in risk preferences are of particular importance [1]. Although some studies have found that older individuals are more risk averse than younger ones [2, 3, 4], there are also conflicting results, and a recent meta-analysis found no evidence for a consistent change in risk taking across the lifespan [5]. There has as yet been little examination of one potential substrate for age-related changes in decision-making, namely age-related decline in dopamine, a neuromodulator associated with risk-taking behavior. Here, we characterized choice preferences in a smartphone-based experiment (n = 25,189) in which participants chose between safe and risky options. The number of risky options chosen in trials with potential gains but not potential losses decreased gradually over the lifespan, a finding with potentially important economic consequences for an aging population. Using a novel approach-avoidance computational model, we found that a Pavlovian attraction to potential reward declined with age. This Pavlovian bias has been linked to dopamine, suggesting that age-related decline in this neuromodulator could lead to the observed decrease in risk taking.

•

Aging reduced risk taking for potential gains but not potential losses

•

Computational models revealed that a Pavlovian influence of reward decreased with age

•

Age-related dopamine decline can explain the decrease in Pavlovian biases

Oxytocin Effect on Collective Decision Making: A Randomized Placebo Controlled Study

Collective decision making often benefits both the individuals and the group in a variety of contexts. However, for the group to be successful, individuals should be able to strike a balance between their level of competence and their influence on the collective decisions. The hormone oxytocin has been shown to promote trust, conformism and attention to social cues. We wondered if this hormone may increase participants’ (unwarranted) reliance on their partners’ opinion, resulting in a reduction in collective benefit by disturbing the balance between influence and competence. To test this hypothesis we employed a randomized double-blind placebo-controlled design in which male dyads self-administered intranasal oxytocin or placebo and then performed a visual search task together. Compared to placebo, collective benefit did not decrease under oxytocin. Using an exploratory time dependent analysis, we observed increase in collective benefit over time under oxytocin. Moreover, trial-by-trial analysis showed that under oxytocin the more competent member of each dyad was less likely to change his mind during disagreements, while the less competent member showed a greater willingness to change his mind and conform to the opinion of his more reliable partner. This role-dependent effect may be mediated by enhanced monitoring of own and other’s performance level under oxytocin. Such enhanced social learning could improve the balance between influence and competence and lead to efficient and beneficial collaboration.

The Impact of Menstrual Cycle Phase on Economic Choice and Rationality

It is well known that hormones affect both brain and behavior, but less is known about the extent to which hormones affect economic decision-making. Numerous studies demonstrate gender differences in attitudes to risk and loss in financial decision-making, often finding that women are more loss and risk averse than men. It is unclear what drives these effects and whether cyclically varying hormonal differences between men and women contribute to differences in economic preferences. We focus here on how economic rationality and preferences change as a function of menstrual cycle phase in women. We tested adherence to the Generalized Axiom of Revealed Preference (GARP), the standard test of economic rationality. If choices satisfy GARP then there exists a well-behaved utility function that the subject’s decisions maximize. We also examined whether risk attitudes and loss aversion change as a function of cycle phase. We found that, despite large fluctuations in hormone levels, women are as technically rational in their choice behavior as their male counterparts at all phases of the menstrual cycle. However, women are more likely to choose risky options that can lead to potential losses while ovulating; during ovulation women are less loss averse than men and therefore more economically rational than men in this regard. These findings may have market-level implications: ovulating women more effectively maximize expected value than do other groups.

The influence of contextual reward statistics on risk preference

Decision theories mandate that organisms should adjust their behaviour in the light of the contextual reward statistics. We tested this notion using a gambling choice task involving distinct contexts with different reward distributions. The best fitting model of subjects' behaviour indicated that the subjective values of options depended on several factors, including a baseline gambling propensity, a gambling preference dependent on reward amount, and a contextual reward adaptation factor. Combining this behavioural model with simultaneous functional magnetic resonance imaging we probed neural responses in three key regions linked to reward and value, namely ventral tegmental area/substantia nigra (VTA/SN), ventromedial prefrontal cortex (vmPFC) and ventral striatum (VST). We show that activity in the VTA/SN reflected contextual reward statistics to the extent that context affected behaviour, activity in the vmPFC represented a value difference between chosen and unchosen options while VST responses reflected a non-linear mapping between the actual objective rewards and their subjective value. The findings highlight a multifaceted basis for choice behaviour with distinct mappings between components of this behaviour and value sensitive brain regions.

Mood as Representation of Momentum

Experiences affect mood, which in turn affects subsequent experiences. Recent studies suggest two specific principles. First, mood depends on how recent reward outcomes differ from expectations. Second, mood biases the way we perceive outcomes (e.g., rewards), and this bias affects learning about those outcomes. We propose that this two-way interaction serves to mitigate inefficiencies in the application of reinforcement learning to real-world problems. Specifically, we propose that mood represents the overall momentum of recent outcomes, and its biasing influence on the perception of outcomes ‘corrects’ learning to account for environmental dependencies. We describe potential dysfunctions of this adaptive mechanism that might contribute to the symptoms of mood disorders.

With increasing use of computational models to understand human behavior, scientists have begun to model the dynamics of subjective states such as mood.

Recent data suggest that mood reflects the cumulative impact of differences between reward outcomes and expectations.

Behavioral and neural findings suggest that mood biases the perception of reward outcomes such that outcomes are perceived as better when one is in a good mood relative to when one is in a bad mood.

These two lines of research establish a bidirectional interaction between mood and reinforcement learning, which may play an important adaptive role in healthy behavior, and whose dysfunction might contribute to psychiatric disorders.

Proactive and Reactive Response Inhibition across the Lifespan

One expression of executive control involves proactive preparation for future events, and this contrasts with stimulus driven reactive control exerted in response to events. Here we describe findings from a response inhibition task, delivered using a smartphone-based platform, that allowed us to index proactive and reactive inhibitory self-control in a large community sample (n = 12,496). Change in stop-signal reaction time (SSRT) when participants are provided with advance information about an upcoming trial, compared to when they are not, provides a measure of proactive control while SSRT in the absence of advance information provides a measure of reactive control. Both forms of control rely on overlapping frontostriatal pathways known to deteriorate in healthy aging, an age-related decline that occurs at an accelerated rate in men compared to women. Here we ask whether these patterns of age-related decline are reflected in similar changes in proactive and reactive inhibitory control across the lifespan. As predicted, we observed a decline in reactive control with natural aging, with a greater rate of decline in men compared to women (~10 ms versus ~8 ms per decade of adult life). Surprisingly, the benefit of preparation, i.e. proactive control, did not change over the lifespan and women showed superior proactive control at all ages compared to men. Our results suggest that reactive and proactive inhibitory control partially rely on distinct neural substrates that are differentially sensitive to age-related change.

No unified reward prediction error in local field potentials from the human nucleus accumbens: evidence from epilepsy patients

Functional magnetic resonance imaging (fMRI), cyclic voltammetry, and single-unit electrophysiology studies suggest that signals measured in the nucleus accumbens (Nacc) during value-based decision making represent reward prediction errors (RPEs), the difference between actual and predicted rewards. Here, we studied the precise temporal and spectral pattern of reward-related signals in the human Nacc. We recorded local field potentials (LFPs) from the Nacc of six epilepsy patients during an economic decision-making task. On each trial, patients decided whether to accept or reject a gamble with equal probabilities of a monetary gain or loss. The behavior of four patients was consistent with choices being guided by value expectations. Expected value signals before outcome onset were observed in three of those patients, at varying latencies and with nonoverlapping spectral patterns. Signals after outcome onset were correlated with RPE regressors in all subjects. However, further analysis revealed that these signals were better explained as outcome valence rather than RPE signals, with gamble gains and losses differing in the power of beta oscillations and in evoked response amplitudes. Taken together, our results do not support the idea that postsynaptic potentials in the Nacc represent a RPE that unifies outcome magnitude and prior value expectation. We discuss the generalizability of our findings to healthy individuals and the relation of our results to measurements of RPE signals obtained from the Nacc with other methods.

Cortical drive of low-frequency oscillations in the human nucleus accumbens during action selection

The nucleus accumbens is thought to contribute to action selection by integrating behaviorally relevant information from multiple regions, including prefrontal cortex. Studies in rodents suggest that information flow to the nucleus accumbens may be regulated via task-dependent oscillatory coupling between regions. During instrumental behavior, local field potentials (LFP) in the rat nucleus accumbens and prefrontal cortex are coupled at delta frequencies (Gruber AJ, Hussain RJ, O'Donnell P. PLoS One 4: e5062, 2009), possibly mediating suppression of afferent input from other areas and thereby supporting cortical control (Calhoon GG, O'Donnell P. Neuron 78: 181–190, 2013). In this report, we demonstrate low-frequency cortico-accumbens coupling in humans, both at rest and during a decision-making task. We recorded LFP from the nucleus accumbens in six epilepsy patients who underwent implantation of deep brain stimulation electrodes. All patients showed significant coherence and phase-synchronization between LFP and surface EEG at delta and low theta frequencies. Although the direction of this coupling as indexed by Granger causality varied between subjects in the resting-state data, all patients showed a cortical drive of the nucleus accumbens during action selection in a decision-making task. In three patients this was accompanied by a significant coherence increase over baseline. Our results suggest that low-frequency cortico-accumbens coupling represents a highly conserved regulatory mechanism for action selection.

Dorsal striatum is necessary for stimulus-value but not action-value learning in humans

Several lines of evidence implicate the striatum in learning from experience on the basis of positive and negative feedback. However, the necessity of the striatum for such learning has been difficult to demonstrate in humans, because brain damage is rarely restricted to this structure. Here we test a rare individual with widespread bilateral damage restricted to the dorsal striatum. His performance was impaired and not significantly different from chance on several classic learning tasks, consistent with current theories regarding the role of the striatum. However, he also exhibited remarkably intact performance on a different subset of learning paradigms. The tasks he could perform can all be solved by learning the value of actions, while those he could not perform can only be solved by learning the value of stimuli. Although dorsal striatum is often thought to play a specific role in action-value learning, we find surprisingly that dorsal striatum is necessary for stimulus-value but not action-value learning in humans.

MEASURING BELIEFS AND REWARDS: A NEUROECONOMIC APPROACH

The neurotransmitter dopamine is central to the emerging discipline of neuroeconomics; it is hypothesized to encode the difference between expected and realized rewards and thereby to mediate belief formation and choice. We develop the first formal test of this theory of dopaminergic function, based on a recent axiomatization by Caplin and Dean [2008A]. These tests are satisfied by neural activity in the nucleus accumbens, an area rich in dopamine receptors. We find evidence for separate positive and negative reward prediction error signals, suggesting that behavioral asymmetries in response to losses and gains may parallel asymmetries in nucleus accumbens activity.

The right tool for the job: what strategies do wild New Caledonian crows use?

New Caledonian crows Corvus moneduloides (NC crows) display sophisticated tool manufacture in the wild, but the cognitive strategy underlying these skills is poorly understood. Here, we investigate what strategy two free-living NC crows used in response to a tool-length task. The crows manufactured tools to extract food from vertical holes of different depths. The first tools they made in visits were of a similar length regardless of the hole depth. The typical length was usually too short to extract food from the deep holes, which ruled out a strategy of immediate causal inference on the first attempt in a trial. When the first tool failed, the crows made second tools significantly longer than the unsuccessful first tools. There was no evidence that the crows made the lengths of first tools to directly match hole depth. We argue that NC crows may generally use a two-stage heuristic strategy to solve tool problems and that performance on the first attempt in a trial is not necessarily the 'gold standard' for assessing folk physics.

Melanesian and Asian Origins of Polynesians: mtDNA and Y Chromosome Gradients Across the Pacific

The human settlement of the Pacific Islands represents one of the most recent major migration events of mankind. Polynesians originated in Asia according to linguistic evidence or in Melanesia according to archaeological evidence. To shed light on the genetic origins of Polynesians, we investigated over 400 Polynesians from 8 island groups, in comparison with over 900 individuals from potential parental populations of Melanesia, Southeast and East Asia, and Australia, by means of Y chromosome (NRY) and mitochondrial DNA (mtDNA) markers. Overall, we classified 94.1% of Polynesian Y chromosomes and 99.8% of Polynesian mtDNAs as of either Melanesian (NRY-DNA: 65.8%, mtDNA: 6%) or Asian (NRY-DNA: 28.3%, mtDNA: 93.8%) origin, suggesting a dual genetic origin of Polynesians in agreement with the "Slow Boat" hypothesis. Our data suggest a pronounced admixture bias in Polynesians toward more Melanesian men than women, perhaps as a result of matrilocal residence in the ancestral Polynesian society. Although dating methods are consistent with somewhat similar entries of NRY/mtDNA haplogroups into Polynesia, haplotype sharing suggests an earlier appearance of Melanesian haplogroups than those from Asia. Surprisingly, we identified gradients in the frequency distribution of some NRY/mtDNA haplogroups across Polynesia and a gradual west-to-east decrease of overall NRY/mtDNA diversity, not only providing evidence for a west-to-east direction of Polynesian settlements but also suggesting that Pacific voyaging was regular rather than haphazard. We also demonstrate that Fiji played a pivotal role in the history of Polynesia: humans probably first migrated to Fiji, and subsequent settlement of Polynesia probably came from Fiji.