Momentary subjective well-being depends on learning and not reward

Longitudinal Dynamic Relationships Between Videogame Use and Symptoms of Gaming Disorder and Depression Among Chinese Children and Adolescents

Although previous studies have shown a close relationship between gaming disorder and depressive symptoms, few have measured normal videogame use, symptoms of gaming disorder, and depressive symptoms concurrently. The longitudinal dynamics between these variables remain unclear. This study used two demographic cohorts to examine the longitudinal relationship between gaming and depressive symptoms: children (n = 1513, 46.9% girls, Mage ± SD = 9.63 ± 0.58 years) and adolescents (n = 1757, 48.5% girls, Mage ± SD = 12.55 ± 0.70 years). Random intercept cross-lagged panel models (RI-CLPMs) were employed to distinguish between within- and between-person levels of gaming and depressive symptoms. The RI-CLPM results showed a stable link between symptoms of gaming disorder and depression at the between-person level for both children and adolescents. At the within-person level, among children, depressive symptoms positively predicted subsequent gaming disorder symptoms, but gaming disorder symptoms were not a significant predictor of depressive symptoms at this level. Among adolescents, there was no significant cross-lagged effect between symptoms of gaming disorder and depression at the within-person level. Additionally, there was no significant cross-lagged effect between normal videogame use and depressive symptoms in either cohort. These results highlight the different effects of normal videogame use and gaming disorder symptoms associated with depressive symptoms. The different effects on children and adolescents underscore the importance of considering the different developmental stages in the study of gaming and mental health outcomes.

Adapting to uncertainty: The role of anxiety and fear of negative evaluation in learning in social and non-social contexts

BACKGROUND

Navigating social situations can be challenging due to uncertainty surrounding the intentions and strategies of others, which remain hidden and subject to change. Prior research suggests that individuals with anxiety-related symptoms struggle to adapt their learning in uncertain, non-social environments. Anxiety-prone individuals encounter challenges in social functioning, yet research on learning under uncertainty in social contexts is limited. In this preregistered study, we investigated whether individuals with higher levels of trait anxiety and fear of negative evaluation encounter difficulties in adjusting their learning rates in social contexts with stable or volatile outcome contingencies.

METHODS

We implemented a modified trust game (N = 190), where participants either retained or lost their investments based on their interactions with two players in volatile or stable environments. Participants also completed a matching non-social control task involving interactions with slot machines.

RESULTS

Results from computational modeling revealed significantly higher learning rates in social compared to non-social settings. Trait anxiety did not affect the adaptability of learning rates. Individuals with heightened fear of negative evaluation were more sensitive to social compared to non-social outcomes, as reflected in their stay/switch behavior and, though less conclusive, in their learning rates.

LIMITATIONS

While transdiagnostic and dimensional approaches are important for investigating disturbed social functioning, the inclusion of clinical samples in future studies may contribute to a broader generalization of these findings regarding behavioral variances in uncertain social environments.

CONCLUSIONS

Individuals with increased fear of negative evaluation may demonstrate heightened sensitivity to learning in uncertain social contexts. This leads to heightened responsiveness to recent outcomes in their interactions with others, potentially contributing to their problems in social functioning.

Ethological computational psychiatry: Challenges and opportunities

Studying the intricacies of individual subjects' moods and cognitive processing over extended periods of time presents a formidable challenge in medicine. While much of systems neuroscience appropriately focuses on the link between neural circuit functions and well-constrained behaviors over short timescales (e.g., trials, hours), many mental health conditions involve complex interactions of mood and cognition that are non-stationary across behavioral contexts and evolve over extended timescales. Here, we discuss opportunities, challenges, and possible future directions in computational psychiatry to quantify non-stationary continuously monitored behaviors. We suggest that this exploratory effort may contribute to a more precision-based approach to treating mental disorders and facilitate a more robust reverse translation across animal species. We conclude with ethical considerations for any field that aims to bridge artificial intelligence and patient monitoring.

Hidden Reward: Affect and Its Prediction Errors as Windows Into Subjective Value

Scientists increasingly apply concepts from reinforcement learning to affect, but which concepts should apply? And what can their application reveal that we cannot know from directly observable states? An important reinforcement learning concept is the difference between reward expectations and outcomes. Such reward prediction errors have become foundational to research on adaptive behavior in humans, animals, and machines. Owing to historical focus on animal models and observable reward (e.g., food or money), however, relatively little attention has been paid to the fact that humans can additionally report correspondingly expected and experienced affect (e.g., feelings). Reflecting a broader "rise of affectivism," attention has started to shift, revealing explanatory power of expected and experienced feelings-including prediction errors-above and beyond observable reward. We propose that applying concepts from reinforcement learning to affect holds promise for elucidating subjective value. Simultaneously, we urge scientists to test-rather than inherit-concepts that may not apply directly.

A cognitive-computational account of mood swings in adolescence

Teenagers have a reputation for being fickle, in both their choices and their moods. This variability may help adolescents as they begin to independently navigate novel environments. Recently, however, adolescent moodiness has also been linked to psychopathology. Here, we consider adolescents' mood swings from a novel computational perspective, grounded in reinforcement learning (RL). This model proposes that mood is determined by surprises about outcomes in the environment, and how much we learn from these surprises. It additionally suggests that mood biases learning and choice in a bidirectional manner. Integrating independent lines of research, we sketch a cognitive-computational account of how adolescents' mood, learning, and choice dynamics influence each other, with implications for normative and psychopathological development.

Bridge the gap caused by public health crises: medical humanization and communication skills build a psychological bond that satisfies patients

BACKGROUND

Patient satisfaction is an important outcome domain of patient-centered care. Medical humanization follows the patient-centered principle and provides a more holistic view to treat patients. The COVID-19 pandemic posed significant barriers to maintaining medical humanization. However, empirical study on the relationship between medical humanization and patient satisfaction is clearly absent.

OBJECTIVES

We examined the mediation effects of communication on the relationship between medical humanization and patient satisfaction when faced with a huge public health crisis like the COVID-19 pandemic, and the moderation effect of medical institutional trust on the mediation models.

METHODS

A cross-sectional survey study was performed. A final sample size of 1445 patients was surveyed on medical humanization, communication, patient satisfaction and medical institutional trust.

RESULTS

All correlations were significantly positive across the main variables (r = 0.35-0.67, p < 0.001 for all) except for medical institutional trust, which was negatively correlated with the medical humanization (r=-0.14, p < 0.001). Moderated mediation analysis showed that the indirect effect of medical humanization on patient satisfaction through communication was significant (b = 0.22, 95% CI: 0.18 ~ 0.25). Medical institutional trust significantly moderated the effect of medical humanization on patient satisfaction (b=-0.09, p < 0.001) and the effect of medical humanization on communication (b= -0.14, p < 0.001).

CONCLUSION

Medical humanization positively influence patient satisfaction, communication mediated the association between medical humanization and patient satisfaction, and medical institutional trust negatively moderated the effects of medical humanization on patient satisfaction and communication. These findings suggest that humanistic communication contributes to patient satisfaction in the face of a huge public health crisis, and patients' evaluation of satisfaction is also regulated by rational cognition.

Pramipexole for the Treatment of Depression: Efficacy and Mechanisms

Dopaminergic mechanisms are a plausible treatment target for patients with clinical depression but are relatively underexplored in conventional antidepressant medications. There is continuing interest in the potential antidepressant effects of the dopamine receptor agonist, pramipexole, with data from both case series and controlled trials indicating that this agent may produce benefit for patients with difficult-to-treat depression. Pramipexole's therapeutic utility in depression is likely to be expressed through alterations in reward mechanisms which are strongly influenced by dopamine pathways and are known to function abnormally in depressed patients. Our work in healthy participants using brain imaging in conjunction with computational modelling suggests that repeated pramipexole facilitates reward learning by inhibiting value decay. This mechanism needs to be confirmed in larger clinical trials in depressed patients. Such studies will also allow assessment of whether baseline performance in reward learning in depression predicts therapeutic response to pramipexole treatment.

Reward and punishment learning deficits among bipolar disorder subtypes

BACKGROUND

Reward sensitivity is an essential dimension related to mood fluctuations in bipolar disorder (BD), but there is currently a debate around hypersensitivity or hyposensitivity hypotheses to reward in BD during remission, probably related to a heterogeneous population within the BD spectrum and a lack of reward bias evaluation. Here, we examine reward maximization vs. punishment avoidance learning within the BD spectrum during remission.

METHODS

Patients with BD-I (n = 45), BD-II (n = 34) and matched (n = 30) healthy controls (HC) were included. They performed an instrumental learning task designed to dissociate reward-based from punishment-based reinforcement learning. Computational modeling was used to identify the mechanisms underlying reinforcement learning performances.

RESULTS

Behavioral results showed a significant reward learning deficit across BD subtypes compared to HC, captured at the computational level by a lower sensitivity to rewards compared to punishments in both BD subtypes. Computational modeling also revealed a higher choice randomness in BD-II compared to BD-I that reflected a tendency of BD-I to perform better during punishment avoidance learning than BD-II.

LIMITATIONS

Our patients were not naive to antipsychotic treatment and were not euthymic (but in syndromic remission) according to the International Society for Bipolar Disorder definition.

CONCLUSIONS

Our results are consistent with the reward hyposensitivity theory in BD. Computational modeling suggests distinct underlying mechanisms that produce similar observable behaviors, making it a useful tool for distinguishing how symptoms interact in BD versus other disorders. In the long run, a better understanding of these processes could contribute to better prevention and management of BD.

Sensitivity to intrinsic rewards is domain general and related to mental health

Humans frequently engage in intrinsically rewarding activities (for example, consuming art, reading). Despite such activities seeming diverse, we show that sensitivity to intrinsic rewards is domain general and associated with mental health. In this cross-sectional study, participants online (N = 483) were presented with putative visual, cognitive and social intrinsic rewards as well as monetary rewards and neutral stimuli. All rewards elicited positive feelings (were 'liked'), generated consummatory behaviour (were 'wanted') and increased the likelihood of the action leading to them (were 'reinforcing'). Factor analysis revealed that ~40% of response variance across stimuli was explained by a general sensitivity to all rewards, but not to neutral stimuli. Affective aspects of mental health were associated with sensitivity to intrinsic, but not monetary, rewards. These results may help explain thriving and suffering: individuals with high reward sensitivity will engage in a variety of intrinsically rewarding activities, eventually finding those they excel at, whereas low sensitivity individuals will not.

Reliability of gamified reinforcement learning in densely sampled longitudinal assessments

Reinforcement learning is a core facet of motivation and alterations have been associated with various mental disorders. To build better models of individual learning, repeated measurement of value-based decision-making is crucial. However, the focus on lab-based assessment of reward learning has limited the number of measurements and the test-retest reliability of many decision-related parameters is therefore unknown. In this paper, we present an open-source cross-platform application Influenca that provides a novel reward learning task complemented by ecological momentary assessment (EMA) of current mental and physiological states for repeated assessment over weeks. In this task, players have to identify the most effective medication by integrating reward values with changing probabilities to win (according to random Gaussian walks). Participants can complete up to 31 runs with 150 trials each. To encourage replay, in-game screens provide feedback on the progress. Using an initial validation sample of 384 players (9729 runs), we found that reinforcement learning parameters such as the learning rate and reward sensitivity show poor to fair intra-class correlations (ICC: 0.22-0.53), indicating substantial within- and between-subject variance. Notably, items assessing the psychological state showed comparable ICCs as reinforcement learning parameters. To conclude, our innovative and openly customizable app framework provides a gamified task that optimizes repeated assessments of reward learning to better quantify intra- and inter-individual differences in value-based decision-making over time.

A mesocorticolimbic signature of pleasure in the human brain

Pleasure is a fundamental driver of human behaviour, yet its neural basis remains largely unknown. Rodent studies highlight opioidergic neural circuits connecting the nucleus accumbens, ventral pallidum, insula and orbitofrontal cortex as critical for the initiation and regulation of pleasure, and human neuroimaging studies exhibit some translational parity. However, whether activation in these regions conveys a generalizable representation of pleasure regulated by opioidergic mechanisms remains unclear. Here we use pattern recognition techniques to develop a human functional magnetic resonance imaging signature of mesocorticolimbic activity unique to states of pleasure. In independent validation tests, this signature is sensitive to pleasant tastes and affect evoked by humour. The signature is spatially co-extensive with mu-opioid receptor gene expression, and its response is attenuated by the opioid antagonist naloxone. These findings provide evidence for a basis of pleasure in humans that is distributed across brain systems.

Neurocomputational mechanisms underlying fear-biased adaptation learning in changing environments

Humans are able to adapt to the fast-changing world by estimating statistical regularities of the environment. Although fear can profoundly impact adaptive behaviors, the computational and neural mechanisms underlying this phenomenon remain elusive. Here, we conducted a behavioral experiment (n = 21) and a functional magnetic resonance imaging experiment (n = 37) with a novel cue-biased adaptation learning task, during which we simultaneously manipulated emotional valence (fearful/neutral expressions of the cue) and environmental volatility (frequent/infrequent reversals of reward probabilities). Across 2 experiments, computational modeling consistently revealed a higher learning rate for the environment with frequent versus infrequent reversals following neutral cues. In contrast, this flexible adjustment was absent in the environment with fearful cues, suggesting a suppressive role of fear in adaptation to environmental volatility. This suppressive effect was underpinned by activity of the ventral striatum, hippocampus, and dorsal anterior cingulate cortex (dACC) as well as increased functional connectivity between the dACC and temporal-parietal junction (TPJ) for fear with environmental volatility. Dynamic causal modeling identified that the driving effect was located in the TPJ and was associated with dACC activation, suggesting that the suppression of fear on adaptive behaviors occurs at the early stage of bottom-up processing. These findings provide a neuro-computational account of how fear interferes with adaptation to volatility during dynamic environments.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Computational models of subjective feelings in psychiatry

Research in computational psychiatry is dominated by models of behavior. Subjective experience during behavioral tasks is not well understood, even though it should be relevant to understanding the symptoms of psychiatric disorders. Here, we bridge this gap and review recent progress in computational models for subjective feelings. For example, happiness reflects not how well people are doing, but whether they are doing better than expected. This dependence on recent reward prediction errors is intact in major depression, although depressive symptoms lower happiness during tasks. Uncertainty predicts subjective feelings of stress in volatile environments. Social prediction errors influence feelings of self-worth more in individuals with low self-esteem despite a reduced willingness to change beliefs due to social feedback. Measuring affective state during behavioral tasks provides a tool for understanding psychiatric symptoms that can be dissociable from behavior. When smartphone tasks are collected longitudinally, subjective feelings provide a potential means to bridge the gap between lab-based behavioral tasks and real-life behavior, emotion, and psychiatric symptoms.

Emotions as computations

Emotions ubiquitously impact action, learning, and perception, yet their essence and role remain widely debated. Computational accounts of emotion aspire to answer these questions with greater conceptual precision informed by normative principles and neurobiological data. We examine recent progress in this regard and find that emotions may implement three classes of computations, which serve to evaluate states, actions, and uncertain prospects. For each of these, we use the formalism of reinforcement learning to offer a new formulation that better accounts for existing evidence. We then consider how these distinct computations may map onto distinct emotions and moods. Integrating extensive research on the causes and consequences of different emotions suggests a parsimonious one-to-one mapping, according to which emotions are integral to how we evaluate outcomes (pleasure & pain), learn to predict them (happiness & sadness), use them to inform our (frustration & content) and others' (anger & gratitude) actions, and plan in order to realize (desire & hope) or avoid (fear & anxiety) uncertain outcomes.

Comparing gratitude and pride: evidence from brain and behavior

Gratitude and pride are both positive emotions. Yet gratitude motivates people to help others and build up relationships, whereas pride motivates people to pursue achievements and build on self-esteem. Although these social outcomes are crucial for humans to be evolutionarily adaptive, no study so far has systematically compared gratitude and pride to understand why and how they can motivate humans differently. In this review, we compared gratitude and pride from their etymologies, cognitive prerequisites, motivational functions, and brain regions involved. By integrating the evidence from brain and behavior, we suggest that gratitude and pride share a common reward basis, yet gratitude is more related to theory of mind, while pride is more related to self-referential processing. Moreover, we proposed a cognitive neuroscientific model to explain the dynamics in gratitude and pride under a reinforcement learning framework.

The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.

Even in favorable circumstances, we often find it hard to remain happy with what we have. One might enjoy a newly bought car for a season, but over time it brings fewer positive feelings and one eventually begins dreaming of the next rewarding thing to pursue. Here, we present a series of computational simulations that suggest these presumable “flaws” might play an important role in promoting adaptive behavior. We explore the value of prior expectations and relative comparisons as a useful reward signal and find that across a wide range of environments, these features help an agent learn faster and adapt better to changes in the environment. Our simulations also highlight scenarios when these relative features can be harmful to decision-making and happiness. Together, our results help explain why we have the propensity to keep wanting more, even if it contributes to depression, materialism, and overconsumption.

Computational models of adaptive behavior and prefrontal cortex

The real world is uncertain, and while ever changing, it constantly presents itself in terms of new sets of behavioral options. To attain the flexibility required to tackle these challenges successfully, most mammalian brains are equipped with certain computational abilities that rely on the prefrontal cortex (PFC). By examining learning in terms of internal models associating stimuli, actions, and outcomes, we argue here that adaptive behavior relies on specific interactions between multiple systems including: (1) selective models learning stimulus-action associations through rewards; (2) predictive models learning stimulus- and/or action-outcome associations through statistical inferences anticipating behavioral outcomes; and (3) contextual models learning external cues associated with latent states of the environment. Critically, the PFC combines these internal models by forming task sets to drive behavior and, moreover, constantly evaluates the reliability of actor task sets in predicting external contingencies to switch between task sets or create new ones. We review different models of adaptive behavior to demonstrate how their components map onto this unifying framework and specific PFC regions. Finally, we discuss how our framework may help to better understand the neural computations and the cognitive architecture of PFC regions guiding adaptive behavior.

A Neurocomputational Model for Intrinsic Reward

Standard economic indicators provide an incomplete picture of what we value both as individuals and as a society. Furthermore, canonical macroeconomic measures, such as GDP, do not account for non-market activities (e.g., cooking, childcare) that nevertheless impact well-being. Here, we introduce a computational tool that measures the affective value of experiences (e.g., playing a musical instrument without errors). We go on to validate this tool with neural data, using fMRI to measure neural activity in male and female human subjects performing a reinforcement learning task that incorporated periodic ratings of subjective affective state. Learning performance determined level of payment (i.e., extrinsic reward). Crucially, the task also incorporated a skilled performance component (i.e., intrinsic reward) which did not influence payment. Both extrinsic and intrinsic rewards influenced affective dynamics, and their relative influence could be captured in our computational model. Individuals for whom intrinsic rewards had a greater influence on affective state than extrinsic rewards had greater ventromedial prefrontal cortex (vmPFC) activity for intrinsic than extrinsic rewards. Thus, we show that computational modeling of affective dynamics can index the subjective value of intrinsic relative to extrinsic rewards, a “computational hedonometer” that reflects both behavior and neural activity that quantifies the affective value of experience.

SIGNIFICANCE STATEMENT Traditional economic indicators are increasingly recognized to provide an incomplete picture of what we value as a society. Standard economic approaches struggle to accurately assign values to non-market activities that nevertheless may be intrinsically rewarding, prompting a need for new tools to measure what really matters to individuals. Using a combination of neuroimaging and computational modeling, we show that despite their lack of instrumental value, intrinsic rewards influence subjective affective state and ventromedial prefrontal cortex (vmPFC) activity. The relative degree to which extrinsic and intrinsic rewards influence affective state is predictive of their relative impacts on neural activity, confirming the utility of our approach for measuring the affective value of experiences and other non-market activities in individuals.

Timescales of Cognition in the Brain

We live in a world that changes on many timescales. To learn and make decisions appropriately, the human brain has evolved to integrate various types of information, such as sensory evidence and reward feedback, on multiple timescales. This is reflected in cortical hierarchies of timescales consisting of heterogeneous neuronal activities and expression of genes related to neurotransmitters critical for learning. We review the recent findings on how timescales of sensory and reward integration are affected by the temporal properties of sensory and reward signals in the environment. Despite existing evidence linking behavioral and neuronal timescales, future studies must examine how neural computations at multiple timescales are adjusted and combined to influence behavior flexibly.

Additively Combining Utilities and Beliefs: Research Gaps and Algorithmic Developments

Value-based decision making in complex environments, such as those with uncertain and volatile mapping of reward probabilities onto options, may engender computational strategies that are not necessarily optimal in terms of normative frameworks but may ensure effective learning and behavioral flexibility in conditions of limited neural computational resources. In this article, we review a suboptimal strategy - additively combining reward magnitude and reward probability attributes of options for value-based decision making. In addition, we present computational intricacies of a recently developed model (named MIX model) representing an algorithmic implementation of the additive strategy in sequential decision-making with two options. We also discuss its opportunities; and conceptual, inferential, and generalization issues. Furthermore, we suggest future studies that will reveal the potential and serve the further development of the MIX model as a general model of value-based choice making.

Smartphones and the Neuroscience of Mental Health

Improvements in understanding the neurobiological basis of mental illness have unfortunately not translated into major advances in treatment. At this point, it is clear that psychiatric disorders are exceedingly complex and that, in order to account for and leverage this complexity, we need to collect longitudinal data sets from much larger and more diverse samples than is practical using traditional methods. We discuss how smartphone-based research methods have the potential to dramatically advance our understanding of the neuroscience of mental health. This, we expect, will take the form of complementing lab-based hard neuroscience research with dense sampling of cognitive tests, clinical questionnaires, passive data from smartphone sensors, and experience-sampling data as people go about their daily lives. Theory- and data-driven approaches can help make sense of these rich data sets, and the combination of computational tools and the big data that smartphones make possible has great potential value for researchers wishing to understand how aspects of brain function give rise to, or emerge from, states of mental health and illness.