Intelligence moderates reinforcement learning: a mini-review of the neural evidence

The Effect of Brief Stair-Climbing on Divergent and Convergent Thinking

Recent studies show that even a brief bout of aerobic exercise may enhance creative thinking. However, few studies have investigated the effect of exercise conducted in natural settings. Here, in a crossover randomized controlled trial, we investigated the effect of a common daily activity, stair-climbing, on creative thinking. As experimental intervention, subjects were asked to walk downstairs from the fourth to the first floor and back at their usual pace. As control intervention, they walked the same path but using the elevator instead. Compared to using the elevator, stair-climbing enhanced subsequent divergent but not convergent thinking in that it increased originality on the Alternate Use Test (d = 0.486). Subjects on average generated 61% more original uses after stair-climbing. This is the first study to investigate the effect of stair-climbing on creative thinking. Our findings suggest that stair-climbing may be a useful strategy for enhancing divergent thinking in everyday life.

Sex difference in the weighting of expected uncertainty under chronic stress

The neurobiological literature implicates chronic stress induced decision-making deficits as a major contributor to depression and anxiety. Given that females are twice as likely to suffer from these disorders, we hypothesized the existence of sex difference in the effects of chronic stress on decision-making. Here employing a decision-making paradigm that relies on reinforcement learning of probabilistic predictive relationships, we show female volunteers with a high level of perceived stress in the past month are more likely to make suboptimal choices than males. Computational characterizations of this sex difference suggest that while under high stress, females and males differ in their weighting but not learning of the expected uncertainty in the predictive relationships. These findings provide a mechanistic account of the sex difference in decision-making under chronic stress and may have important implications for the epidemiology of sex difference in depression and anxiety.

Reinforcement learning as an intermediate phenotype in psychosis? Deficits sensitive to illness stage but not associated with polygenic risk of schizophrenia in the general population

BACKGROUND

Schizophrenia is a complex disorder in which the causal relations between risk genes and observed clinical symptoms are not well understood and the explanatory gap is too wide to be clarified without considering an intermediary level. Thus, we aimed to test the hypothesis of a pathway from molecular polygenic influence to clinical presentation occurring via deficits in reinforcement learning.

METHODS

We administered a reinforcement learning task (Go/NoGo) that measures reinforcement learning and the effect of Pavlovian bias on decision making. We modelled the behavioural data with a hierarchical Bayesian approach (hBayesDM) to decompose task performance into its underlying learning mechanisms. Study 1 included controls (n = 29, F|M = 0.81), At Risk Mental State for psychosis (ARMS, n = 23, F|M = 0.35) and FEP (First-episode psychosis, n = 26, F|M = 0.18). Study 2 included healthy adolescents (n = 735, F|M = 1.06), 390 of whom had their polygenic risk scores for schizophrenia (PRSs) calculated.

RESULTS

Patients with FEP showed significant impairments in overriding Pavlovian conflict, a lower learning rate and a lower sensitivity to both reward and punishment. Less widespread deficits were observed in ARMS. PRSs did not significantly predict performance on the task in the general population, which only partially correlated with measures of psychopathology.

CONCLUSIONS

Reinforcement learning deficits are observed in first episode psychosis and, to some extent, in those at clinical risk for psychosis, and were not predicted by molecular genetic risk for schizophrenia in healthy individuals. The study does not support the role of reinforcement learning as an intermediate phenotype in psychosis.

Probabilistic Category Learning and Striatal Functional Activation in Psychosis Risk

BACKGROUND

Psychosis risk is associated with striatal dysfunction, including a previous behavioral study that found that psychosis risk is associated with impaired performance on a probabilistic category learning task (PCLT; ie, the Weather Prediction Task), a task strongly associated with striatal activation. The current study examined whether psychosis risk based on symptom levels was associated with both poor behavioral performance and task-related physiological dysfunction in specific regions of the striatum while performing the PCLT.

METHODS

There were 2 groups of participants: psychosis risk (n = 21) who had both (a) extreme levels of self-reported psychotic-like beliefs and experiences and (b) interview-rated current attenuated psychotic symptoms (APS); and a comparison group (n = 20) who had average levels of self-reported psychotic-like beliefs and experiences. Participants completed the PCLT during fMRI scanning.

RESULTS

The current research replicated previous work finding behavioral PCLT deficits at the end of the task in psychosis risk. Furthermore, as expected, the psychosis risk group exhibited decreased striatal activation on the task, especially in the associative striatum. The psychosis risk group also displayed decreased activation in a range of cortical regions connected to the associative striatum. In contrast, the psychosis risk group exhibited greater activation predominantly in cortical regions not connected to the associative striatum.

CONCLUSIONS

Psychosis risk was associated with both behavioral and striatal dysfunction during performance on the PCLT, suggesting that behavioral and imaging measures using this task could be a marker for psychosis risk.

Predictive Movements and Human Reinforcement Learning of Sequential Action

Sequential action makes up the bulk of human daily activity, and yet much remains unknown about how people learn such actions. In one motor learning paradigm, the serial reaction time (SRT) task, people are taught a consistent sequence of button presses by cueing them with the next target response. However, the SRT task only records keypress response times to a cued target, and thus it cannot reveal the full time‐course of motion, including predictive movements. This paper describes a mouse movement trajectory SRT task in which the cursor must be moved to a cued location. We replicated keypress SRT results, but also found that predictive movement—before the next cue appears—increased during the experiment. Moreover, trajectory analyses revealed that people developed a centering strategy under uncertainty. In a second experiment, we made prediction explicit, no longer cueing targets. Thus, participants had to explore the response alternatives and learn via reinforcement, receiving rewards and penalties for correct and incorrect actions, respectively. Participants were not told whether the sequence of stimuli was deterministic, nor if it would repeat, nor how long it was. Given the difficulty of the task, it is unsurprising that some learners performed poorly. However, many learners performed remarkably well, and some acquired the full 10‐item sequence within 10 repetitions. Comparing the high‐ and low‐performers’ detailed results in this reinforcement learning (RL) task with the first experiment's cued trajectory SRT task, we found similarities between the two tasks, suggesting that the effects in Experiment 1 are due to predictive, rather than reactive processes. Finally, we found that two standard model‐free reinforcement learning models fit the high‐performing participants, while the four low‐performing participants provide better fit with a simple negative recency bias model.

Mental Imagery Training Increases Wanting of Rewards and Reward Sensitivity and Reduces Depressive Symptoms

High reward sensitivity and wanting of rewarding stimuli help to identify and motivate repetition of pleasant activities. This behavioral activation is thought to increase positive emotions. Therefore, both mechanisms are highly relevant for resilience against depressive symptoms. Yet, these mechanisms have not been targeted by psychotherapeutic interventions. In the present study, we tested a mental imagery training comprising eight 10-minute sessions every second day and delivered via the Internet to healthy volunteers (N = 30, 21 female, mean age of 23.8 years, Caucasian) who were preselected for low reward sensitivity. Participants were paired according to age, sex, reward sensitivity, and mental imagery ability. Then, members of each pair were randomly assigned to either the intervention or wait condition. Ratings of wanting and response bias toward probabilistic reward cues (Probabilistic Reward Task) served as primary outcomes. We further tested whether training effects extended to approach behavior (Approach Avoidance Task) and depressive symptoms (Beck Depression Inventory). The intervention led to an increase in wanting (p < .001, η²_p= .45) and reward sensitivity (p = .004, η²_p= .27). Further, the training group displayed faster approach toward positive edibles and activities (p = .025, η²_p= .18) and reductions in depressive symptoms (p = .028, η²_p= .16). Results extend existing literature by showing that mental imagery training can increase wanting of rewarding stimuli and reward sensitivity. Further, the training appears to reduce depressive symptoms and thus may foster the successful implementation of exsiting treatments for depression such as behavioral activation and could also increase resilience against depressive symptoms.