A Computational Analysis of Abnormal Belief Updating Processes and Their Association With Psychotic Experiences and Childhood Trauma in a UK Birth Cohort

BACKGROUND

Psychotic experiences emerge from abnormalities in perception and belief formation and occur more commonly in those experiencing childhood trauma. However, which precise aspects of belief formation are atypical in psychosis is not well understood. We used a computational modeling approach to characterize belief updating in young adults in the general population, examine their relationship with psychotic outcomes and trauma, and determine the extent to which they mediate the trauma-psychosis relationship.

METHODS

We used data from 3360 individuals from the Avon Longitudinal Study of Parents and Children birth cohort who completed assessments for psychotic outcomes, depression, anxiety, and two belief updating tasks at age 24 and had data available on traumatic events assessed from birth to late adolescence. Unadjusted and adjusted regression and counterfactual mediation methods were used for the analyses.

RESULTS

Basic behavioral measures of belief updating (draws-to-decision and disconfirmatory updating) were not associated with psychotic experiences. However, computational modeling revealed an association between increased decision noise with both psychotic experiences and trauma exposure, although <3% of the trauma-psychotic experience association was mediated by decision noise. Belief updating measures were also associated with intelligence and sociodemographic characteristics, confounding most of the associations with psychotic experiences. There was little evidence that belief updating parameters were differentially associated with delusions compared with hallucinations or that they were differentially associated with psychotic outcomes compared with depression or anxiety.

CONCLUSIONS

These findings challenge the hypothesis that atypical belief updating mechanisms (as indexed by the computational models and behavioral measures we used) underlie the development of psychotic phenomena.

Retrospective Inference as a Form of Bounded Rationality, and Its Beneficial Influence on Learning

Probabilistic models of cognition typically assume that agents make inferences about current states by combining new sensory information with fixed beliefs about the past, an approach known as Bayesian filtering. This is computationally parsimonious, but, in general, leads to suboptimal beliefs about past states, since it ignores the fact that new observations typically contain information about the past as well as the present. This is disadvantageous both because knowledge of past states may be intrinsically valuable, and because it impairs learning about fixed or slowly changing parameters of the environment. For these reasons, in offline data analysis it is usual to infer on every set of states using the entire time series of observations, an approach known as (fixed-interval) Bayesian smoothing. Unfortunately, however, this is impractical for real agents, since it requires the maintenance and updating of beliefs about an ever-growing set of states. We propose an intermediate approach, finite retrospective inference (FRI), in which agents perform update beliefs about a limited number of past states (Formally, this represents online fixed-lag smoothing with a sliding window). This can be seen as a form of bounded rationality in which agents seek to optimize the accuracy of their beliefs subject to computational and other resource costs. We show through simulation that this approach has the capacity to significantly increase the accuracy of both inference and learning, using a simple variational scheme applied to both randomly generated Hidden Markov models (HMMs), and a specific application of the HMM, in the form of the widely used probabilistic reversal task. Our proposal thus constitutes a theoretical contribution to normative accounts of bounded rationality, which makes testable empirical predictions that can be explored in future work.

Modeling subjective belief states in computational psychiatry: interoceptive inference as a candidate framework

The nascent field computational psychiatry has undergone exponential growth since its inception. To date, much of the published work has focused on choice behaviors, which are primarily modeled within a reinforcement learning framework. While this initial normative effort represents a milestone in psychiatry research, the reality is that many psychiatric disorders are defined by disturbances in subjective states (e.g., depression, anxiety) and associated beliefs (e.g., dysmorphophobia, paranoid ideation), which are not considered in normative models. In this paper, we present interoceptive inference as a candidate framework for modeling subjective-and associated belief-states in computational psychiatry. We first introduce the notion and significance of modeling subjective states in computational psychiatry. Next, we present the interoceptive inference framework, and in particular focus on the relationship between interoceptive inference (i.e., belief updating) and emotions. Lastly, we will use drug craving as an example of subjective states to demonstrate the feasibility of using interoceptive inference to model the psychopathology of subjective states.

Sequential inference as a mode of cognition and its correlates in fronto-parietal and hippocampal brain regions

Normative models of human cognition often appeal to Bayesian filtering, which provides optimal online estimates of unknown or hidden states of the world, based on previous observations. However, in many cases it is necessary to optimise beliefs about sequences of states rather than just the current state. Importantly, Bayesian filtering and sequential inference strategies make different predictions about beliefs and subsequent choices, rendering them behaviourally dissociable. Taking data from a probabilistic reversal task we show that subjects’ choices provide strong evidence that they are representing short sequences of states. Between-subject measures of this implicit sequential inference strategy had a neurobiological underpinning and correlated with grey matter density in prefrontal and parietal cortex, as well as the hippocampus. Our findings provide, to our knowledge, the first evidence for sequential inference in human cognition, and by exploiting between-subject variation in this measure we provide pointers to its neuronal substrates.

When studying human cognition, it is often assumed that agents form and update beliefs only about the current state of the world, an approach known as Bayesian filtering. However, in many situations there are advantages to making inferences about the most likely sequence of states that have occurred, which involves simultaneously updating beliefs about the present and the past, based on incoming information. Currently, very little is known about whether humans adopt such sequential inference strategies, and if they do, about the neuronal mechanisms involved. We addressed this by applying computational modelling to data collected during a probabilistic reversal task. At a group level, subjects’ behaviour showed clear evidence of sequential inference, and between-subject differences in the strategies adopted were reflected in variations in brain structure in the prefrontal and parietal cortices, as well as the hippocampus. Our results provide new insight into the strategies employed in human cognition, as well as the neuronal substrates of sequential inference.

Linking canonical microcircuits and neuronal activity: Dynamic causal modelling of laminar recordings

Neural models describe brain activity at different scales, ranging from single cells to whole brain networks. Here, we attempt to reconcile models operating at the microscopic (compartmental) and mesoscopic (neural mass) scales to analyse data from microelectrode recordings of intralaminar neural activity. Although these two classes of models operate at different scales, it is relatively straightforward to create neural mass models of ensemble activity that are equipped with priors obtained after fitting data generated by detailed microscopic models. This provides generative (forward) models of measured neuronal responses that retain construct validity in relation to compartmental models. We illustrate our approach using cross spectral responses obtained from V1 during a visual perception paradigm that involved optogenetic manipulation of the basal forebrain. We find that the resulting neural mass model can distinguish between activity in distinct cortical layers - both with and without optogenetic activation - and that cholinergic input appears to enhance (disinhibit) superficial layer activity relative to deep layers. This is particularly interesting from the perspective of predictive coding, where neuromodulators are thought to boost prediction errors that ascend the cortical hierarchy.

Evidence for surprise minimization over value maximization in choice behavior

Classical economic models are predicated on the idea that the ultimate aim of choice is to maximize utility or reward. In contrast, an alternative perspective highlights the fact that adaptive behavior requires agents' to model their environment and minimize surprise about the states they frequent. We propose that choice behavior can be more accurately accounted for by surprise minimization compared to reward or utility maximization alone. Minimizing surprise makes a prediction at variance with expected utility models; namely, that in addition to attaining valuable states, agents attempt to maximize the entropy over outcomes and thus 'keep their options open'. We tested this prediction using a simple binary choice paradigm and show that human decision-making is better explained by surprise minimization compared to utility maximization. Furthermore, we replicated this entropy-seeking behavior in a control task with no explicit utilities. These findings highlight a limitation of purely economic motivations in explaining choice behavior and instead emphasize the importance of belief-based motivations.

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

Neural signals encoding shifts in beliefs

Dopamine is implicated in a diverse range of cognitive functions including cognitive flexibility, task switching, signalling novel or unexpected stimuli as well as advance information. There is also longstanding line of thought that links dopamine with belief formation and, crucially, aberrant belief formation in psychosis. Integrating these strands of evidence would suggest that dopamine plays a central role in belief updating and more specifically in encoding of meaningful information content in observations. The precise nature of this relationship has remained unclear. To directly address this question we developed a paradigm that allowed us to decompose two distinct types of information content, information-theoretic surprise that reflects the unexpectedness of an observation, and epistemic value that induces shifts in beliefs or, more formally, Bayesian surprise. Using functional magnetic-resonance imaging in humans we show that dopamine-rich midbrain regions encode shifts in beliefs whereas surprise is encoded in prefrontal regions, including the pre-supplementary motor area and dorsal cingulate cortex. By linking putative dopaminergic activity to belief updating these data provide a link to false belief formation that characterises hyperdopaminergic states associated with idiopathic and drug induced psychosis.

The Dopaminergic Midbrain Encodes the Expected Certainty about Desired Outcomes

Dopamine plays a key role in learning; however, its exact function in decision making and choice remains unclear. Recently, we proposed a generic model based on active (Bayesian) inference wherein dopamine encodes the precision of beliefs about optimal policies. Put simply, dopamine discharges reflect the confidence that a chosen policy will lead to desired outcomes. We designed a novel task to test this hypothesis, where subjects played a "limited offer" game in a functional magnetic resonance imaging experiment. Subjects had to decide how long to wait for a high offer before accepting a low offer, with the risk of losing everything if they waited too long. Bayesian model comparison showed that behavior strongly supported active inference, based on surprise minimization, over classical utility maximization schemes. Furthermore, midbrain activity, encompassing dopamine projection neurons, was accurately predicted by trial-by-trial variations in model-based estimates of precision. Our findings demonstrate that human subjects infer both optimal policies and the precision of those inferences, and thus support the notion that humans perform hierarchical probabilistic Bayesian inference. In other words, subjects have to infer both what they should do as well as how confident they are in their choices, where confidence may be encoded by dopaminergic firing.

Active inference, evidence accumulation, and the urn task

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology.

Active inference, evidence accumulation, and the urn task

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology.

Revealing a brain network endophenotype in families with idiopathic generalised epilepsy

Idiopathic generalised epilepsy (IGE) has a genetic basis. The mechanism of seizure expression is not fully known, but is assumed to involve large-scale brain networks. We hypothesised that abnormal brain network properties would be detected using EEG in patients with IGE, and would be manifest as a familial endophenotype in their unaffected first-degree relatives. We studied 117 participants: 35 patients with IGE, 42 unaffected first-degree relatives, and 40 normal controls, using scalp EEG. Graph theory was used to describe brain network topology in five frequency bands for each subject. Frequency bands were chosen based on a published Spectral Factor Analysis study which demonstrated these bands to be optimally robust and independent. Groups were compared, using Bonferroni correction to account for nonindependent measures and multiple groups. Degree distribution variance was greater in patients and relatives than controls in the 6-9 Hz band (p = 0.0005, p = 0.0009 respectively). Mean degree was greater in patients than healthy controls in the 6-9 Hz band (p = 0.0064). Clustering coefficient was higher in patients and relatives than controls in the 6-9 Hz band (p = 0.0025, p = 0.0013). Characteristic path length did not differ between groups. No differences were found between patients and unaffected relatives. These findings suggest brain network topology differs between patients with IGE and normal controls, and that some of these network measures show similar deviations in patients and in unaffected relatives who do not have epilepsy. This suggests brain network topology may be an inherited endophenotype of IGE, present in unaffected relatives who do not have epilepsy, as well as in affected patients. We propose that abnormal brain network topology may be an endophenotype of IGE, though not in itself sufficient to cause epilepsy.

Model averaging, optimal inference, and habit formation

Postulating that the brain performs approximate Bayesian inference generates principled and empirically testable models of neuronal function-the subject of much current interest in neuroscience and related disciplines. Current formulations address inference and learning under some assumed and particular model. In reality, organisms are often faced with an additional challenge-that of determining which model or models of their environment are the best for guiding behavior. Bayesian model averaging-which says that an agent should weight the predictions of different models according to their evidence-provides a principled way to solve this problem. Importantly, because model evidence is determined by both the accuracy and complexity of the model, optimal inference requires that these be traded off against one another. This means an agent's behavior should show an equivalent balance. We hypothesize that Bayesian model averaging plays an important role in cognition, given that it is both optimal and realizable within a plausible neuronal architecture. We outline model averaging and how it might be implemented, and then explore a number of implications for brain and behavior. In particular, we propose that model averaging can explain a number of apparently suboptimal phenomena within the framework of approximate (bounded) Bayesian inference, focusing particularly upon the relationship between goal-directed and habitual behavior.

Transcranial direct current stimulation of right dorsolateral prefrontal cortex does not affect model-based or model-free reinforcement learning in humans

There is broad consensus that the prefrontal cortex supports goal-directed, model-based decision-making. Consistent with this, we have recently shown that model-based control can be impaired through transcranial magnetic stimulation of right dorsolateral prefrontal cortex in humans. We hypothesized that an enhancement of model-based control might be achieved by anodal transcranial direct current stimulation of the same region. We tested 22 healthy adult human participants in a within-subject, double-blind design in which participants were given Active or Sham stimulation over two sessions. We show Active stimulation had no effect on model-based control or on model-free ('habitual') control compared to Sham stimulation. These null effects are substantiated by a power analysis, which suggests that our study had at least 60% power to detect a true effect, and by a Bayesian model comparison, which favors a model of the data that assumes stimulation had no effect over models that assume stimulation had an effect on behavioral control. Although we cannot entirely exclude more trivial explanations for our null effect, for example related to (faults in) our experimental setup, these data suggest that anodal transcranial direct current stimulation over right dorsolateral prefrontal cortex does not improve model-based control, despite existing evidence that transcranial magnetic stimulation can disrupt such control in the same brain region.

Disruption of dorsolateral prefrontal cortex decreases model-based in favor of model-free control in humans

Human choice behavior often reflects a competition between inflexible computationally efficient control on the one hand and a slower more flexible system of control on the other. This distinction is well captured by model-free and model-based reinforcement learning algorithms. Here, studying human subjects, we show it is possible to shift the balance of control between these systems by disruption of right dorsolateral prefrontal cortex, such that participants manifest a dominance of the less optimal model-free control. In contrast, disruption of left dorsolateral prefrontal cortex impaired model-based performance only in those participants with low working memory capacity.

•

Disrupting right dorsolateral prefrontal cortex impairs flexible model-based choice

•

This drives behavior toward simpler, model-free control

•

Results provide causal evidence for neural underpinnings of flexible choice