Computation noise in human learning and decision-making: origin, impact, function

Adolescent-to-adult gains in cognitive flexibility are adaptively supported by reward sensitivity, exploration, and neural variability

Cognitive flexibility exhibits dynamic changes throughout development, with different forms of flexibility showing dissociable developmental trajectories. In this review, we propose that an adolescent-specific mode of flexibility in the face of changing environmental contingencies supports the emergence of adolescent-to-adult gains in cognitive shifting efficiency. We first describe how cognitive shifting abilities monotonically improve from childhood to adulthood, accompanied by increases in brain state flexibility, neural variability, and excitatory/inhibitory balance. We next summarize evidence supporting the existence of a dopamine-driven, adolescent peak in flexible behavior that results in reward seeking, undirected exploration, and environmental sampling. We propose a neurodevelopmental framework that relates these adolescent behaviors to the refinement of neural phenotypes relevant to mature cognitive flexibility, and thus highlight the importance of the adolescent period in fostering healthy neurocognitive trajectories.

The neural network RTNet exhibits the signatures of human perceptual decision-making

Convolutional neural networks show promise as models of biological vision. However, their decision behaviour, including the facts that they are deterministic and use equal numbers of computations for easy and difficult stimuli, differs markedly from human decision-making, thus limiting their applicability as models of human perceptual behaviour. Here we develop a new neural network, RTNet, that generates stochastic decisions and human-like response time (RT) distributions. We further performed comprehensive tests that showed RTNet reproduces all foundational features of human accuracy, RT and confidence and does so better than all current alternatives. To test RTNet's ability to predict human behaviour on novel images, we collected accuracy, RT and confidence data from 60 human participants performing a digit discrimination task. We found that the accuracy, RT and confidence produced by RTNet for individual novel images correlated with the same quantities produced by human participants. Critically, human participants who were more similar to the average human performance were also found to be closer to RTNet's predictions, suggesting that RTNet successfully captured average human behaviour. Overall, RTNet is a promising model of human RTs that exhibits the critical signatures of perceptual decision-making.

Regulation of reinforcement learning parameters captures long-term changes in rat behaviour

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Modelling attention allocation and takeover performance in two-stage takeover system via a cognitive computational model: considering the role of multiple monitoring requests

Studies have demonstrated two-stage takeover systems' feasibility and advantages. However, existing cognitive models mainly focus on simulating drivers' performance in single-stage takeover systems, with limited insights into cognitive modelling of effects of monitoring requests (MRs) within two-stage takeover systems. This study constructed a cognitive computational model for two-stage takeover systems based on queueing network-adaptive control of thought rational (QN-ACTR) architecture. Our model aims to capture variations in drivers' attention allocation and takeover performance resulting from different MR experiences. Five components, representing distinct cognitive processes, were designed to closely align with drivers' behavioural patterns. This model was validated through an experiment using metrics such as percentage time in road-centre and takeover time. Results revealed significant concordance between the model predictions and experimental data, with R-squared ≥ 0.76, RMSE ≤ 0.41, and MAPE ≤ 15%. The findings of this work extended beyond the two-stage takeover system investigation to include human factor modelling.

Impaired value-based decision-making in Parkinson's disease apathy

Apathy is a common and disabling complication of Parkinson's disease characterized by reduced goal-directed behaviour. Several studies have reported dysfunction within prefrontal cortical regions and projections from brainstem nuclei whose neuromodulators include dopamine, serotonin and noradrenaline. Work in animal and human neuroscience have confirmed contributions of these neuromodulators on aspects of motivated decision-making. Specifically, these neuromodulators have overlapping contributions to encoding the value of decisions, and influence whether to explore alternative courses of action or persist in an existing strategy to achieve a rewarding goal. Building upon this work, we hypothesized that apathy in Parkinson's disease should be associated with an impairment in value-based learning. Using a four-armed restless bandit reinforcement learning task, we studied decision-making in 75 volunteers; 53 patients with Parkinson's disease, with and without clinical apathy, and 22 age-matched healthy control subjects. Patients with apathy exhibited impaired ability to choose the highest value bandit. Task performance predicted an individual patient's apathy severity measured using the Lille Apathy Rating Scale (R = -0.46, P < 0.001). Computational modelling of the patient's choices confirmed the apathy group made decisions that were indifferent to the learnt value of the options, consistent with previous reports of reward insensitivity. Further analysis demonstrated a shift away from exploiting the highest value option and a reduction in perseveration, which also correlated with apathy scores (R = -0.5, P < 0.001). We went on to acquire functional MRI in 59 volunteers; a group of 19 patients with and 20 without apathy and 20 age-matched controls performing the Restless Bandit Task. Analysis of the functional MRI signal at the point of reward feedback confirmed diminished signal within ventromedial prefrontal cortex in Parkinson's disease, which was more marked in apathy, but not predictive of their individual apathy severity. Using a model-based categorization of choice type, decisions to explore lower value bandits in the apathy group activated prefrontal cortex to a similar degree to the age-matched controls. In contrast, Parkinson's patients without apathy demonstrated significantly increased activation across a distributed thalamo-cortical network. Enhanced activity in the thalamus predicted individual apathy severity across both patient groups and exhibited functional connectivity with dorsal anterior cingulate cortex and anterior insula. Given that task performance in patients without apathy was no different to the age-matched control subjects, we interpret the recruitment of this network as a possible compensatory mechanism, which compensates against symptomatic manifestation of apathy in Parkinson's disease.

Multiple and subject-specific roles of uncertainty in reward-guided decision-making

Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices. First, stable and psychologically meaningful individual differences in uncertainty preferences actually range from seeking to avoidance, which can manifest as null group-level effects. Second, uncertainty modulates the use of basic decision heuristics that imperfectly exploit immediate rewards: a repetition bias and win-stay-lose-shift heuristic. These heuristics interact with uncertainty, favoring heuristic choices under higher uncertainty. These results, highlighting the rich and varied structure of reward-based choice, are a step to understanding its functional basis and dysfunction in psychopathology.

Dynamic noise estimation: A generalized method for modeling noise fluctuations in decision-making

Computational cognitive modeling is an important tool for understanding the processes supporting human and animal decision-making. Choice data in decision-making tasks are inherently noisy, and separating noise from signal can improve the quality of computational modeling. Common approaches to model decision noise often assume constant levels of noise or exploration throughout learning (e.g., the ϵ -softmax policy). However, this assumption is not guaranteed to hold - for example, a subject might disengage and lapse into an inattentive phase for a series of trials in the middle of otherwise low-noise performance. Here, we introduce a new, computationally inexpensive method to dynamically infer the levels of noise in choice behavior, under a model assumption that agents can transition between two discrete latent states (e.g., fully engaged and random). Using simulations, we show that modeling noise levels dynamically instead of statically can substantially improve model fit and parameter estimation, especially in the presence of long periods of noisy behavior, such as prolonged attentional lapses. We further demonstrate the empirical benefits of dynamic noise estimation at the individual and group levels by validating it on four published datasets featuring diverse populations, tasks, and models. Based on the theoretical and empirical evaluation of the method reported in the current work, we expect that dynamic noise estimation will improve modeling in many decision-making paradigms over the static noise estimation method currently used in the modeling literature, while keeping additional model complexity and assumptions minimal.

Controlling morpho-electrophysiological variability of neurons with detailed biophysical models

Variability, which is known to be a universal feature among biological units such as neuronal cells, holds significant importance, as, for example, it enables a robust encoding of a high volume of information in neuronal circuits and prevents hypersynchronizations. While most computational studies on electrophysiological variability in neuronal circuits were done with single-compartment neuron models, we instead focus on the variability of detailed biophysical models of neuron multi-compartmental morphologies. We leverage a Markov chain Monte Carlo method to generate populations of electrical models reproducing the variability of experimental recordings while being compatible with a set of morphologies to faithfully represent specifi morpho-electrical type. We demonstrate our approach on layer 5 pyramidal cells and study the morpho-electrical variability and in particular, find that morphological variability alone is insufficient to reproduce electrical variability. Overall, this approach provides a strong statistical basis to create detailed models of neurons with controlled variability.

Expecting the unexpected: a review of learning under uncertainty across development

Many of our decisions take place under uncertainty. To successfully navigate the environment, individuals need to estimate the degree of uncertainty and adapt their behaviors accordingly by learning from experiences. However, uncertainty is a broad construct and distinct types of uncertainty may differentially influence our learning. We provide a semi-systematic review to illustrate cognitive and neurobiological processes involved in learning under two types of uncertainty: learning in environments with stochastic outcomes, and with volatile outcomes. We specifically reviewed studies (N = 26 studies) that included an adolescent population, because adolescence is a period in life characterized by heightened exploration and learning, as well as heightened uncertainty due to experiencing many new, often social, environments. Until now, reviews have not comprehensively compared learning under distinct types of uncertainties in this age range. Our main findings show that although the overall developmental patterns were mixed, most studies indicate that learning from stochastic outcomes, as indicated by increased accuracy in performance, improved with age. We also found that adolescents tended to have an advantage compared with adults and children when learning from volatile outcomes. We discuss potential mechanisms explaining these age-related differences and conclude by outlining future research directions.

Adaptive tuning of human learning and choice variability to unexpected uncertainty

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Pallidal neuromodulation of the explore/exploit trade-off in decision-making

Every decision that we make involves a conflict between exploiting our current knowledge of an action's value or exploring alternative courses of action that might lead to a better, or worse outcome. The sub-cortical nuclei that make up the basal ganglia have been proposed as a neural circuit that may contribute to resolving this explore-exploit 'dilemma'. To test this hypothesis, we examined the effects of neuromodulating the basal ganglia's output nucleus, the globus pallidus interna, in patients who had undergone deep brain stimulation (DBS) for isolated dystonia. Neuromodulation enhanced the number of exploratory choices to the lower value option in a two-armed bandit probabilistic reversal-learning task. Enhanced exploration was explained by a reduction in the rate of evidence accumulation (drift rate) in a reinforcement learning drift diffusion model. We estimated the functional connectivity profile between the stimulating DBS electrode and the rest of the brain using a normative functional connectome derived from heathy controls. Variation in the extent of neuromodulation induced exploration between patients was associated with functional connectivity from the stimulation electrode site to a distributed brain functional network. We conclude that the basal ganglia's output nucleus, the globus pallidus interna, can adaptively modify decision choice when faced with the dilemma to explore or exploit.

Efficient stabilization of imprecise statistical inference through conditional belief updating

Statistical inference is the optimal process for forming and maintaining accurate beliefs about uncertain environments. However, human inference comes with costs due to its associated biases and limited precision. Indeed, biased or imprecise inference can trigger variable beliefs and unwarranted changes in behaviour. Here, by studying decisions in a sequential categorization task based on noisy visual stimuli, we obtained converging evidence that humans reduce the variability of their beliefs by updating them only when the reliability of incoming sensory information is judged as sufficiently strong. Instead of integrating the evidence provided by all stimuli, participants actively discarded as much as a third of stimuli. This conditional belief updating strategy shows good test-retest reliability, correlates with perceptual confidence and explains human behaviour better than previously described strategies. This seemingly suboptimal strategy not only reduces the costs of imprecise computations but also, counterintuitively, increases the accuracy of resulting decisions.

The effects of base rate neglect on sequential belief updating and real-world beliefs

Base-rate neglect is a pervasive bias in judgment that is conceptualized as underweighting of prior information and can have serious consequences in real-world scenarios. This bias is thought to reflect variability in inferential processes but empirical support for a cohesive theory of base-rate neglect with sufficient explanatory power to account for longer-term and real-world beliefs is lacking. A Bayesian formalization of base-rate neglect in the context of sequential belief updating predicts that belief trajectories should exhibit dynamic patterns of dependence on the order in which evidence is presented and its consistency with prior beliefs. To test this, we developed a novel 'urn-and-beads' task that systematically manipulated the order of colored bead sequences and elicited beliefs via an incentive-compatible procedure. Our results in two independent online studies confirmed the predictions of the sequential base-rate neglect model: people exhibited beliefs that are more influenced by recent evidence and by evidence inconsistent with prior beliefs. We further found support for a noisy-sampling inference model whereby base-rate neglect results from rational discounting of noisy internal representations of prior beliefs. Finally, we found that model-derived indices of base-rate neglect-including noisier prior representation-correlated with propensity for unusual beliefs outside the laboratory. Our work supports the relevance of Bayesian accounts of sequential base-rate neglect to real-world beliefs and hints at strategies to minimize deleterious consequences of this pervasive bias.

The metacontrol hypothesis as diagnostic framework of OCD and ADHD: A dimensional approach based on shared neurobiological vulnerability

Obsessive-compulsive disorder (OCD) and attention-deficit hyperactivity disorder (ADHD) are multi-faceted neuropsychiatric conditions that in many aspects appear to be each other's antipodes. We suggest a dimensional approach, according to which these partially opposing disorders fall onto a continuum that reflects variability regarding alterations of cortico-striato-thalamo-cortical (CSTC) circuits and of the processing of neural noise during cognition. By using theoretical accounts of human cognitive metacontrol, we develop a framework according to which OCD can be characterized by a chronic bias towards exaggerated cognitive persistence, equivalent to a high signal-to-noise ratio (SNR)-which facilitates perseverative behaviour but impairs mental flexibility. In contrast, ADHD is characterized by a chronic bias towards inflated cognitive flexibility, equivalent to a low SNR-which increases behavioural variability but impairs the focusing on one goal and on relevant information. We argue that, when pharmacology is not feasible, novel treatments of these disorders may involve methods to manipulate the signal-to-noise ratio via non-invasive brain stimulation techniques, in order to normalize the situational imbalance between cognitive persistence and cognitive flexibility.

Rational arbitration between statistics and rules in human sequence processing

Detecting and learning temporal regularities is essential to accurately predict the future. A long-standing debate in cognitive science concerns the existence in humans of a dissociation between two systems, one for handling statistical regularities governing the probabilities of individual items and their transitions, and another for handling deterministic rules. Here, to address this issue, we used finger tracking to continuously monitor the online build-up of evidence, confidence, false alarms and changes-of-mind during sequence processing. All these aspects of behaviour conformed tightly to a hierarchical Bayesian inference model with distinct hypothesis spaces for statistics and rules, yet linked by a single probabilistic currency. Alternative models based either on a single statistical mechanism or on two non-commensurable systems were rejected. Our results indicate that a hierarchical Bayesian inference mechanism, capable of operating over distinct hypothesis spaces for statistics and rules, underlies the human capability for sequence processing.

Premature commitment to uncertain decisions during human NMDA receptor hypofunction

Making accurate decisions based on unreliable sensory evidence requires cognitive inference. Dysfunction of n-methyl-d-aspartate (NMDA) receptors impairs the integration of noisy input in theoretical models of neural circuits, but whether and how this synaptic alteration impairs human inference and confidence during uncertain decisions remains unknown. Here we use placebo-controlled infusions of ketamine to characterize the causal effect of human NMDA receptor hypofunction on cognitive inference and its neural correlates. At the behavioral level, ketamine triggers inference errors and elevated decision uncertainty. At the neural level, ketamine is associated with imbalanced coding of evidence and premature response preparation in electroencephalographic (EEG) activity. Through computational modeling of inference and confidence, we propose that this specific pattern of behavioral and neural impairments reflects an early commitment to inaccurate decisions, which aims at resolving the abnormal uncertainty generated by NMDA receptor hypofunction.

Current and expected affective valence interact to predict choice in recurrent decisions

Research on the role of affect in decision-making indicates that both predecisional current and expected affective valence predict choice. However, the exact role of current and expected affect for recurrent decision-making is still a matter of debate. We used a generalised mixed effect model to predict gambling responses in an experience-based learning task from ratings of current and expected affective valence. Results indicate that current and expected affective valence interact to predict choice. While expected valence had the biggest effect size, current valence and the interaction still contributed significantly to the prediction of choice. Resolving the interaction showed that participants relied more on the current valence if expectations were unclear or positive. These findings are discussed in the context of dual-process accounts and the affective signalling hypothesis. In conclusion, current and expected valence depend on one another and interact to predict choice in recurrent decision tasks.

Neural excitability and sensory input determine intensity perception with opposing directions in initial cortical responses

Perception of sensory information is determined by stimulus features (e.g., intensity) and instantaneous neural states (e.g., excitability). Commonly, it is assumed that both are reflected similarly in evoked brain potentials, that is, larger amplitudes are associated with a stronger percept of a stimulus. We tested this assumption in a somatosensory discrimination task in humans, simultaneously assessing (i) single-trial excitatory post-synaptic currents inferred from short-latency somatosensory evoked potentials (SEPs), (ii) pre-stimulus alpha oscillations (8–13 Hz), and (iii) peripheral nerve measures. Fluctuations of neural excitability shaped the perceived stimulus intensity already during the very first cortical response (at ~20 ms) yet demonstrating opposite neural signatures as compared to the effect of presented stimulus intensity. We reconcile this discrepancy via a common framework based on the modulation of electro-chemical membrane gradients linking neural states and responses, which calls for reconsidering conventional interpretations of brain potential magnitudes in stimulus intensity encoding.