Active learning impairments in substance use disorders when resolving the explore-exploit dilemma: A replication and extension of previous computational modeling results

Background

Substance use disorders (SUDs) represent a major public health risk. Yet, our understanding of the mechanisms that maintain these disorders remains incomplete. In a recent computational modeling study, we found initial evidence that SUDs are associated with slower learning rates from negative outcomes and less value-sensitive choice (low "action precision"), which could help explain continued substance use despite harmful consequences.

Methods

Here we aimed to replicate and extend these results in a pre-registered study with a new sample of 168 individuals with SUDs and 99 healthy comparisons (HCs). We performed the same computational modeling and group comparisons as in our prior report (doi: 10.1016/j.drugalcdep.2020.108208) to confirm previously observed effects. After completing all pre-registered replication analyses, we then combined the previous and current datasets (N = 468) to assess whether differences were transdiagnostic or driven by specific disorders.

Results

Replicating prior results, SUDs showed slower learning rates for negative outcomes in both Bayesian and frequentist analyses (η 2 =.02). Previously observed differences in action precision were not confirmed. Logistic regressions including all computational parameters as predictors in the combined datasets could differentiate several specific disorders from HCs, but could not differentiate most disorders from each other.

Conclusions

These results provide robust evidence that individuals with SUDs have more difficulty adjusting behavior in the face of negative outcomes than HCs. They also suggest this effect is common across several different SUDs. Future research should examine its neural basis and whether learning rates could represent a new treatment target or moderator of treatment outcome.

An astonishing regularity in student learning rate

Leveraging a scientific infrastructure for exploring how students learn, we have developed cognitive and statistical models of skill acquisition and used them to understand fundamental similarities and differences across learners. Our primary question was why do some students learn faster than others? Or, do they? We model data from student performance on groups of tasks that assess the same skill component and that provide follow-up instruction on student errors. Our models estimate, for both students and skills, initial correctness and learning rate, that is, the increase in correctness after each practice opportunity. We applied our models to 1.3 million observations across 27 datasets of student interactions with online practice systems in the context of elementary to college courses in math, science, and language. Despite the availability of up-front verbal instruction, like lectures and readings, students demonstrate modest initial prepractice performance, at about 65% accuracy. Despite being in the same course, students' initial performance varies substantially from about 55% correct for those in the lower half to 75% for those in the upper half. In contrast, and much to our surprise, we found students to be astonishingly similar in estimated learning rate, typically increasing by about 0.1 log odds or 2.5% in accuracy per opportunity. These findings pose a challenge for theories of learning to explain the odd combination of large variation in student initial performance and striking regularity in student learning rate.

The more, the better? Learning rate and self-pacing in neurofeedback enhance cognitive performance in healthy adults

Real time electroencephalogram (EEG) based neurofeedback has been shown to be effective in regulating brain activity, thereby modifying cognitive performance and behavior. Nevertheless, individual variations in neurofeedback learning rates limit the overall efficacy of EEG based neurofeedback. In the present study we investigated the effects of learning rate and control over training realized by self-pacing on cognitive performance and electrocortical activity. Using a double-blind design, we randomly allocated 60 participants to either individual upper alpha (IUA) or sham neurofeedback and subsequently to self- or externally paced training. Participants receiving IUA neurofeedback improved their IUA activity more than participants receiving sham neurofeedback. Furthermore, the learning rate predicted enhancements in resting-state activity and mental rotation ability. The direction of this linear relationship depended on the neurofeedback condition being positive for IUA and negative for sham neurofeedback. Finally, self-paced training increased higher-level cognitive skills more than externally paced training. These results underpin the important role of learning rate in enhancing both resting-state activity and cognitive performance. Our design allowed us to differentiate the effect of learning rate between neurofeedback conditions, and to demonstrate the positive effect of self-paced training on cognitive performance in IUA neurofeedback.

Early Diagnosis of COVID-19 Images Using Optimal CNN Hyperparameters

Coronavirus disease (COVID-19) is a worldwide epidemic that poses substantial health hazards. However, COVID-19 diagnostic test sensitivity is still restricted due to abnormalities in specimen processing. Meanwhile, optimizing the highly defined number of convolutional neural network (CNN) hyperparameters (hundreds to thousands) is a useful direction to improve its overall performance and overcome its cons. Hence, this paper proposes an optimization strategy for obtaining the optimal learning rate and momentum of a CNN's hyperparameters using the grid search method to improve the network performance. Therefore, three alternative CNN architectures (GoogleNet, VGG16, and ResNet) were used to optimize hyperparameters utilizing two different COVID-19 radiography data sets (Kaggle (X-ray) and China national center for bio-information (CT)). These architectures were tested with/without optimizing the hyperparameters. The results confirm effective disease classification using the CNN structures with optimized hyperparameters. Experimental findings indicate that the new technique outperformed the previous in terms of accuracy, sensitivity, specificity, recall, F-score, false positive and negative rates, and error rate. At epoch 25, the optimized Resnet obtained high classification accuracy, reaching 98.98% for X-ray images and 98.78% for CT images.

Separation of memory span and learning rate: Evidence from behavior and spontaneous brain activity in older adults

It is unclear how the ability to initially acquire information in a first learning trial relates to learning rate in subsequent repeated trials. The separation of memory span and learning rate is an important psychological dilemma that remains unaddressed. Given the potential effects of aging on memory and learning, this study investigated the separation of memory span and learning rate from behavior and spontaneous brain activity in older adults. We enrolled a total of 758 participants, including 707 healthy older adults and 51 mild cognitive impairment (MCI) patients. Sixty-five participants out of 707 completed resting-state functional magnetic resonance imaging (fMRI) scanning. Behaviorally, memory span and learning rate were not correlated with each other in the paired-associative learning test (PALT) but were negatively correlated in the auditory verbal learning test (AVLT). This indicated that the relationship between memory span and learning rate for item memory might be differentially affected by aging. Interaction analysis confirmed that these two capacities were differentially affected by test type (associative memory vs. item memory). Additionally, at three progressive brain activity indexes (ALFF, ReHo, and DC), the right brain regions (right inferior temporal gyrus and right middle frontal gyrus) were more negatively correlated with memory span, whereas, the left precuneus was more positively correlated with learning rate. Regarding pathological aging, none of the correlations between memory span and learning rate were significant in either PALT or AVLT in MCI. This study provides novel evidence for the dissociation of memory span and learning rate at behavioral and brain activity levels, which may have useful applications in detecting cognitive deficits or conducting cognitive interventions.

Intelligent Detection of Hazardous Goods Vehicles and Determination of Risk Grade Based on Deep Learning

Currently, deep learning has been widely applied in the field of object detection, and some relevant scholars have applied it to vehicle detection. In this paper, the deep learning EfficientDet model is analyzed, and the advantages of the model in the detection of hazardous good vehicles are determined. The adaptive training model is built based on the optimization of the training process, and the training model is used to detect hazardous goods vehicles. The detection results are compared with Cascade R-CNN and CenterNet, and the results show that the proposed method is superior to the other two methods in two aspects of computational complexity and detection accuracy. Simultaneously, the proposed method is suitable for the detection of hazardous goods vehicles in different scenarios. We make statistics on the number of detected hazardous goods vehicles at different times and places. The risk grade of different locations is determined according to the statistical results. Finally, the case study shows that the proposed method can be used to detect hazardous goods vehicles and determine the risk level of different places.

Contributions by metaplasticity to solving the Catastrophic Forgetting Problem

Catastrophic forgetting (CF) refers to the sudden and severe loss of prior information in learning systems when acquiring new information. CF has been an Achilles heel of standard artificial neural networks (ANNs) when learning multiple tasks sequentially. The brain, by contrast, has solved this problem during evolution. Modellers now use a variety of strategies to overcome CF, many of which have parallels to cellular and circuit functions in the brain. One common strategy, based on metaplasticity phenomena, controls the future rate of change at key connections to help retain previously learned information. However, the metaplasticity properties so far used are only a subset of those existing in neurobiology. We propose that as models become more sophisticated, there could be value in drawing on a richer set of metaplasticity rules, especially when promoting continual learning in agents moving about the environment.

Predator or provider? How wild animals respond to mixed messages from humans

Wild animals encounter humans on a regular basis, but humans vary widely in their behaviour: whereas many people ignore wild animals, some people present a threat, while others encourage animals' presence through feeding. Humans thus send mixed messages to which animals must respond appropriately to be successful. Some species appear to circumvent this problem by discriminating among and/or socially learning about humans, but it is not clear whether such learning strategies are actually beneficial in most cases. Using an individual-based model, we consider how learning rate, individual recognition (IR) of humans, and social learning (SL) affect wild animals' ability to reach an optimal avoidance strategy when foraging in areas frequented by humans. We show that 'true' IR of humans could be costly. We also find that a fast learning rate, while useful when human populations are homogeneous or highly dangerous, can cause unwarranted avoidance in other scenarios if animals generalize. SL reduces this problem by allowing conspecifics to observe benign interactions with humans. SL and a fast learning rate also improve the viability of IR. These results provide an insight into how wild animals may be affected by, and how they may cope with, contrasting human behaviour.

Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate

Learning to predict rewards is essential for the sustained fitness of animals. Contemporary views suggest that such learning is driven by a reward prediction error (RPE)-the difference between received and predicted rewards. The magnitude of learning induced by an RPE is proportional to the product of the RPE and a learning rate. Here we demonstrate using two-photon calcium imaging and optogenetics in mice that certain functionally distinct subpopulations of ventral/medial orbitofrontal cortex (vmOFC) neurons signal learning rate control. Consistent with learning rate control, trial-by-trial fluctuations in vmOFC activity positively correlate with behavioral updating when the RPE is positive, and negatively correlates with behavioral updating when the RPE is negative. Learning rate is affected by many variables including the salience of a reward. We found that the average reward response of these neurons signals the relative salience of a reward, because it decreases after reward prediction learning or the introduction of another highly salient aversive stimulus. The relative salience signaling in vmOFC is sculpted by medial thalamic inputs. These results support emerging theoretical views that prefrontal cortex encodes and controls learning parameters.

Timescales of Cognition in the Brain

We live in a world that changes on many timescales. To learn and make decisions appropriately, the human brain has evolved to integrate various types of information, such as sensory evidence and reward feedback, on multiple timescales. This is reflected in cortical hierarchies of timescales consisting of heterogeneous neuronal activities and expression of genes related to neurotransmitters critical for learning. We review the recent findings on how timescales of sensory and reward integration are affected by the temporal properties of sensory and reward signals in the environment. Despite existing evidence linking behavioral and neuronal timescales, future studies must examine how neural computations at multiple timescales are adjusted and combined to influence behavior flexibly.

Deep Convolutional Neural Network Regularization for Alcoholism Detection Using EEG Signals

Alcoholism is attributed to regular or excessive drinking of alcohol and leads to the disturbance of the neuronal system in the human brain. This results in certain malfunctioning of neurons that can be detected by an electroencephalogram (EEG) using several electrodes on a human skull at appropriate positions. It is of great interest to be able to classify an EEG activity as that of a normal person or an alcoholic person using data from the minimum possible electrodes (or channels). Due to the complex nature of EEG signals, accurate classification of alcoholism using only a small dataset is a challenging task. Artificial neural networks, specifically convolutional neural networks (CNNs), provide efficient and accurate results in various pattern-based classification problems. In this work, we apply CNN on raw EEG data and demonstrate how we achieved 98% average accuracy by optimizing a baseline CNN model and outperforming its results in a range of performance evaluation metrics on the University of California at Irvine Machine Learning (UCI-ML) EEG dataset. This article explains the stepwise improvement of the baseline model using the dropout, batch normalization, and kernel regularization techniques and provides a comparison of the two models that can be beneficial for aspiring practitioners who aim to develop similar classification models in CNN. A performance comparison is also provided with other approaches using the same dataset.

Are efficient learners of verbal stimuli also efficient and precise learners of visuospatial stimuli?

People differ in how quickly they learn information and how long they remember it, and these two variables are correlated such that people who learn more quickly tend to retain more of the newly learned information. Zerr and colleagues [2018. Learning efficiency: Identifying individual differences in learning rate and retention in healthy adults. Psychological Science, 29(9), 1436-1450] termed the relation between learning rate and retention as learning efficiency, with more efficient learners having both a faster acquisition rate and better memory performance after a delay. Zerr et al. also demonstrated in separate experiments that how efficiently someone learns is stable across a range of days and years with the same kind of stimuli. The current experiments (combined N = 231) replicate the finding that quicker learning coincides with better retention and demonstrate that the correlation extends to multiple types of materials. We also address the generalisability of learning efficiency: A person's efficiency with learning Lithuanian-English (verbal-verbal) pairs predicts their efficiency with Chinese-English (visuospatial-verbal) and (to a lesser extent) object-location (visuospatial-visuospatial) paired associates. Finally, we examine whether quicker learners also remember material more precisely by using a continuous measure of recall accuracy with object-location pairs.

Adaptive latent state modeling of brain network dynamics with real-time learning rate optimization

Objective. Dynamic latent state models are widely used to characterize the dynamics of brain network activity for various neural signal types. To date, dynamic latent state models have largely been developed for stationary brain network dynamics. However, brain network dynamics can be non-stationary for example due to learning, plasticity or recording instability. To enable modeling these non-stationarities, two problems need to be resolved. First, novel methods should be developed that can adaptively update the parameters of latent state models, which is difficult due to the state being latent. Second, new methods are needed to optimize the adaptation learning rate, which specifies how fast new neural observations update the model parameters and can significantly influence adaptation accuracy.Approach. We develop a Rate Optimized-adaptive Linear State-Space Modeling (RO-adaptive LSSM) algorithm that solves these two problems. First, to enable adaptation, we derive a computation- and memory-efficient adaptive LSSM fitting algorithm that updates the LSSM parameters recursively and in real time in the presence of the latent state. Second, we develop a real-time learning rate optimization algorithm. We use comprehensive simulations of a broad range of non-stationary brain network dynamics to validate both algorithms, which together constitute the RO-adaptive LSSM.Main results. We show that the adaptive LSSM fitting algorithm can accurately track the broad simulated non-stationary brain network dynamics. We also find that the learning rate significantly affects the LSSM fitting accuracy. Finally, we show that the real-time learning rate optimization algorithm can run in parallel with the adaptive LSSM fitting algorithm. Doing so, the combined RO-adaptive LSSM algorithm rapidly converges to the optimal learning rate and accurately tracks non-stationarities.Significance. These algorithms can be used to study time-varying neural dynamics underlying various brain functions and enhance future neurotechnologies such as brain-machine interfaces and closed-loop brain stimulation systems.

Child-level factors affecting rate of learning to write in first grade

BACKGROUND

Written composition requires handwriting, spelling, and text planning skills, all largely learned through school instruction. Students' rate of learning to compose text in their first months at school will depend, in part, on their literacy-related abilities at school start. These effects have not previously been explored.

AIM

We aimed to establish the effects of various literacy-related abilities on the learning trajectory of first-grade students as they are taught to write.

SAMPLE

179 Spanish first-grade students (94 female, mean age 6.1 years) writing 3,512 texts.

METHOD

Students were assessed at start of school for spelling, transcription fluency, letter knowledge, phonological awareness, handwriting accuracy, word reading, and non-verbal reasoning. They were then taught under a curriculum that included researcher-designed instruction in handwriting, spelling, and ideation. Students' composition performance was probed at very regular intervals over their first 13 weeks at school.

RESULTS

Controlling for age, overall performance was predicted by spelling, transcription fluency, handwriting accuracy, word reading, and non-verbal reasoning. Most students showed rapid initial improvement, but then much slower learning. Weak spellers (and to a lesser extent less fluent hand-writers) showed weaker initial performance, but then steady improvement across the study period.

CONCLUSION

Transcription ability at school entry affects response to writing instruction.

Autistic traits are related to worse performance in a volatile reward learning task despite adaptive learning rates

LAY ABSTRACT

Recent theories propose that autism is characterized by an impairment in determining when to learn and when not. Here, we investigated this hypothesis by estimating learning rates (i.e. the speed with which one learns) in three different environments that differed in rule stability and uncertainty. We found that neurotypical participants with more autistic traits performed worse in a volatile environment (with unstable rules), as they chose less often for the most rewarding option. Exploratory analyses indicated that performance was specifically worse when reward rules were opposite to those initially learned for participants with more autistic traits. However, there were no differences in the adjustment of learning rates between participants with more versus less autistic traits. Together, these results suggest that performance in volatile environments is lower in participants with more autistic traits, but that this performance difference cannot be unambiguously explained by an impairment in adjusting learning rates.

Using reinforcement learning models in social neuroscience: frameworks, pitfalls and suggestions of best practices

The recent years have witnessed a dramatic increase in the use of reinforcement learning (RL) models in social, cognitive and affective neuroscience. This approach, in combination with neuroimaging techniques such as functional magnetic resonance imaging, enables quantitative investigations into latent mechanistic processes. However, increased use of relatively complex computational approaches has led to potential misconceptions and imprecise interpretations. Here, we present a comprehensive framework for the examination of (social) decision-making with the simple Rescorla-Wagner RL model. We discuss common pitfalls in its application and provide practical suggestions. First, with simulation, we unpack the functional role of the learning rate and pinpoint what could easily go wrong when interpreting differences in the learning rate. Then, we discuss the inevitable collinearity between outcome and prediction error in RL models and provide suggestions of how to justify whether the observed neural activation is related to the prediction error rather than outcome valence. Finally, we suggest posterior predictive check is a crucial step after model comparison, and we articulate employing hierarchical modeling for parameter estimation. We aim to provide simple and scalable explanations and practical guidelines for employing RL models to assist both beginners and advanced users in better implementing and interpreting their model-based analyses.

Catecholaminergic modulation of meta-learning

The remarkable expedience of human learning is thought to be underpinned by meta-learning, whereby slow accumulative learning processes are rapidly adjusted to the current learning environment. To date, the neurobiological implementation of meta-learning remains unclear. A burgeoning literature argues for an important role for the catecholamines dopamine and noradrenaline in meta-learning. Here, we tested the hypothesis that enhancing catecholamine function modulates the ability to optimise a meta-learning parameter (learning rate) as a function of environmental volatility. 102 participants completed a task which required learning in stable phases, where the probability of reinforcement was constant, and volatile phases, where probabilities changed every 10-30 trials. The catecholamine transporter blocker methylphenidate enhanced participants' ability to adapt learning rate: Under methylphenidate, compared with placebo, participants exhibited higher learning rates in volatile relative to stable phases. Furthermore, this effect was significant only with respect to direct learning based on the participants' own experience, there was no significant effect on inferred-value learning where stimulus values had to be inferred. These data demonstrate a causal link between catecholaminergic modulation and the adjustment of the meta-learning parameter learning rate.

The Effect of Reduced Learning Ability on Avoidance in Psychopathy: A Computational Approach

Individuals with psychopathy often show deficits in learning, which often have negative consequences. Several theories have been proposed to explain psychopathic behaviors, but the learning mechanisms in psychopathy are still unclear. To clarify the learning anomalies in psychopathy, we fitted reinforcement learning (RL) models to behavioral data. We conducted two experiments to examine the effect of psychopathy as a group difference (Experiment 1) and as a continuum (Experiment 2). Forty-three undergraduates (in Experiment 1) and fifty-five undergraduate and graduate students (in Experiment 2) performed a go/no-go based learning task with accompanying rewards or punishments. Although we observed no differences in learning performance among the levels of psychopathic traits, the learning rate for the positive prediction error in the loss domain was lower for those with high-psychopathic trait than for those with low-psychopathic trait. This finding indicates that individuals with high-psychopathic traits update an action value less when they avoid a negative outcome. Our model can represent previous theories under a computational framework and provide a new perspective on impaired learning in psychopathy.

Social anxiety and dynamic social reinforcement learning in a volatile environment

Adaptive social behavior requires learning probabilities of social reward and punishment, and updating these probabilities when they change. Given prior research on aberrant reinforcement learning in affective disorders, this study examines how social anxiety affects probabilistic social reinforcement learning and dynamic updating of learned probabilities in a volatile environment. N=222 online participants completed questionnaires and a computerized ball-catching game with changing probabilities of reward and punishment. Dynamic learning rates were estimated to assess the relative importance ascribed to new information in response to volatility. Mixed-effects regression was used to analyze throw patterns as a function of social anxiety symptoms. Higher social anxiety predicted fewer throws to the previously punishing avatar and different learning rates after certain role changes, suggesting that social anxiety may be characterized by difficulty updating learned social probabilities. Socially anxious individuals may miss the chance to learn that a once-punishing situation no longer poses a threat.

Sensorimotor priors are effector dependent

During sensorimotor tasks, subjects use sensory feedback but also prior information. It is often assumed that the sensorimotor prior is just given by the experiment and that the details for acquiring this prior (e.g., the effector) are irrelevant. However, recent research has suggested that the construction of priors is nontrivial. To test if the sensorimotor details matter for the construction of a prior, we designed two tasks that differ only in the effectors that subjects use to indicate their estimate. For both a typical reaching setting and an atypical wrist rotation setting, prior and feedback uncertainty matter as quantitatively predicted by Bayesian statistics. However, in violation of simple Bayesian models, the importance of the prior differs across effectors. Subjects overly rely on their prior in the typical reaching case compared with the wrist case. The brain is not naively Bayesian with a single and veridical prior. NEW & NOTEWORTHY Traditional Bayesian models often assume that we learn statistics of movements and use the information as a prior to guide subsequent movements. The effector is merely a reporting modality for information processing. We asked subjects to perform a visuomotor learning task with different effectors (finger or wrist). Surprisingly, we found that prior information is used differently between the effectors, suggesting that learning of the prior is related to the movement context such as the effector involved or that naive models of Bayesian behavior need to be extended.

Robust Adaptive Recurrent Cerebellar Model Neural Network for Non-linear System Based on GPSO

A robust adaptive recurrent cerebellar model articulation controller (RARC) neural network for non-linear systems using the genetic particle swarm optimization (GPSO) algorithm is presented in this study. The RARC is used as the principal tracking controller and the robust compensation controller is designed to recover the residual of the approximation error. In the RARC neural network, the steepest descent gradient method and the Lyapunov function are used for deriving the adaptive law parameter of the system. Besides, the learning rates play an important role in these adaptive laws and they have a great effect on the functions of control systems. In this paper, the combination of the genetic algorithm with the mutation particle swarm optimization algorithm is applied to seek for the optimal learning rates of the RARC adaptation laws. The numerical simulations about the inverted pendulum system as well as the robot manipulator system are given to confirm the effectiveness and practicability of the GPSO-RARC-based control system. Compared with other control schemes, the proposed control scheme is testified to be reliable and can obtain the optimal parameter about the learning rates and the minimum root mean square error for non-linear systems.

Cortical neural modulation by previous trial outcome differentiates fast- from slow-learning rats on a visuomotor directional choice task

To better understand the neural cortical underpinnings that explain behavioral differences in learning rate, we recorded single-unit activity in primary motor (M1) and secondary motor (M2) areas while rats learned to perform a directional (left or right) operant visuomotor association task. Analysis of neural activity during the early portion of the cue period showed that neural modulation in the motor cortex was most strongly associated with two task factors: the previous trial outcome (success or error) and the current trial's directional choice (left or right). Furthermore, the fast learners, defined as those who had steeper learning curves and required fewer learning sessions to reach criterion performance, encoded the previous trial outcome factor more strongly than the directional choice factor. Conversely, the slow learners encoded directional choice more strongly than previous trial outcome. These differences in task factor encoding were observed in both the percentage of neurons and the neural modulation depth. These results suggest that fast learning is accompanied by a stronger component of previous trial outcome in the modulation representation present in motor cortex, which therefore may be a contributing factor to behavioral differences in learning rate. NEW & NOTEWORTHY We chronically recorded neural activity as rats learned a visuomotor directional choice task from a naive state. Learning rates varied. Single-unit neural modulation of two motor areas revealed that the fast learners encoded previous trial outcome more strongly than directional choice, whereas the reverse was true for slow learners. This finding provides novel evidence that rat learning rate is strongly correlated with the strength of neural modulation by previous trial outcome in motor cortex.

Learning Efficiency: Identifying Individual Differences in Learning Rate and Retention in Healthy Adults

People differ in how quickly they learn information and how long they remember it, yet individual differences in learning abilities within healthy adults have been relatively neglected. In two studies, we examined the relation between learning rate and subsequent retention using a new foreign-language paired-associates task (the learning-efficiency task), which was designed to eliminate ceiling effects that often accompany standardized tests of learning and memory in healthy adults. A key finding was that quicker learners were also more durable learners (i.e., exhibited better retention across a delay), despite studying the material for less time. Additionally, measures of learning and memory from this task were reliable in Study 1 ( N = 281) across 30 hr and Study 2 ( N = 92; follow-up n = 46) across 3 years. We conclude that people vary in how efficiently they learn, and we describe a reliable and valid method for assessing learning efficiency within healthy adults.

Functions of Learning Rate in Adaptive Reward Learning

As a crucial cognitive function, learning applies prediction error (the discrepancy between the prediction from learning and the world state) to adjust predictions of the future. How much prediction error affects this adjustment also depends on the learning rate. Our understanding to the learning rate is still limited, in terms of (1) how it is modulated by other factors, and (2) the specific mechanisms of how learning rate interacts with prediction error to update learning. We applied computational modeling and functional magnetic resonance imaging to investigate these issues. We found that, when human participants performed a reward learning task, reward magnitude modulated learning rate. Modulation strength further predicted the difference in behavior following high vs. low reward across subjects. Imaging results further showed that this modulation was reflected in brain regions where the reward feedback is also encoded, such as the medial prefrontal cortex (MFC), precuneus, and posterior cingulate cortex. Furthermore, for the first time, we observed that the integration of the learning rate and the reward prediction error was represented in MFC activity. These findings extend our understanding of adaptive learning by demonstrating how it functions in a chain reaction of prediction updating.

Metaplasticity as a Neural Substrate for Adaptive Learning and Choice under Uncertainty

Value-based decision making often involves integration of reward outcomes over time, but this becomes considerably more challenging if reward assignments on alternative options are probabilistic and non-stationary. Despite the existence of various models for optimally integrating reward under uncertainty, the underlying neural mechanisms are still unknown. Here we propose that reward-dependent metaplasticity (RDMP) can provide a plausible mechanism for both integration of reward under uncertainty and estimation of uncertainty itself. We show that a model based on RDMP can robustly perform the probabilistic reversal learning task via dynamic adjustment of learning based on reward feedback, while changes in its activity signal unexpected uncertainty. The model predicts time-dependent and choice-specific learning rates that strongly depend on reward history. Key predictions from this model were confirmed with behavioral data from non-human primates. Overall, our results suggest that metaplasticity can provide a neural substrate for adaptive learning and choice under uncertainty.

Disruption of the Right Temporoparietal Junction Impairs Probabilistic Belief Updating

Generating and updating probabilistic models of the environment is a fundamental modus operandi of the human brain. Although crucial for various cognitive functions, the neural mechanisms of these inference processes remain to be elucidated. Here, we show the causal involvement of the right temporoparietal junction (rTPJ) in updating probabilistic beliefs and we provide new insights into the chronometry of the process by combining online transcranial magnetic stimulation (TMS) with computational modeling of behavioral responses. Female and male participants performed a modified location-cueing paradigm, where false information about the percentage of cue validity (%CV) was provided in half of the experimental blocks to prompt updating of prior expectations. Online double-pulse TMS over rTPJ 300 ms (but not 50 ms) after target appearance selectively decreased participants' updating of false prior beliefs concerning %CV, reflected in a decreased learning rate of a Rescorla-Wagner model. Online TMS over rTPJ also impacted on participants' explicit beliefs, causing them to overestimate %CV. These results confirm the involvement of rTPJ in updating of probabilistic beliefs, thereby advancing our understanding of this area's function during cognitive processing.SIGNIFICANCE STATEMENT Contemporary views propose that the brain maintains probabilistic models of the world to minimize surprise about sensory inputs. Here, we provide evidence that the right temporoparietal junction (rTPJ) is causally involved in this process. Because neuroimaging has suggested that rTPJ is implicated in divergent cognitive domains, the demonstration of an involvement in updating internal models provides a novel unifying explanation for these findings. We used computational modeling to characterize how participants change their beliefs after new observations. By interfering with rTPJ activity through online transcranial magnetic stimulation, we showed that participants were less able to update prior beliefs with TMS delivered at 300 ms after target onset.

Decision making in healthy participants on the Iowa Gambling Task: new insights from an operant approach

The Iowa Gambling Task (IGT) has contributed greatly to the study of affective decision making. However, researchers have observed high inter-study and inter-individual variability in IGT performance in healthy participants, and many are classified as impaired using standard criteria. Additionally, while decision-making deficits are often attributed to atypical sensitivity to reward and/or punishment, the IGT lacks an integrated sensitivity measure. Adopting an operant perspective, two experiments were conducted to explore these issues. In Experiment 1, 50 healthy participants completed a 200-trial version of the IGT which otherwise closely emulated Bechara et al.'s (1999) original computer task. Group data for Trials 1–100 closely replicated Bechara et al.'s original findings of high net scores and preferences for advantageous decks, suggesting that implementations that depart significantly from Bechara's standard IGT contribute to inter-study variability. During Trials 101–200, mean net scores improved significantly and the percentage of participants meeting the “impaired” criterion was halved. An operant-style stability criterion applied to individual data revealed this was likely related to individual differences in learning rate. Experiment 2 used a novel operant card task—the Auckland Card Task (ACT)—to derive quantitative estimates of sensitivity using the generalized matching law. Relative to individuals who mastered the IGT, persistent poor performers on the IGT exhibited significantly lower sensitivity to magnitudes (but not frequencies) of rewards and punishers on the ACT. Overall, our findings demonstrate the utility of operant-style analysis of IGT data and the potential of applying operant concurrent-schedule procedures to the study of human decision making.

Dual learning processes underlying human decision-making in reversal learning tasks: functional significance and evidence from the model fit to human behavior

Humans are capable of correcting their actions based on actions performed in the past, and this ability enables them to adapt to a changing environment. The computational field of reinforcement learning (RL) has provided a powerful explanation for understanding such processes. Recently, the dual learning system, modeled as a hybrid model that incorporates value update based on reward-prediction error and learning rate modulation based on the surprise signal, has gained attention as a model for explaining various neural signals. However, the functional significance of the hybrid model has not been established. In the present study, we used computer simulation in a reversal learning task to address functional significance in a probabilistic reversal learning task. The hybrid model was found to perform better than the standard RL model in a large parameter setting. These results suggest that the hybrid model is more robust against the mistuning of parameters compared with the standard RL model when decision-makers continue to learn stimulus-reward contingencies, which can create abrupt changes. The parameter fitting results also indicated that the hybrid model fit better than the standard RL model for more than 50% of the participants, which suggests that the hybrid model has more explanatory power for the behavioral data than the standard RL model.

The influence of the noradrenergic system on optimal control of neural plasticity

Decision making under uncertainty is challenging for any autonomous agent. The challenge increases when the environment's stochastic properties change over time, i.e., when the environment is volatile. In order to efficiently adapt to volatile environments, agents must primarily rely on recent outcomes to quickly change their decision strategies; in other words, they need to increase their knowledge plasticity. On the contrary, in stable environments, knowledge stability must be preferred to preserve useful information against noise. Here we propose that in mammalian brain, the locus coeruleus (LC) is one of the nuclei involved in volatility estimation and in the subsequent control of neural plasticity. During a reinforcement learning task, LC activation, measured by means of pupil diameter, coded both for environmental volatility and learning rate. We hypothesize that LC could be responsible, through norepinephrinic modulation, for adaptations to optimize decision making in volatile environments. We also suggest a computational model on the interaction between the anterior cingulate cortex (ACC) and LC for volatility estimation.

Differences in learning rates for item and associative memories between amnestic mild cognitive impairment and healthy controls

BACKGROUND

It has been established that the overall performance of associative memory was disproportionately impaired in contrast to item memory in aMCI (Amnestic mild cognitive impairment) patients, but little is known about the specific aspects of the memory process that show differences between aMCI and healthy controls. By comparing an item-item associative learning test with an individual item learning test, the present study investigated whether the rate of learning was slower in associative memory than in item memory in aMCI. Furthermore, we examined whether deficits in intertrial acquisition and consolidation contributed to the potential disproportionate impairments in the learning rate of associative memory for aMCI patients. In addition, we further explored whether the aMCI-discriminative power of the associative memory test increases more than that of the item memory test when the number of learning-test trials increases.

METHODS

A group of 40 aMCI patients and 40 matched control participants were administered a standardized item memory test (Auditory Verbal Learning Test, AVLT) and a standardized associative memory test (Paired Associative Learning Test, PALT), as well as other neuropsychological tests and clinical assessments.

RESULTS

The results indicated that the learning rate deficits in aMCI patients were more obvious for associative memory than for item memory and that the deficits resulted from impairments in both intertrial acquisition and consolidation. In addition, the receiver operating characteristic curve and logistical regression analysis revealed that the discriminative power of the associative memory test for aMCI was larger than that of the item memory test, especially with more than one learning-test trials.

CONCLUSIONS

Due to more deficits in learning rate of associative memory than that of item memory, the discriminative power for aMCI tended to be larger in associative memory than in item memory when the number of learning-test trials increased. It is suggested that associative memory tests with multiple trials may be particularly useful for early detection of aMCI.