Testing the role of reward and punishment sensitivity in avoidance behavior: a computational modeling approach

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

Artificial neural networks for model identification and parameter estimation in computational cognitive models

Computational cognitive models have been used extensively to formalize cognitive processes. Model parameters offer a simple way to quantify individual differences in how humans process information. Similarly, model comparison allows researchers to identify which theories, embedded in different models, provide the best accounts of the data. Cognitive modeling uses statistical tools to quantitatively relate models to data that often rely on computing/estimating the likelihood of the data under the model. However, this likelihood is computationally intractable for a substantial number of models. These relevant models may embody reasonable theories of cognition, but are often under-explored due to the limited range of tools available to relate them to data. We contribute to filling this gap in a simple way using artificial neural networks (ANNs) to map data directly onto model identity and parameters, bypassing the likelihood estimation. We test our instantiation of an ANN as a cognitive model fitting tool on classes of cognitive models with strong inter-trial dependencies (such as reinforcement learning models), which offer unique challenges to most methods. We show that we can adequately perform both parameter estimation and model identification using our ANN approach, including for models that cannot be fit using traditional likelihood-based methods. We further discuss our work in the context of the ongoing research leveraging simulation-based approaches to parameter estimation and model identification, and how these approaches broaden the class of cognitive models researchers can quantitatively investigate.

Tackling Costly Fearful Avoidance Using Pavlovian Counterconditioning

Avoidance behavior constitutes a major transdiagnostic symptom that exacerbates anxiety. It hampers fear extinction and predicts poor therapy-outcome. Pavlovian counterconditioning with a reward could alleviate avoidance better than traditional extinction by reducing negative valence of the feared situation. However, previous studies are scarce and did not consider that pathological avoidance is often costly and typically evolves from an approach-avoidance conflict. Therefore, we used an approach-avoidance conflict paradigm to model effects of counterconditioning on costly avoidance (i.e., avoidance that leads to missing out on rewards). Results from our preregistered Bayesian Mixed Model analyses in 51 healthy participants (43 females) indicated that counterconditioning was more effective in reducing negative valuation and decreasing costly avoidance than traditional extinction. This study supports application of a simple counterconditioning technique, shows that its efficacy transfers to more complex avoidance situations, and suggests treatment may benefit from increasing reward drive in combination with extinction to overcome avoidance. Application in a clinical sample is a necessary next step to assess clinical utility of counterconditioning.

The empirical status of predictive coding and active inference

Research on predictive processing models has focused largely on two specific algorithmic theories: Predictive Coding for perception and Active Inference for decision-making. While these interconnected theories possess broad explanatory potential, they have only recently begun to receive direct empirical evaluation. Here, we review recent studies of Predictive Coding and Active Inference with a focus on evaluating the degree to which they are empirically supported. For Predictive Coding, we find that existing empirical evidence offers modest support. However, some positive results can also be explained by alternative feedforward (e.g., feature detection-based) models. For Active Inference, most empirical studies have focused on fitting these models to behavior as a means of identifying and explaining individual or group differences. While Active Inference models tend to explain behavioral data reasonably well, there has not been a focus on testing empirical validity of active inference theory per se, which would require formal comparison to other models (e.g., non-Bayesian or model-free reinforcement learning models). This review suggests that, while promising, a number of specific research directions are still necessary to evaluate the empirical adequacy and explanatory power of these algorithms.

Measuring maladaptive avoidance: from animal models to clinical anxiety

Avoiding stimuli that predict danger is required for survival. However, avoidance can become maladaptive in individuals who overestimate threat and thus avoid safe situations as well as dangerous ones. Excessive avoidance is a core feature of anxiety disorders, post-traumatic stress disorder (PTSD), and obsessive-compulsive disorder (OCD). This avoidance prevents patients from confronting maladaptive threat beliefs, thereby maintaining disordered anxiety. Avoidance is associated with high levels of psychosocial impairment yet is poorly understood at a mechanistic level. Many objective laboratory assessments of avoidance measure adaptive avoidance, in which an individual learns to successfully avoid a truly noxious stimulus. However, anxiety disorders are characterized by maladaptive avoidance, for which there are fewer objective laboratory measures. We posit that maladaptive avoidance behavior depends on a combination of three altered neurobehavioral processes: (1) threat appraisal, (2) habitual avoidance, and (3) trait avoidance tendency. This heterogeneity in underlying processes presents challenges to the objective measurement of maladaptive avoidance behavior. Here we first review existing paradigms for measuring avoidance behavior and its underlying neural mechanisms in both human and animal models, and identify how existing paradigms relate to these neurobehavioral processes. We then propose a new framework to improve the translational understanding of maladaptive avoidance behavior by adapting paradigms to better differentiate underlying processes and mechanisms and applying these paradigms in clinical populations across diagnoses with the goal of developing novel interventions to engage specific identified neurobehavioral targets.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Greater decision uncertainty characterizes a transdiagnostic patient sample during approach-avoidance conflict: a computational modelling approach

BACKGROUND

Imbalances in approach-avoidance conflict (AAC) decision-making (e.g., sacrificing rewards to avoid negative outcomes) are considered central to multiple psychiatric disorders. We used computational modelling to examine 2 factors that are often not distinguished in descriptive analyses of AAC: decision uncertainty and sensitivity to negative outcomes versus rewards (emotional conflict).

METHODS

A previously validated AAC task was completed by 478 participants, including healthy controls (n = 59), people with substance use disorders (n = 159) and people with depression and/or anxiety disorders who did not have substance use disorders (n = 260). Using an active inference model, we estimated individual-level values for a model parameter that reflected decision uncertainty and another that reflected emotional conflict. We also repeated analyses in a subsample (59 healthy controls, 161 people with depression and/or anxiety disorders, 56 people with substance use disorders) that was propensity-matched for age and general intelligence.

RESULTS

The model showed high accuracy (72%). As further validation, parameters correlated with reaction times and self-reported task motivations in expected directions. The emotional conflict parameter further correlated with self-reported anxiety during the task (r = 0.32, p < 0.001), and the decision uncertainty parameter correlated with self-reported difficulty making decisions (r = 0.45, p < 0.001). Compared to healthy controls, people with depression and/or anxiety disorders and people with substance use disorders showed higher decision uncertainty in the propensity-matched sample (t = 2.16, p = 0.03, and t = 2.88, p = 0.005, respectively), with analogous results in the full sample; people with substance use disorders also showed lower emotional conflict in the full sample (t = 3.17, p = 0.002).

LIMITATIONS

This study was limited by heterogeneity of the clinical sample and an inability to examine learning.

CONCLUSION

These results suggest that reduced confidence in how to act, rather than increased emotional conflict, may explain maladaptive approach-avoidance behaviours in people with psychiatric disorders.

Individual differences in costly fearful avoidance and the relation to psychophysiology

Excessive avoidance behaviour is a cardinal symptom of anxiety disorders. Avoidance is not only associated with the benefits of avoiding threats, but also with the costs of missing out on rewards upon exploration. Psychological and psychophysiological mechanisms contributing to these costly avoidance decisions in prospect of mixed outcomes remain unclear. We developed a novel Fearful Avoidance Task (FAT) that resembles characteristics of real-life approach-avoidance conflicts, enabling to disentangle reward and threat effects. Using the FAT, we investigated individual differences in avoidance behaviour and anticipatory psychophysiological states (i.e. startle reflex and skin conductance) in a relatively large sample of 343 (78 females) participants. Avoidance under acute threat of shock depends on a trade-off between perceived reward and threat. Both increased startle and skin conductance in the absence of threat of shock emerged as predictors of increased avoidance (potentially indicative of fear generalization). Increased avoidance was also associated with female sex and trait anxiety, dependent on reward and threat levels. Our findings highlight distinct possible predictors of heightened avoidance and add to mechanistic understanding of how individual propensity for costly avoidance may emerge. Distinct avoidance typologies based on differential reward and threat sensitivities may have different mechanistic origins and thereby could benefit from different treatment strategies.

Cardiorespiratory Response to Moderate Hypercapnia in Female College Students Expressing Behaviorally Inhibited Temperament

Behaviorally inhibited (BI) temperament is marked by heightened behavioral sensitivity to environmental threats. The degree to which threat sensitivity is reflected in cardiorespiratory responses has been relatively unexplored. Female college students were exposed to modest hypercapnia (7.0% CO₂) or ambient air (AA) while engaging in a computerized task with cued reinforcement features. All physiological variables except for blood pressure were processed in 4 min epochs corresponding to pre-exposure, exposure, and post-exposure. Primary respiratory measures were respiratory frequency (f_b), tidal volume (V_T), and minute ventilation (V_E). Electrocardiograms (ECGs) were processed using ARTiiFACT software with resultant heart rate variability (HRV) measures in the frequency domain and time domain. Consistent with the literature, modest hypercapnia increased V_T, F_b, and V_E. No differences in respiratory parameters were detected between BI and non-behaviorally inhibited individuals (NI). For HRV in the time domain, RMSSD and NN50 values increased during CO₂ inhalation which then returned to pre-exposure levels after CO₂ cessation. Hypercapnia increased high frequency (HF) power which then recovered. BI exhibited reduced low frequency (LF) power during the pre-exposure period. For NI, LF power reduced over the subsequent phases ameliorating differences between BI and NI. Hypercapnia improved the task performance of BI. This is the largest study of female reactivity to hypercapnia and associated HRV to date. In general, hypercapnia increased time domain HRV and HF power, suggesting a strong vagal influence. Those expressing BI exhibited similar respiratory and HRV reactivity to NI despite inherently reduced LF power. Although 7% CO₂ represents a mild challenge to the respiratory and cardiovascular systems, it is nonetheless sufficient to explore inherent difference in stress reactivity in those vulnerable to develop anxiety disorders.

Quantifying Human Avoidance Behavior in Immersive Virtual Reality

Avoidance behavior is a key symptom of most anxiety disorders and a central readout in animal research. However, the quantification of real-life avoidance behavior in humans is typically restricted to clinical populations, who show actual avoidance of phobic objects. In experimental approaches for healthy participants, many avoidance tasks utilize button responses or a joystick navigation on the screen as indicators of avoidance behavior. To allow the ecologically valid assessment of avoidance behavior in healthy participants, we developed a new automated immersive Virtual Reality paradigm, where participants could freely navigate in virtual 3-dimensional, 360-degrees scenes by real naturalistic body movements. A differential fear conditioning procedure was followed by three newly developed behavioral tasks to assess participants’ avoidance behavior of the conditioned stimuli: an approach, a forced-choice, and a search task. They varied in instructions, degrees of freedom, and high or low task-related relevance of the stimuli. We initially examined the tasks in a quasi-experiment (N = 55), with four consecutive runs and various experimental adaptations. Here, although we observed avoidance behavior in all three tasks after additional reinforcement, we only detected fear-conditioned avoidance behavior in the behavioral forced-choice and search tasks. These findings were largely replicated in a confirmatory experiment (N = 72) with randomized group allocation, except that fear-conditioned avoidance behavior was only manifest in the behavioral search task. This supports the notion that the behavioral search task is sensitive to detect avoidance behavior after fear conditioning only, whereas the behavioral approach and forced-choice tasks are still able to detect “strong” avoidance behavior after fear conditioning and additional reinforcement.

Predictors of risky foraging behaviour in healthy young people

During adolescence and early adulthood, learning when to avoid threats and when to pursue rewards becomes crucial. Using a risky foraging task, we investigated individual differences in this dynamic across 781 individuals aged 14-24 years who were split into a hypothesis-generating discovery sample and a hold-out confirmation sample. Sex was the most important predictor of cautious behaviour and performance. Males earned one standard deviation (or 20%) more reward than females, collected more reward when there was little to lose and reduced foraging to the same level as females when potential losses became high. Other independent predictors of cautiousness and performance were self-reported daringness, IQ and self-reported cognitive complexity. We found no evidence for an impact of age or maturation. Thus, maleness, a high IQ or self-reported cognitive complexity, and self-reported daringness predicted greater success in risky foraging, possibly due to better exploitation of low-risk opportunities in high-risk environments.

Sex differences and personality in the modulation of the acoustic startle reflex

The modulation of the eyeblink component of the acoustic startle reflex (ASR) has been used to study human motivation, attention, and emotion towards affective stimuli of different valence. However, sex and individual differences in personality have been rather overlooked concerning the change in the ASR to brief affective sequences. In this study, we aimed to evaluate sex differences in the ASR, together with the influence of sensitivity to punishment (SP) and sensitivity to reward (SR) in the affective modulation of the ASR to pleasant and unpleasant pictures. We addressed this topic with a latent curve model (LCM) representing the change in the ASR of an extensive group of men (n = 166) and women (n = 109). There was a significant habituation of the ASR to the pleasant pictures, and a significant sensitization of the ASR to the unpleasant pictures. Both effects were higher and more variable for women than for men. There were in addition interactive and quadratic effects of SP and SR on the ASR to the pleasant and unpleasant pictures. Men and women with extreme scores in SP, and women with low scores in SR habituated faster to the pleasant stimuli. For men scoring low in SP, higher scores in SR related with an attenuated initial ASR to the unpleasant stimuli. Women with extreme scores in SP had a higher initial ASR to the unpleasant stimuli. There were remarkable asymmetries between men and women concerning personality effects on the change in the ASR to affective stimuli.

Rethinking avoidance: Toward a balanced approach to avoidance in treating anxiety disorders

Avoidance is typically considered a maladaptive behavioral response to excessive fear and anxiety, leading to the maintenance of anxiety disorders. Exposure is a core element of cognitive-behavioral therapy for anxiety disorders. One important aspect of this treatment is repeated and prolonged exposure to a threat while discouraging patients from using avoidance strategies, such as escape or safety behaviors. We will first revisit the role of avoidance learning in the development and maintenance of anxiety disorders, including important insights from the neuroscience literature. Next, we will consider both the negative and positive aspects of avoidance for therapeutic interventions. Finally, we will explore the application of adaptive avoidance in exposure therapy for anxiety disorders. We will argue that there are occasions when avoidance behaviors can serve as effective coping strategies to enhance the person's perception of control over the environment and the potential threat. We conclude that avoidance behaviors can be a valuable therapeutic element, depending on the function of these behaviors.

Greater avoidance behavior in individuals with posttraumatic stress disorder symptoms

While avoidance is a core symptom of PTSD, little is known about whether individuals with PTSD show a general cognitive bias to acquire and express avoidance, in situations not related to trauma or fear. Here, we used a computer-based task to examine operant acquisition and extinction of avoidance in participants with and without severe self-reported PTSD symptoms. A total of 119 participants (77 male, 42 female; 74 veteran, 45 civilian) with symptoms (PTSS; n = 63) or with few/no symptoms (noPTSS; n = 56) performed a task, in which they controlled a spaceship and could shoot a target to gain points or hide in "safe areas" to escape or avoid on-screen aversive events. Results show that participants with PTSS exhibited more avoidance across trials than noPTSS participants, particularly due to more avoidance behavior in PTSS females compared to noPTSS females. Avoidance behavior decreased across extinction trials but interactions with PTSS and gender fell short of significance. Overall, PTSD symptoms were associated with propensity to acquire and express avoidance behavior, in both civilians and veterans, and even in a cognitive task that does not explicitly involve trauma or fear. This effect was more pronounced in females, highlighting the role of gender differences in PTSD symptomatology. Importantly, this study also demonstrates the potential of an objective assessment of avoidance behavior, which could be used to supplement the common but limited self-report tools.

Development of a dynamic computational model of social cognitive theory

Social cognitive theory (SCT) is among the most influential theories of behavior change and has been used as the conceptual basis of health behavior interventions for smoking cessation, weight management, and other health behaviors. SCT and other behavior theories were developed primarily to explain differences between individuals, but explanatory theories of within-person behavioral variability are increasingly needed as new technologies allow for intensive longitudinal measures and interventions adapted from these inputs. These within-person explanatory theoretical applications can be modeled as dynamical systems. SCT constructs, such as reciprocal determinism, are inherently dynamical in nature, but SCT has not been modeled as a dynamical system. This paper describes the development of a dynamical system model of SCT using fluid analogies and control systems principles drawn from engineering. Simulations of this model were performed to assess if the model performed as predicted based on theory and empirical studies of SCT. This initial model generates precise and testable quantitative predictions for future intensive longitudinal research. Dynamic modeling approaches provide a rigorous method for advancing health behavior theory development and refinement and for guiding the development of more potent and efficient interventions.

On the Complexity of Brain Disorders: A Symptom-Based Approach

Mounting evidence shows that brain disorders involve multiple and different neural dysfunctions, including regional brain damage, change to cell structure, chemical imbalance, and/or connectivity loss among different brain regions. Understanding the complexity of brain disorders can help us map these neural dysfunctions to different symptom clusters as well as understand subcategories of different brain disorders. Here, we discuss data on the mapping of symptom clusters to different neural dysfunctions using examples from brain disorders such as major depressive disorder (MDD), Parkinson’s disease (PD), schizophrenia, posttraumatic stress disorder (PTSD) and Alzheimer’s disease (AD). In addition, we discuss data on the similarities of symptoms in different disorders. Importantly, computational modeling work may be able to shed light on plausible links between various symptoms and neural damage in brain disorders.

Sex differences in a rat model of risky decision making

Many debilitating psychiatric conditions, including drug addiction, are characterized by poor decision making and maladaptive risk-taking. Recent research has begun to probe this relationship to determine how brain mechanisms mediating risk-taking become compromised after chronic drug use. Currently, however, the majority of work in this field has used male subjects. Given the well-established sex differences in drug addiction, it is conceivable that such differences are also evident in risk-based decision making. To test this possibility, male and female adult rats were trained in a risky decision making task (RDT), in which they chose between a small, "safe" food reward and a large, "risky" food reward accompanied by an increasing probability of mild footshock punishment. Consistent with findings in human subjects, females were more risk averse, choosing the large, risky reward significantly less than males. This effect was not due to differences in shock reactivity or body weight, and risk-taking in females was not modulated by estrous phase. Systemic amphetamine administration decreased risk-taking in both males and females; however, females exhibited greater sensitivity to amphetamine, suggesting that dopaminergic signaling may partially account for sex differences in risk-taking. Finally, although males displayed greater instrumental responding for food reward, reward choice in the RDT was not affected by satiation, indicating that differences in motivation to obtain food reward cannot fully account for sex differences in risk-taking. These results should prove useful for developing targeted treatments for psychiatric conditions in which risk-taking is altered and that are known to differentially affect males and females.

Probabilistic reward- and punishment-based learning in opioid addiction: Experimental and computational data

Addiction is the continuation of a habit in spite of negative consequences. A vast literature gives evidence that this poor decision-making behavior in individuals addicted to drugs also generalizes to laboratory decision making tasks, suggesting that the impairment in decision-making is not limited to decisions about taking drugs. In the current experiment, opioid-addicted individuals and matched controls with no history of illicit drug use were administered a probabilistic classification task that embeds both reward-based and punishment-based learning trials, and a computational model of decision making was applied to understand the mechanisms describing individuals' performance on the task. Although behavioral results showed that opioid-addicted individuals performed as well as controls on both reward- and punishment-based learning, the modeling results suggested subtle differences in how decisions were made between the two groups. Specifically, the opioid-addicted group showed decreased tendency to repeat prior responses, meaning that they were more likely to "chase reward" when expectancies were violated, whereas controls were more likely to stick with a previously-successful response rule, despite occasional expectancy violations. This tendency to chase short-term reward, potentially at the expense of developing rules that maximize reward over the long term, may be a contributing factor to opioid addiction. Further work is indicated to better understand whether this tendency arises as a result of brain changes in the wake of continued opioid use/abuse, or might be a pre-existing factor that may contribute to risk for addiction.

Using signals associated with safety in avoidance learning: computational model of sex differences

Avoidance behavior involves learning responses that prevent upcoming aversive events; these responses typically extinguish when the aversive events stop materializing. Stimuli that signal safety from aversive events can paradoxically inhibit extinction of avoidance behavior. In animals, males and females process safety signals differently. These differences help explain why women are more likely to be diagnosed with an anxiety disorder and exhibit differences in symptom presentation and course compared to men. In the current study, we extend an existing model of strain differences in avoidance behavior to simulate sex differences in rats. The model successfully replicates data showing that the omission of a signal associated with a period of safety can facilitate extinction in females, but not males, and makes novel predictions that this effect should depend on the duration of the period, the duration of the signal itself, and its occurrence within that period. Non-reinforced responses during the safe period were also found to be important in the expression of these patterns. The model also allowed us to explore underlying mechanisms for the observed sex effects, such as whether safety signals serve as occasion setters for aversive events, to determine why removing them can facilitate extinction of avoidance. The simulation results argue against this account, and instead suggest the signal may serve as a conditioned reinforcer of avoidance behavior.