1
|
Rosenberg BM, Barnes-Horowitz NM, Zbozinek TD, Craske MG. Reward processes in extinction learning and applications to exposure therapy. J Anxiety Disord 2024; 106:102911. [PMID: 39128178 PMCID: PMC11384290 DOI: 10.1016/j.janxdis.2024.102911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 10/25/2023] [Revised: 07/08/2024] [Accepted: 07/26/2024] [Indexed: 08/13/2024]
Abstract
Anxiety disorders are common and highly distressing mental health conditions. Exposure therapy is a gold-standard treatment for anxiety disorders. Mechanisms of Pavlovian fear learning, and particularly fear extinction, are central to exposure therapy. A growing body of evidence suggests an important role of reward processes during Pavlovian fear extinction. Nonetheless, predominant models of exposure therapy do not currently incorporate reward processes. Herein, we present a theoretical model of reward processes in relation to Pavlovian mechanisms of exposure therapy, including a focus on dopaminergic prediction error signaling, coinciding positive emotional experiences (i.e., relief), and unexpected positive outcomes. We then highlight avenues for further research and discuss potential strategies to leverage reward processes to maximize exposure therapy response, such as pre-exposure interventions to increase reward sensitivity or post-exposure rehearsal (e.g., savoring, imaginal recounting strategies) to enhance retrieval and retention of learned associations.
Collapse
Affiliation(s)
- Benjamin M Rosenberg
- Department of Psychology, University of California, Los Angeles (UCLA), Los Angeles, CA, USA.
| | - Nora M Barnes-Horowitz
- Department of Psychology, University of California, Los Angeles (UCLA), Los Angeles, CA, USA.
| | - Tomislav D Zbozinek
- Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles (UCLA), Los Angeles, CA, USA
| | - Michelle G Craske
- Department of Psychology, University of California, Los Angeles (UCLA), Los Angeles, CA, USA; Department of Psychiatry and Biobehavioral Sciences, University of California, Los Angeles (UCLA), Los Angeles, CA, USA
| |
Collapse
|
2
|
Hassall CD, Yan Y, Hunt LT. The neural correlates of continuous feedback processing. Psychophysiology 2023; 60:e14399. [PMID: 37485986 PMCID: PMC10851313 DOI: 10.1111/psyp.14399] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2022] [Revised: 07/12/2023] [Accepted: 07/12/2023] [Indexed: 07/25/2023]
Abstract
Feedback processing is commonly studied by analyzing the brain's response to discrete rather than continuous events. Such studies have led to the hypothesis that rapid phasic midbrain dopaminergic activity tracks reward prediction errors (RPEs), the effects of which are measurable at the scalp via electroencephalography (EEG). Although studies using continuous feedback are sparse, recent animal work suggests that moment-to-moment changes in reward are tracked by slowly ramping midbrain dopaminergic activity. Some have argued that these ramping signals index state values rather than RPEs. Our goal here was to develop an EEG measure of continuous feedback processing in humans, then test whether its behavior could be accounted for by the RPE hypothesis. Participants completed a stimulus-response learning task in which a continuous reward cue gradually increased or decreased over time. A regression-based unmixing approach revealed EEG activity with a topography and time course consistent with the stimulus-preceding negativity (SPN), a scalp potential previously linked to reward anticipation and tonic dopamine release. Importantly, this reward-related activity depended on outcome expectancy: as predicted by the RPE hypothesis, activity for expected reward cues was reduced compared to unexpected reward cues. These results demonstrate the possibility of using human scalp-recorded potentials to track continuous feedback processing, and test candidate hypotheses of this activity.
Collapse
Affiliation(s)
- Cameron D. Hassall
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of PsychologyMacEwan UniversityEdmontonAlbertaCanada
| | - Yan Yan
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of PsychologyStanford UniversityStanfordCaliforniaUSA
| | - Laurence T. Hunt
- Department of PsychiatryUniversity of OxfordOxfordUK
- Department of Experimental PsychologyUniversity of OxfordOxfordUK
| |
Collapse
|
3
|
Riels K, Ramos Campagnoli R, Thigpen N, Keil A. Oscillatory brain activity links experience to expectancy during associative learning. Psychophysiology 2021; 59:e13946. [PMID: 34622471 PMCID: PMC10150413 DOI: 10.1111/psyp.13946] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2021] [Revised: 08/20/2021] [Accepted: 08/30/2021] [Indexed: 01/23/2023]
Abstract
Associating a novel situation with a specific outcome involves a cascade of cognitive processes, including selecting relevant stimuli, forming predictions regarding expected outcomes, and updating memorized predictions based on experience. The present manuscript uses computational modeling and machine learning to test the hypothesis that alpha-band (8-12 Hz) oscillations are involved in the updating of expectations based on experience. Participants learned that a visual cue predicted an aversive loud noise with a probability of 50%. The Rescorla-Wagner model of associative learning explained trial-wise changes in self-reported noise expectancy as well as alpha power changes. Experience in the past trial and self-reported expectancy for the subsequent trial were accurately decoded based on the topographical distribution of alpha power at specific latencies. Decodable information during initial association formation and contingency report recurred when viewing the conditioned cue. Findings support the idea that alpha oscillations have multiple, temporally specific, roles in the formation of associations between cues and outcomes.
Collapse
Affiliation(s)
- Kierstin Riels
- Department of Psychology, University of Florida, Gainesville, Florida, USA
| | - Rafaela Ramos Campagnoli
- Department of Neurobiology, Institute of Biology, Universidade Federal Fluminense, Niterói, Brazil
| | - Nina Thigpen
- Department of Psychology, University of Florida, Gainesville, Florida, USA
| | - Andreas Keil
- Department of Psychology, University of Florida, Gainesville, Florida, USA
| |
Collapse
|
4
|
Champ versus Chump: Viewing an Opponent’s Face Engages Attention but Not Reward Systems. GAMES 2021. [DOI: 10.3390/g12030062] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
When we play competitive games, the opponents that we face act as predictors of the outcome of the game. For instance, if you are an average chess player and you face a Grandmaster, you anticipate a loss. Framed in a reinforcement learning perspective, our opponents can be thought of as predictors of rewards and punishments. The present study investigates whether facing an opponent would be processed as a reward or punishment depending on the level of difficulty the opponent poses. Participants played Rock, Paper, Scissors against three computer opponents while electroencephalographic (EEG) data was recorded. In a key manipulation, one opponent (HARD) was programmed to win most often, another (EASY) was made to lose most often, and the third (AVERAGE) had equiprobable outcomes of wins, losses, and ties. Through practice, participants learned to anticipate the relative challenge of a game based on the opponent they were facing that round. An analysis of our EEG data revealed that winning outcomes elicited a reward positivity relative to losing outcomes. Interestingly, our analysis of the predictive cues (i.e., the opponents’ faces) demonstrated that attentional engagement (P3a) was contextually sensitive to anticipated game difficulty. As such, our results for the predictive cue are contrary to what one might expect for a reinforcement model associated with predicted reward, but rather demonstrate that the neural response to the predictive cue was encoding the level of engagement with the opponent as opposed to value relative to the anticipated outcome.
Collapse
|
5
|
Hoy CW, Steiner SC, Knight RT. Single-trial modeling separates multiple overlapping prediction errors during reward processing in human EEG. Commun Biol 2021; 4:910. [PMID: 34302057 PMCID: PMC8302587 DOI: 10.1038/s42003-021-02426-1] [Citation(s) in RCA: 15] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2020] [Accepted: 07/05/2021] [Indexed: 02/07/2023] Open
Abstract
Learning signals during reinforcement learning and cognitive control rely on valenced reward prediction errors (RPEs) and non-valenced salience prediction errors (PEs) driven by surprise magnitude. A core debate in reward learning focuses on whether valenced and non-valenced PEs can be isolated in the human electroencephalogram (EEG). We combine behavioral modeling and single-trial EEG regression to disentangle sequential PEs in an interval timing task dissociating outcome valence, magnitude, and probability. Multiple regression across temporal, spatial, and frequency dimensions characterized a spatio-tempo-spectral cascade from early valenced RPE value to non-valenced RPE magnitude, followed by outcome probability indexed by a late frontal positivity. Separating negative and positive outcomes revealed the valenced RPE value effect is an artifact of overlap between two non-valenced RPE magnitude responses: frontal theta feedback-related negativity on losses and posterior delta reward positivity on wins. These results reconcile longstanding debates on the sequence of components representing reward and salience PEs in the human EEG.
Collapse
Affiliation(s)
- Colin W Hoy
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA.
| | - Sheila C Steiner
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA
| | - Robert T Knight
- Helen Wills Neuroscience Institute, University of California Berkeley, Berkeley, CA, USA
- Department of Psychology, University of California Berkeley, Berkeley, CA, USA
| |
Collapse
|
6
|
Hammerstrom MR, Ferguson TD, Williams CC, Krigolson OE. What happens when right means wrong? The impact of conflict arising from competing feedback responses. Brain Res 2021; 1761:147393. [PMID: 33639202 DOI: 10.1016/j.brainres.2021.147393] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2020] [Revised: 02/19/2021] [Accepted: 02/22/2021] [Indexed: 12/12/2022]
Abstract
Humans often rely on feedback to learn. Indeed, in learning the difference between feedback and an expected outcome is computed to inform future actions. Further, recent work has found that reward and feedback have a unique role in modulating conflict processing and cognitive control. However, it is still not clear how conflict, especially concerning the processing and evaluation of feedback, impacts learning. To address this, we examined the effects of feedback competition on feedback evaluation in a reinforcement learning task. Specifically, we had participants play a simple two-choice gambling game while electroencephalographic (EEG) data were recorded. On half of the experiment blocks, we reversed the meaning of performance feedback for each trial from its prepotent meaning to induce response conflict akin to the Stroop effect (e.g., '✓' meant incorrect). Behaviourally, we found that participants' accuracy was reduced as a result of incongruent feedback. Paralleling this, an analysis of our EEG revealed that incongruent feedback resulted in a reduction in amplitude of the reward positivity and the P300, components of the human event-related brain potential implicated in reward processing. Our results demonstrate the negative impact of conflict on feedback evaluation and the impact of this on subsequent performance.
Collapse
Affiliation(s)
- Mathew R Hammerstrom
- Theoretical and Applied Neuroscience Laboratory, University of Victoria, Canada.
| | - Thomas D Ferguson
- Theoretical and Applied Neuroscience Laboratory, University of Victoria, Canada
| | - Chad C Williams
- Theoretical and Applied Neuroscience Laboratory, University of Victoria, Canada
| | - Olave E Krigolson
- Theoretical and Applied Neuroscience Laboratory, University of Victoria, Canada
| |
Collapse
|
7
|
Williams CC, Ferguson TD, Hassall CD, Abimbola W, Krigolson OE. The ERP, frequency, and time-frequency correlates of feedback processing: Insights from a large sample study. Psychophysiology 2020; 58:e13722. [PMID: 33169842 DOI: 10.1111/psyp.13722] [Citation(s) in RCA: 21] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2020] [Revised: 10/10/2020] [Accepted: 10/12/2020] [Indexed: 01/28/2023]
Abstract
Human learning, at least in part, appears to be dependent on the evaluation of how outcomes of our actions align with our expectations. Over the past 23 years, electroencephalography (EEG) has been used to probe the neural signatures of feedback processing. Seminal work demonstrated a difference in the human event-related potential (ERP) dependent on whether people were processing correct or incorrect feedback. Since then, these feedback evoked ERPs have been associated with reinforcement learning and conflict monitoring, tied to subsequent behavioral adaptations, and shown to be sensitive to a wide range of factors (e.g., Parkinson's disease). Recently, research has turned to frequency decomposition techniques to examine how changes in the EEG power spectra are related to underlying learning mechanisms. Although the literature on the neural correlates of feedback processing is vast, there are still methodological discrepancies and differences in results across studies. Here, we provide reference results and an investigation of methodological considerations for the ERP (reward positivity) and frequency (delta and theta power) correlates of feedback evaluation with a large sample size. Specifically, participants (n = 500) performed a two-armed bandit task while we recorded EEG. Our findings provide key information about the data characteristics and relationships that exist between the neural signatures of feedback evaluation. Additionally, we conclude with selected methodological recommendations for standardization of future research. All data and scripts are freely provided to facilitate open science.
Collapse
Affiliation(s)
- Chad C Williams
- Centre for Biomedical Research, University of Victoria, Victoria, BC, Canada
| | - Thomas D Ferguson
- Centre for Biomedical Research, University of Victoria, Victoria, BC, Canada
| | - Cameron D Hassall
- Centre for Biomedical Research, University of Victoria, Victoria, BC, Canada
| | - Wande Abimbola
- Centre for Biomedical Research, University of Victoria, Victoria, BC, Canada
| | - Olave E Krigolson
- Centre for Biomedical Research, University of Victoria, Victoria, BC, Canada
| |
Collapse
|
8
|
Reward Prediction Errors Reflect an Underlying Learning Process That Parallels Behavioural Adaptations: A Trial-to-Trial Analysis. ACTA ACUST UNITED AC 2019. [DOI: 10.1007/s42113-019-00069-4] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/25/2022]
|
9
|
Wilhelm RA, Miller MW, Gable PA. Neural and Attentional Correlates of Intrinsic Motivation Resulting from Social Performance Expectancy. Neuroscience 2019; 416:137-146. [PMID: 31369789 DOI: 10.1016/j.neuroscience.2019.07.039] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2019] [Revised: 07/12/2019] [Accepted: 07/22/2019] [Indexed: 11/28/2022]
Abstract
Some models of motivation distinguish between intrinsic and extrinsic motivation. While past work has examined the neural and cognitive correlates of extrinsic motivation, research on intrinsic motivation has relied primarily on behavioral measures of performance and learning. In particular, no past work has examined the neural and cognitive correlates of social performance expectancy, which is linked to intrinsic motivation. The current study manipulated expectancy of difficult (vs. easy) trials on a cued flanker task and assessed attentional scope and performance. EEG was used to examine motor-action preparation as measured by suppression of beta band activity over the motor cortex and feedback processing as measured by the Reward Positivity (RewP). Results revealed expectancy of difficult (vs. easy) trials narrowed attentional scope, reduced beta activity over the motor cortex, and enhanced RewP amplitudes to win feedback. These findings suggest that enhancing intrinsic motivation through expectancies of positive social comparison engages similar neural and cognitive correlates as extrinsic motivators high in motivational intensity.
Collapse
Affiliation(s)
- Ricardo A Wilhelm
- Department of Psychology, The University of Alabama, Box 870348, Tuscaloosa, AL 35487-0348, United States.
| | | | - Philip A Gable
- Department of Psychology, The University of Alabama, Box 870348, Tuscaloosa, AL 35487-0348, United States.
| |
Collapse
|
10
|
de Bruijn ERA, Mars RB, Hester R. Processing of performance errors predicts memory formation: Enhanced feedback-related negativities for corrected versus repeated errors in an associative learning paradigm. Eur J Neurosci 2019; 51:881-890. [PMID: 31494976 PMCID: PMC7079158 DOI: 10.1111/ejn.14566] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2019] [Revised: 08/20/2019] [Accepted: 08/27/2019] [Indexed: 11/27/2022]
Abstract
Learning from errors or negative feedback is crucial for adaptive behavior. FMRI studies have demonstrated enhanced anterior cingulate cortex activity for errors that were later corrected versus repeated errors even when a substantial delay between the error and the opportunity to correct was introduced. We aimed at identifying the electrophysiological correlates of these processes by investigating the feedback‐related negativity (FRN) and stimulus‐locked P3. Participants had to learn and recall the location of 2‐digit targets over consecutive rounds. Feedback was provided in two steps, first a color change indicated a correct or incorrect response (feedback phase) followed by presentation of the correct digit information (re‐encoding phase). Behaviorally, participants improved performance from the first to the third round. FRN amplitudes time‐locked to feedback were enhanced for corrected compared to repeated errors. The P3 in response to re‐encoding did not differ between the two error types. The finding that FRN amplitudes positively predicted memory performance is consistent with the idea that the FRN reflects prediction errors and the need for enhanced cognitive control. Interestingly, this happens early during feedback processing and not at a later time point when re‐encoding of correct information takes place. The prediction error signal reflected in the FRN is usually elicited by performance errors, but may thus also play a role in preparing/optimizing the system for memory formation. This supports the existence of a close link between action control and memory processes even when there is a substantial delay between error feedback and the opportunity to correct the error.
Collapse
Affiliation(s)
- Ellen R A de Bruijn
- Department of Clinical Psychology, Leiden University, Leiden, The Netherlands.,Leiden Institute for Brain and Cognition (LIBC), Leiden, The Netherlands
| | - Rogier B Mars
- Wellcome Centre for Integrative Neuroimaging, Centre for Functional MRI of the Brain (FMRIB), Nuffield Department of Clinical Neurosciences, John Radcliffe Hospital, University of Oxford, Oxford, UK.,Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen, The Netherlands
| | - Rob Hester
- Melbourne School of Psychological Sciences, University of Melbourne, Melbourne, Vic., Australia
| |
Collapse
|