1
|
Ruan Z, Liu S, Liu YA, Yang Q, Peng Z. Disorders of compulsivity: Deficits in arbitrating learning strategies. Addict Biol 2024; 29:e13433. [PMID: 39122356 PMCID: PMC11315606 DOI: 10.1111/adb.13433] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Revised: 06/12/2024] [Accepted: 07/19/2024] [Indexed: 08/12/2024]
Abstract
While previous research has shown that compulsivity is related to an imbalance between goal-directed and habitual learning systems, very little is known about whether this effect is due to the impairment of a single system or the impairment of the arbitration mechanism that determines which system controls behaviour at any given moment; the current study aims to address this disagreement. Nineteen alcohol use disorder, 30 obsessive-compulsive disorder (OCD) and 20 major depressive disorder patients and corresponding sex- and age-matched controls performed two-choice, three-stage Markov decision-making paradigm. Model-based and mode-free reinforcement learning models were used to independently fitted their behavioural data. Alcohol use disorder and OCD patients showed less model-based strategy choice than healthy controls in task conditions where the model-based strategy was optimal. Only OCD patients showed higher behavioural control system switching in task conditions where model-free use was optimal. Major depressive disorder patients did not differ from the matched control in both. These findings suggest that dysfunction in arbitration control between dual systems may be the basis for diverse disorders involving compulsivity.
Collapse
Affiliation(s)
- Zhongqiang Ruan
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive ScienceSouth China Normal UniversityGuangzhouChina
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of EducationGuangzhouChina
| | - Shilin Liu
- The Affiliated Brain Hospital, Guangzhou Medical UniversityGuangzhouChina
| | - Yu an Liu
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive ScienceSouth China Normal UniversityGuangzhouChina
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of EducationGuangzhouChina
| | - Qiong Yang
- The Affiliated Brain Hospital, Guangzhou Medical UniversityGuangzhouChina
| | - Ziwen Peng
- School of Psychology, Center for Studies of Psychological Application, and Guangdong Key Laboratory of Mental Health and Cognitive ScienceSouth China Normal UniversityGuangzhouChina
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of EducationGuangzhouChina
| |
Collapse
|
2
|
Wang X, Zhou X, Li J, Gong Y, Feng Z. A feasibility study of goal-directed network-based real-time fMRI neurofeedback for anhedonic depression. Front Psychiatry 2023; 14:1253727. [PMID: 38125285 PMCID: PMC10732355 DOI: 10.3389/fpsyt.2023.1253727] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/07/2023] [Accepted: 11/06/2023] [Indexed: 12/23/2023] Open
Abstract
Anhedonia is a hallmark symptom of depression that often lacks adequate interventions. The translational gap remains in clinical treatments based on neural substrates of anhedonia. Our pilot study found that depressed individuals depended less on goal-directed (GD) reward learning (RL), with reduced reward prediction error (RPE) BOLD signal. Previous studies have found that anhedonia is related to abnormal activities and/or functional connectivities of the central executive network (CEN) and salience network (SN), both of which belong to the goal-directed system. In addition, it was found that real-time functional magnetic resonance imaging (rt-fMRI) neurofeedback (NF) could improve the balance between CEN and SN in healthy individuals. Therefore, we speculate that rt-fMRI NF of the CEN and SN associated with the GD system may improve depressive and/or anhedonic symptoms. Therefore, this study (1) will examine individuals with anhedonic depression using GD-RL behavioral task, combined with functional magnetic resonance imaging and computational modeling to explore the role of CEN/SN deficits in anhedonic depression; and (2) will utilize network-based rt-fMRI NF to investigate whether it is feasible to regulate the differential signals of brain CEN/SN of GD system through rt-fMRI NF to alleviate depressive and/or anhedonic symptoms. This study highlights the need to elucidate the intervention effects of rt-fMRI NF and the underlying computational network neural mechanisms.
Collapse
Affiliation(s)
- Xiaoxia Wang
- Department of Basic Psychology, School of Psychology, Army Medical University, Chongqing, China
| | - Xiaoyan Zhou
- Chongqing City Mental Health Center, Southwest University, Chongqing, China
| | - Jing Li
- Department of Radiology, Southwest Hospital, Army Medical University, Chongqing, China
| | - Yushun Gong
- Department of Medical Equipment and Metrology, College of Biomedical Engineering, Army Medical University, Chongqing, China
| | - Zhengzhi Feng
- School of Psychology, Army Medical University, Chongqing, China
| |
Collapse
|
3
|
Ruan Z, Seger CA, Yang Q, Kim D, Lee SW, Chen Q, Peng Z. Impairment of arbitration between model-based and model-free reinforcement learning in obsessive-compulsive disorder. Front Psychiatry 2023; 14:1162800. [PMID: 37304449 PMCID: PMC10250695 DOI: 10.3389/fpsyt.2023.1162800] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/10/2023] [Accepted: 05/05/2023] [Indexed: 06/13/2023] Open
Abstract
Introduction Obsessive-compulsive disorder (OCD) is characterized by an imbalance between goal-directed and habitual learning systems in behavioral control, but it is unclear whether these impairments are due to a single system abnormality of the goal-directed system or due to an impairment in a separate arbitration mechanism that selects which system controls behavior at each point in time. Methods A total of 30 OCD patients and 120 healthy controls performed a 2-choice, 3-stage Markov decision-making paradigm. Reinforcement learning models were used to estimate goal-directed learning (as model-based reinforcement learning) and habitual learning (as model-free reinforcement learning). In general, 29 high Obsessive-Compulsive Inventory-Revised (OCI-R) score controls, 31 low OCI-R score controls, and all 30 OCD patients were selected for the analysis. Results Obsessive-compulsive disorder (OCD) patients showed less appropriate strategy choices than controls regardless of whether the OCI-R scores in the control subjects were high (p = 0.012) or low (p < 0.001), specifically showing a greater model-free strategy use in task conditions where the model-based strategy was optimal. Furthermore, OCD patients (p = 0.001) and control subjects with high OCI-R scores (H-OCI-R; p = 0.009) both showed greater system switching rather than consistent strategy use in task conditions where model-free use was optimal. Conclusion These findings indicated an impaired arbitration mechanism for flexible adaptation to environmental demands in both OCD patients and healthy individuals reporting high OCI-R scores.
Collapse
Affiliation(s)
- Zhongqiang Ruan
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
| | - Carol A. Seger
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Department of Psychology, Colorado State University, Fort Collins, CO, United States
| | - Qiong Yang
- Affective Disorder Center, Affiliated Brain Hospital of Guangzhou Medical University (Guangzhou Huiai Hospital), Guangzhou, China
| | - Dongjae Kim
- Department of AI-based Convergence, College of Engineering, Dankook University, Yongin, Republic of Korea
| | - Sang Wan Lee
- Department of Bio and Brain Engineering, Program of Brain and Cognitive Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon, Republic of Korea
| | - Qi Chen
- School of Psychology, Shenzhen University, Shenzhen, China
| | - Ziwen Peng
- Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, Center for Studies of Psychological Application, South China Normal University, Guangzhou, China
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Guangzhou, China
- Department of Child Psychiatry, Shenzhen Kangning Hospital, Shenzhen University School of Medicine, Shenzhen, China
| |
Collapse
|
4
|
Feher da Silva C, Lombardi G, Edelson M, Hare TA. Rethinking model-based and model-free influences on mental effort and striatal prediction errors. Nat Hum Behav 2023:10.1038/s41562-023-01573-1. [PMID: 37012365 DOI: 10.1038/s41562-023-01573-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2022] [Accepted: 02/27/2023] [Indexed: 04/05/2023]
Abstract
A standard assumption in neuroscience is that low-effort model-free learning is automatic and continuously used, whereas more complex model-based strategies are only used when the rewards they generate are worth the additional effort. We present evidence refuting this assumption. First, we demonstrate flaws in previous reports of combined model-free and model-based reward prediction errors in the ventral striatum that probably led to spurious results. More appropriate analyses yield no evidence of model-free prediction errors in this region. Second, we find that task instructions generating more correct model-based behaviour reduce rather than increase mental effort. This is inconsistent with cost-benefit arbitration between model-based and model-free strategies. Together, our data indicate that model-free learning may not be automatic. Instead, humans can reduce mental effort by using a model-based strategy alone rather than arbitrating between multiple strategies. Our results call for re-evaluation of the assumptions in influential theories of learning and decision-making.
Collapse
Affiliation(s)
| | - Gaia Lombardi
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Micah Edelson
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland
| | - Todd A Hare
- Zurich Center for Neuroeconomics, Department of Economics, University of Zurich, Zurich, Switzerland.
| |
Collapse
|
5
|
Pupillary response in reward processing in adults with major depressive disorder in remission. J Int Neuropsychol Soc 2023; 29:306-315. [PMID: 35545874 DOI: 10.1017/s1355617722000224] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
OBJECTIVE Major depressive disorder (MDD) is associated with impaired reward processing and reward learning. The literature is inconclusive regarding whether these impairments persist after remission. The current study examined reward processing during a probabilistic learning task in individuals in remission from MDD (n = 19) and never depressed healthy controls (n = 31) matched for age and sex. The outcome measures were pupil dilation (an indirect index of noradrenergic activity and arousal) and computational modeling parameters. METHOD Participants completed two versions (facial/nonfacial feedback) of probabilistic reward learning task with changing contingencies. Pupil dilation was measured with a corneal reflection eye tracker. The hypotheses and analysis plan were preregistered. RESULT Healthy controls had larger pupil dilation following losses than gains (p <.001), whereas no significant difference between outcomes was found in individuals with a history of MDD, resulting in an interaction between group and outcome (β = 0.81, SE = 0.34, t = 2.37, p = .018). The rMDD group also achieved lower mean score at the last trial (t[46.77] = 2.12, p = .040) as well as a smaller proportion of correct choices (t[46.70] = 2.09, p = .041) compared with healthy controls. CONCLUSION Impaired reward processing may persist after remission from MDD and could constitute a latent risk factor for relapse. Measuring pupil dilation in a reward learning task is a promising method for identifying reward processing abnormalities linked to MDD. The task is simple and noninvasive, which makes it feasible for clinical research.
Collapse
|
6
|
Kustubayeva AM, Nelson EB, Smith ML, Allendorfer JB, Eliassen JC. Functional MRI study of feedback-based reinforcement learning in depression. Front Neuroinform 2022; 16:1028121. [PMID: 36605827 PMCID: PMC9807874 DOI: 10.3389/fninf.2022.1028121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Accepted: 10/18/2022] [Indexed: 12/24/2022] Open
Abstract
Reinforcement learning depends upon the integrity of emotional circuitry to establish associations between environmental cues, decisions, and positive or negative outcomes in order to guide behavior through experience. The emotional dysregulation characteristic of major depressive disorder (MDD) may alter activity in frontal and limbic structures that are key to learning. Although reward and decision-making have been examined in MDD, the effects of depression on associative learning is less well studied. We investigated whether depressive symptoms would be related to abnormalities in learning-related brain activity as measured by functional magnetic resonance imaging (fMRI). Also, we explored whether melancholic and atypical features were associated with altered brain activity. We conducted MRI scans on a 4T Varian MRI system in 10 individuals with MDD and 10 healthy subjects. We examined event-related brain activation during feedback-based learning task using Analysis of Functional NeuroImages (AFNI) for image processing and statistical analysis. We observed that MDD patients exhibited reduced activation in visual cortex but increased activation in cingulate and insular regions compared to healthy participants. Also, in relation to features of depressive subtypes, we observed that levels of activation in striatal, thalamic, and precuneus regions were negatively correlated with atypical characteristics. These results suggest that the effects of MDD change the neural circuitry underlying associative learning, and these effects may depend upon subtype features of MDD.
Collapse
Affiliation(s)
- Almira M. Kustubayeva
- Center for Cognitive Neuroscience, Department of Physiology, Biophysics, and Neuroscience, Al-Farabi Kazakh National University, Almaty, Kazakhstan,Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States
| | - Erik B. Nelson
- Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States
| | - Michael L. Smith
- Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States,Department of Speech-Language-Hearing Sciences, University of Minnesota, Minneapolis, MN, United States
| | - Jane B. Allendorfer
- Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States,Department of Neurology, University of Alabama at Birmingham, Birmingham, AL, United States
| | - James C. Eliassen
- Department of Psychiatry and Behavioral Neuroscience, College of Medicine, University of Cincinnati, Cincinnati, OH, United States,Robert Bosch Automotive Steering LLC, Florence, KY, United States,*Correspondence: James C. Eliassen,
| |
Collapse
|