1
|
Dubinsky JM, Hamid AA. The neuroscience of active learning and direct instruction. Neurosci Biobehav Rev 2024; 163:105737. [PMID: 38796122 DOI: 10.1016/j.neubiorev.2024.105737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 05/13/2024] [Accepted: 05/20/2024] [Indexed: 05/28/2024]
Abstract
Throughout the educational system, students experiencing active learning pedagogy perform better and fail less than those taught through direct instruction. Can this be ascribed to differences in learning from a neuroscientific perspective? This review examines mechanistic, neuroscientific evidence that might explain differences in cognitive engagement contributing to learning outcomes between these instructional approaches. In classrooms, direct instruction comprehensively describes academic content, while active learning provides structured opportunities for learners to explore, apply, and manipulate content. Synaptic plasticity and its modulation by arousal or novelty are central to all learning and both approaches. As a form of social learning, direct instruction relies upon working memory. The reinforcement learning circuit, associated agency, curiosity, and peer-to-peer social interactions combine to enhance motivation, improve retention, and build higher-order-thinking skills in active learning environments. When working memory becomes overwhelmed, additionally engaging the reinforcement learning circuit improves retention, providing an explanation for the benefits of active learning. This analysis provides a mechanistic examination of how emerging neuroscience principles might inform pedagogical choices at all educational levels.
Collapse
Affiliation(s)
- Janet M Dubinsky
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA.
| | - Arif A Hamid
- Department of Neuroscience, University of Minnesota, Minneapolis, MN, USA
| |
Collapse
|
2
|
Messanvi F, Visocky V, Senneca C, Berkun K, Taori M, Bradley SP, Wang H, Tejeda H, Chudasama Y. Galanin receptor 1 expressing neurons in hippocampal-prefrontal circuitry modulate goal directed attention and impulse control. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.07.29.605653. [PMID: 39131306 PMCID: PMC11312591 DOI: 10.1101/2024.07.29.605653] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/13/2024]
Abstract
While amino acid neurotransmitters are the main chemical messengers in the brain, they are co-expressed with neuropeptides which are increasingly recognized as modulators of cognitive pathways. For example, the neuropeptide galanin has been implicated in a wide range of pathological conditions in which frontal and temporal structures are compromised. In a recent study in rats, we discovered that direct pharmacological stimulation of galanin receptor type 1 (GalR1) in the ventral prefrontal cortex (vPFC) and ventral hippocampus (vHC) led to opposing effects on attention and impulse control behavior. In the present study, we investigate how subtypes of neurons expressing GalR1 in these two areas differentially contribute to these behaviors. We first establish that GalR1 is predominantly expressed in glutamatergic neurons in both the vPFC and HC. We develop a novel viral approach to gain genetic access to GalR1-expressing neurons and demonstrate that optogenetic excitation of GalR1 expressing neurons in the vPFC, but not vHC, selectively disrupts attention in a complex behavioral task. Finally, using fiber photometry, we measure the bulk calcium dynamics in GalR1-expressing neurons during the same task to demonstrate opposing activity in vPFC and vHC. These results are consistent with our previous work demonstrating differential behavioral effects induced by GalR1 activating in vPFC and vHC. These results indicate the distinct neuromodulatory and behavioral contributions of galanin mediated by subclasses of neurons in the hippocampal and prefrontal circuitry.
Collapse
Affiliation(s)
- Fany Messanvi
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
| | - Vladimir Visocky
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
| | - Carolyn Senneca
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
| | - Kathleen Berkun
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
| | - Maansi Taori
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
| | - Sean P Bradley
- Rodent Behavioral Core, National Institute of Mental Health, Bethesda, MD, USA
| | - Huikun Wang
- Unit on Neuromodulation and Synaptic Integration, National Institute of Mental Health, Bethesda, MD, USA
| | - Hugo Tejeda
- Unit on Neuromodulation and Synaptic Integration, National Institute of Mental Health, Bethesda, MD, USA
| | - Yogita Chudasama
- Section on Behavioral Neuroscience, National Institute of Mental Health, Bethesda, MD, USA
- Rodent Behavioral Core, National Institute of Mental Health, Bethesda, MD, USA
| |
Collapse
|
3
|
Anlló H, Bavard S, Benmarrakchi F, Bonagura D, Cerrotti F, Cicue M, Gueguen M, Guzmán EJ, Kadieva D, Kobayashi M, Lukumon G, Sartorio M, Yang J, Zinchenko O, Bahrami B, Silva Concha J, Hertz U, Konova AB, Li J, O'Madagain C, Navajas J, Reyes G, Sarabi-Jamab A, Shestakova A, Sukumaran B, Watanabe K, Palminteri S. Comparing experience- and description-based economic preferences across 11 countries. Nat Hum Behav 2024:10.1038/s41562-024-01894-9. [PMID: 38877287 DOI: 10.1038/s41562-024-01894-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Accepted: 04/19/2024] [Indexed: 06/16/2024]
Abstract
Recent evidence indicates that reward value encoding in humans is highly context dependent, leading to suboptimal decisions in some cases, but whether this computational constraint on valuation is a shared feature of human cognition remains unknown. Here we studied the behaviour of n = 561 individuals from 11 countries of markedly different socioeconomic and cultural makeup. Our findings show that context sensitivity was present in all 11 countries. Suboptimal decisions generated by context manipulation were not explained by risk aversion, as estimated through a separate description-based choice task (that is, lotteries) consisting of matched decision offers. Conversely, risk aversion significantly differed across countries. Overall, our findings suggest that context-dependent reward value encoding is a feature of human cognition that remains consistently present across different countries, as opposed to description-based decision-making, which is more permeable to cultural factors.
Collapse
Affiliation(s)
- Hernán Anlló
- Human Reinforcement Learning Team, Laboratory of Cognitive and Computational Neuroscience, Paris, France.
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan.
- Intercultural Cognitive Network, Paris, France.
| | - Sophie Bavard
- Human Reinforcement Learning Team, Laboratory of Cognitive and Computational Neuroscience, Paris, France
- Intercultural Cognitive Network, Paris, France
- General Psychology Lab, Hamburg University, Hamburg, Germany
| | - FatimaEzzahra Benmarrakchi
- Intercultural Cognitive Network, Paris, France
- School of Collective Intelligence, Université Mohammed VI Polytechnique, Rabat, Morocco
| | - Darla Bonagura
- Intercultural Cognitive Network, Paris, France
- Department of Psychiatry, University Behavioral Health Care and Brain Health Institute, Rutgers University-New Brunswick, Piscataway, NJ, USA
| | - Fabien Cerrotti
- Human Reinforcement Learning Team, Laboratory of Cognitive and Computational Neuroscience, Paris, France
- Intercultural Cognitive Network, Paris, France
| | - Mirona Cicue
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
| | - Maelle Gueguen
- Intercultural Cognitive Network, Paris, France
- Department of Psychiatry, University Behavioral Health Care and Brain Health Institute, Rutgers University-New Brunswick, Piscataway, NJ, USA
| | - Eugenio José Guzmán
- Facultad de Psicología, Universidad del Desarrollo, Santiago de Chile, Chile
| | - Dzerassa Kadieva
- International Laboratory for Social Neurobiology, Institute for Cognitive Neuroscience, HSE University, Moscow, Russia
| | - Maiko Kobayashi
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
| | - Gafari Lukumon
- School of Collective Intelligence, Université Mohammed VI Polytechnique, Rabat, Morocco
| | - Marco Sartorio
- Laboratorio de Neurociencia, Universidad Torcuato Di Tella, Buenos Aires, Argentina
| | - Jiong Yang
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China
| | - Oksana Zinchenko
- Intercultural Cognitive Network, Paris, France
- Centre for Cognition and Decision Making, Institute for Cognitive Neuroscience, HSE University, Moscow, Russia
| | - Bahador Bahrami
- Intercultural Cognitive Network, Paris, France
- Department of Psychology, Ludwig Maximilian University, Munich, Germany
| | - Jaime Silva Concha
- Intercultural Cognitive Network, Paris, France
- Facultad de Psicología, Universidad del Desarrollo, Santiago de Chile, Chile
| | - Uri Hertz
- Intercultural Cognitive Network, Paris, France
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
| | - Anna B Konova
- Intercultural Cognitive Network, Paris, France
- Department of Psychiatry, University Behavioral Health Care and Brain Health Institute, Rutgers University-New Brunswick, Piscataway, NJ, USA
| | - Jian Li
- Intercultural Cognitive Network, Paris, France
- School of Psychological and Cognitive Sciences and Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China
- IDG/McGovern Institute for Brain Research, Peking University, Beijing, China
| | - Cathal O'Madagain
- Intercultural Cognitive Network, Paris, France
- School of Collective Intelligence, Université Mohammed VI Polytechnique, Rabat, Morocco
| | - Joaquin Navajas
- Intercultural Cognitive Network, Paris, France
- Laboratorio de Neurociencia, Universidad Torcuato Di Tella, Buenos Aires, Argentina
- Escuela de Negocios, Universidad Torcuato Di Tella, Buenos Aires, Argentina
- Consejo Nacional de Investigaciones Científicas y Técnicas, Buenos Aires, Argentina
| | - Gabriel Reyes
- Intercultural Cognitive Network, Paris, France
- Facultad de Psicología, Universidad del Desarrollo, Santiago de Chile, Chile
| | - Atiye Sarabi-Jamab
- Intercultural Cognitive Network, Paris, France
- School of Cognitive Sciences, Institute for Research in Fundamental Sciences, Tehran, Iran
| | - Anna Shestakova
- Intercultural Cognitive Network, Paris, France
- Centre for Cognition and Decision Making, Institute for Cognitive Neuroscience, HSE University, Moscow, Russia
| | - Bhasi Sukumaran
- Intercultural Cognitive Network, Paris, France
- Department of Clinical Psychology, SRM Medical College Hospital and Research Centre, Chennai, India
| | - Katsumi Watanabe
- Faculty of Science and Engineering, Waseda University, Tokyo, Japan
- Intercultural Cognitive Network, Paris, France
| | - Stefano Palminteri
- Human Reinforcement Learning Team, Laboratory of Cognitive and Computational Neuroscience, Paris, France.
- Intercultural Cognitive Network, Paris, France.
- Departement d'études cognitives, Ecole normale supérieure, PSL Research University, Paris, France.
| |
Collapse
|
4
|
Pereg M, Hertz U, Ben-Artzi I, Shahar N. Disentangling the contribution of individual and social learning processes in human advice-taking behavior. NPJ SCIENCE OF LEARNING 2024; 9:4. [PMID: 38245562 PMCID: PMC10799906 DOI: 10.1038/s41539-024-00214-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/25/2023] [Accepted: 01/03/2024] [Indexed: 01/22/2024]
Abstract
The study of social learning examines how individuals learn from others by means of observation, imitation, or compliance with advice. However, it still remains largely unknown whether social learning processes have a distinct contribution to behavior, independent from non-social trial-and-error learning that often occurs simultaneously. 153 participants completed a reinforcement learning task, where they were asked to make choices to gain rewards. Advice from an artificial teacher was presented in 60% of the trials, allowing us to compare choice behavior with and without advice. Results showed a strong and reliable tendency to follow advice (test-retest reliability ~0.73). Computational modeling suggested a unique contribution of three distinct learning strategies: (a) individual learning (i.e., learning the value of actions, independent of advice), (b) informed advice-taking (i.e., learning the value of following advice), and (c) non-informed advice-taking (i.e., a constant bias to follow advice regardless of outcome history). Comparing artificial and empirical data provided specific behavioral regression signatures to both informed and non-informed advice taking processes. We discuss the theoretical implications of integrating internal and external information during the learning process.
Collapse
Affiliation(s)
- Maayan Pereg
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel.
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel.
- Minducate Center for the Science of Learning, Sagol School of Neuroscience, Tel Aviv, Israel.
- Department of Psychology, Achva Academic College, Arugot, Israel.
| | - Uri Hertz
- Department of Cognitive Sciences, University of Haifa, Haifa, Israel
- Institute of Information Processing and Decision Making, University of Haifa, Haifa, Israel
| | - Ido Ben-Artzi
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
- Minducate Center for the Science of Learning, Sagol School of Neuroscience, Tel Aviv, Israel
| | - Nitzan Shahar
- School of Psychological Sciences, Tel Aviv University, Tel Aviv, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
5
|
Motzkin JC, Hiser J, Carroll I, Wolf R, Baskaya MK, Koenigs M, Atlas LY. Human ventromedial prefrontal cortex lesions enhance the effect of expectations on pain perception. Cortex 2023; 166:188-206. [PMID: 37390595 PMCID: PMC10528632 DOI: 10.1016/j.cortex.2023.04.017] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 03/23/2023] [Accepted: 04/21/2023] [Indexed: 07/02/2023]
Abstract
Pain is strongly modulated by expectations and beliefs. Across species, subregions of ventromedial prefrontal cortex (VMPFC) are implicated in a variety of functions germane to pain, predictions, and learning. Human fMRI studies show that VMPFC activity tracks expectations about pain and mediates expectancy effects on pain-related activity in other brain regions. Prior lesion studies suggest that VMPFC may instead play a more general role in generating affective responses to painful stimuli. To test whether VMPFC is required to generate affective responses to pain or is more specifically involved in expectancy-based pain modulation, we studied responses to heat stimuli in five adults with bilateral surgical lesions of VMPFC and twenty healthy adults without brain damage. All participants underwent a quantitative sensory testing procedure followed by a pain expectancy task in which cues predicting either low or high pain were followed by intermittent medium intensity heat stimuli. Compared to adults without brain damage, individuals with VMPFC lesions reported larger differences in expected pain based on predictive cues and failed to update expectations following the covert introduction of unexpected medium temperature stimuli. Consistent with observed expectancy differences, subjective pain unpleasantness ratings in the VMPFC lesion group were more strongly modulated by cue during thermal stimulation. We found no group differences in overall pain sensitivity, nor in relationships between pain and autonomic arousal, suggesting that VMPFC damage specifically enhances the effect of expectations on pain processing, likely driven by impaired integration of new sensory feedback to update expectations about pain. These results provide essential new data regarding the specific functional contribution of VMPFC to pain modulation.
Collapse
Affiliation(s)
- Julian C Motzkin
- Department of Neurology, University of California, San Francisco, CA, USA; Department of Anesthesia and Perioperative Care, University of California, San Francisco CA, USA
| | - Jaryd Hiser
- Department of Psychiatry, University of Wisconsin-Madison, WI, USA
| | - Ian Carroll
- Department of Psychology, University of Wisconsin-Madison, WI, USA
| | - Richard Wolf
- Department of Psychiatry, University of Wisconsin-Madison, WI, USA
| | - Mustafa K Baskaya
- Department of Neurological Surgery, University of Wisconsin-Madison, WI, USA
| | - Michael Koenigs
- Department of Psychiatry, University of Wisconsin-Madison, WI, USA
| | - Lauren Y Atlas
- National Center for Complementary and Integrative Health, National Institutes of Health, Bethesda, MD, USA; National Institute of Mental Health, National Institutes of Health, Bethesda, MD, USA; National Institute on Drug Abuse, National Institutes of Health, Baltimore, MD, USA.
| |
Collapse
|
6
|
Vélez N, Chen AM, Burke T, Cushman FA, Gershman SJ. Teachers recruit mentalizing regions to represent learners' beliefs. Proc Natl Acad Sci U S A 2023; 120:e2215015120. [PMID: 37216526 PMCID: PMC10235937 DOI: 10.1073/pnas.2215015120] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2022] [Accepted: 04/20/2023] [Indexed: 05/24/2023] Open
Abstract
Teaching enables humans to impart vast stores of culturally specific knowledge and skills. However, little is known about the neural computations that guide teachers' decisions about what information to communicate. Participants (N = 28) played the role of teachers while being scanned using fMRI; their task was to select examples that would teach learners how to answer abstract multiple-choice questions. Participants' examples were best described by a model that selects evidence that maximizes the learner's belief in the correct answer. Consistent with this idea, participants' predictions about how well learners would do closely tracked the performance of an independent sample of learners (N = 140) who were tested on the examples they had provided. In addition, regions that play specialized roles in processing social information, namely the bilateral temporoparietal junction and middle and dorsal medial prefrontal cortex, tracked learners' posterior belief in the correct answer. Our results shed light on the computational and neural architectures that support our extraordinary abilities as teachers.
Collapse
Affiliation(s)
- Natalia Vélez
- Department of Psychology, Harvard University, Cambridge, MA20138
| | - Alicia M. Chen
- Department of Brain and Cognitive Sciences, Massachusetts Institute of Technology, Cambridge, MA02139
| | - Taylor Burke
- Department of Psychology, Harvard University, Cambridge, MA20138
| | - Fiery A. Cushman
- Department of Psychology, Harvard University, Cambridge, MA20138
| | | |
Collapse
|
7
|
Lempert KM, Schaefer L, Breslow D, Peterson TD, Kable JW, McGuire JT. Statistical information about reward timing is insufficient for promoting optimal persistence decisions. Cognition 2023; 237:105468. [PMID: 37148639 DOI: 10.1016/j.cognition.2023.105468] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2022] [Revised: 04/20/2023] [Accepted: 04/22/2023] [Indexed: 05/08/2023]
Abstract
When deciding how long to keep waiting for delayed rewards that will arrive at an uncertain time, different distributions of possible reward times dictate different optimal strategies for maximizing reward. When reward timing distributions are heavy-tailed (e.g., waiting on hold) there is a point at which waiting is no longer advantageous because the opportunity cost of waiting is too high. Alternatively, when reward timing distributions have more predictable timing (e.g., uniform), it is advantageous to wait as long as necessary for the reward. Although people learn to approximate optimal strategies, little is known about how this learning occurs. One possibility is that people learn a general cognitive representation of the probability distribution that governs reward timing and then infer a strategy from that model of the environment. Another possibility is that they learn an action policy in a way that depends more narrowly on direct task experience, such that general knowledge of the reward timing distribution is insufficient for expressing the optimal strategy. Here, in a series of studies in which participants decided how long to persist for delayed rewards before quitting, we provided participants with information about the reward timing distribution in several ways. Whether the information was provided through counterfactual feedback (Study 1), previous exposure (Studies 2a and 2b), or description (Studies 3a and 3b), it did not obviate the need for direct, feedback-driven learning in a decision context. Therefore, learning when to quit waiting for delayed rewards might depend on task-specific experience, not solely on probabilistic reasoning.
Collapse
Affiliation(s)
- Karolina M Lempert
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, United States of America
| | - Lena Schaefer
- Department of Psychological & Brain Sciences, Boston University, Boston, MA 02215, United States of America
| | - Darby Breslow
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, United States of America
| | - Thomas D Peterson
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, United States of America
| | - Joseph W Kable
- Department of Psychology, University of Pennsylvania, Philadelphia, PA 19104, United States of America
| | - Joseph T McGuire
- Department of Psychological & Brain Sciences, Boston University, Boston, MA 02215, United States of America.
| |
Collapse
|
8
|
Hubbard NA, Miller KB, Aloi J, Bajaj S, Wakabayashi KT, Blair RJR. Evaluating instrumental learning and striatal-cortical functional connectivity in adolescent alcohol and cannabis use. Addict Biol 2023; 28:e13258. [PMID: 36577718 PMCID: PMC10173870 DOI: 10.1111/adb.13258] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Revised: 11/03/2022] [Accepted: 11/08/2022] [Indexed: 12/03/2022]
Abstract
Adolescence is a vulnerable time for the acquisition of substance use disorders, potentially relating to ongoing development of neural circuits supporting instrumental learning. Striatal-cortical circuits undergo dynamic changes during instrumental learning and are implicated in contemporary addiction theory. Human studies have not yet investigated these dynamic changes in relation to adolescent substance use. Here, functional magnetic resonance imaging was used while 135 adolescents without (AUD-CUDLow ) and with significant alcohol (AUDHigh ) or cannabis use disorder symptoms (CUDHigh ) performed an instrumental learning task. We assessed how cumulative experience with instrumental cues altered cue selection preferences and functional connectivity strength between reward-sensitive striatal and cortical regions. Adolescents in AUDHigh and CUDHigh groups were slower in learning to select optimal instrumental cues relative to AUD-CUDLow adolescents. The relatively fast learning observed for AUD-CUDLow adolescents coincided with stronger functional connectivity between striatal and frontoparietal regions during early relative to later periods of task experience, whereas the slower learning for the CUDHigh group coincided with the opposite pattern. The AUDHigh group not only exhibited slower learning but also produced more instrumental choice errors relative to AUD-CUDLow adolescents. For the AUDHigh group, Bayesian analyses evidenced moderate support for no experience-related changes in striatal-frontoparietal connectivity strength during the task. Findings suggest that adolescent cannabis use is related to slowed instrumental learning and delays in peak functional connectivity strength between the striatal-frontoparietal regions that support this learning, whereas adolescent alcohol use may be more closely linked to broader impairments in instrumental learning and a general depression of the neural circuits supporting it.
Collapse
Affiliation(s)
- NA Hubbard
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE
| | - KB Miller
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE
| | - J Aloi
- Department of Psychiatry, Indiana University School of Medicine, Indianapolis, IN
| | - S Bajaj
- Center for Neurobehavioral Research in Children, Boys Town National Research Hospital, Boys Town, NE
| | - KT Wakabayashi
- Department of Psychology, University of Nebraska-Lincoln, Lincoln, NE
| | - RJR Blair
- Child and Adolescent Mental Health Centre, Mental Health Services, Capital Region of Denmark, Copenhagen, Denmark
| |
Collapse
|
9
|
Atlas LY, Dildine TC, Palacios-Barrios EE, Yu Q, Reynolds RC, Banker LA, Grant SS, Pine DS. Instructions and experiential learning have similar impacts on pain and pain-related brain responses but produce dissociations in value-based reversal learning. eLife 2022; 11:73353. [PMID: 36317867 PMCID: PMC9681218 DOI: 10.7554/elife.73353] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Accepted: 10/25/2022] [Indexed: 11/22/2022] Open
Abstract
Recent data suggest that interactions between systems involved in higher order knowledge and associative learning drive responses during value-based learning. However, it is unknown how these systems impact subjective responses, such as pain. We tested how instructions and reversal learning influence pain and pain-evoked brain activation. Healthy volunteers (n=40) were either instructed about contingencies between cues and aversive outcomes or learned through experience in a paradigm where contingencies reversed three times. We measured predictive cue effects on pain and heat-evoked brain responses using functional magnetic resonance imaging. Predictive cues dynamically modulated pain perception as contingencies changed, regardless of whether participants received contingency instructions. Heat-evoked responses in the insula, anterior cingulate, and other regions updated as contingencies changed, and responses in the prefrontal cortex mediated dynamic cue effects on pain, whereas responses in the brainstem’s rostroventral medulla (RVM) were shaped by initial contingencies throughout the task. Quantitative modeling revealed that expected value was shaped purely by instructions in the Instructed Group, whereas expected value updated dynamically in the Uninstructed Group as a function of error-based learning. These differences were accompanied by dissociations in the neural correlates of value-based learning in the rostral anterior cingulate, thalamus, and posterior insula, among other regions. These results show how predictions dynamically impact subjective pain. Moreover, imaging data delineate three types of networks involved in pain generation and value-based learning: those that respond to initial contingencies, those that update dynamically during feedback-driven learning as contingencies change, and those that are sensitive to instruction. Together, these findings provide multiple points of entry for therapies designs to impact pain.
Collapse
Affiliation(s)
- Lauren Y Atlas
- National Center for Complementary and Integrative Health, National Institutes of Health
- National Institute on Drug Abuse, National Institutes of Health
- National Institute of Mental Health, National Institutes of Health
| | - Troy C Dildine
- National Center for Complementary and Integrative Health, National Institutes of Health
- Department of Clinical Neuroscience, Karolinska Institutet
| | | | - Qingbao Yu
- National Center for Complementary and Integrative Health, National Institutes of Health
| | | | - Lauren A Banker
- National Center for Complementary and Integrative Health, National Institutes of Health
| | - Shara S Grant
- National Center for Complementary and Integrative Health, National Institutes of Health
| | - Daniel S Pine
- National Institute of Mental Health, National Institutes of Health
| |
Collapse
|
10
|
Castro-Rodrigues P, Akam T, Snorasson I, Camacho M, Paixão V, Maia A, Barahona-Corrêa JB, Dayan P, Simpson HB, Costa RM, Oliveira-Maia AJ. Explicit knowledge of task structure is a primary determinant of human model-based action. Nat Hum Behav 2022; 6:1126-1141. [PMID: 35589826 DOI: 10.1038/s41562-022-01346-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2020] [Revised: 03/19/2022] [Accepted: 03/31/2022] [Indexed: 11/09/2022]
Abstract
Explicit information obtained through instruction profoundly shapes human choice behaviour. However, this has been studied in computationally simple tasks, and it is unknown how model-based and model-free systems, respectively generating goal-directed and habitual actions, are affected by the absence or presence of instructions. We assessed behaviour in a variant of a computationally more complex decision-making task, before and after providing information about task structure, both in healthy volunteers and in individuals suffering from obsessive-compulsive or other disorders. Initial behaviour was model-free, with rewards directly reinforcing preceding actions. Model-based control, employing predictions of states resulting from each action, emerged with experience in a minority of participants, and less in those with obsessive-compulsive disorder. Providing task structure information strongly increased model-based control, similarly across all groups. Thus, in humans, explicit task structural knowledge is a primary determinant of model-based reinforcement learning and is most readily acquired from instruction rather than experience.
Collapse
Affiliation(s)
- Pedro Castro-Rodrigues
- Champalimaud Clinical Centre, Champalimaud Foundation, Lisbon, Portugal.,Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,NOVA Medical School, NMS, Universidade Nova de Lisboa, Lisbon, Portugal.,Centro Hospitalar Psiquiátrico de Lisboa, Lisbon, Portugal
| | - Thomas Akam
- Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Ivar Snorasson
- Center for Obsessive-Compulsive & Related Disorders, New York State Psychiatric Institute, New York, NY, USA
| | - Marta Camacho
- Champalimaud Clinical Centre, Champalimaud Foundation, Lisbon, Portugal.,Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,John Van Geest Center for Brain Repair, University of Cambridge, Cambridge, UK
| | - Vitor Paixão
- Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal
| | - Ana Maia
- Champalimaud Clinical Centre, Champalimaud Foundation, Lisbon, Portugal.,Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,NOVA Medical School, NMS, Universidade Nova de Lisboa, Lisbon, Portugal.,Department of Psychiatry and Mental Health, Centro Hospitalar de Lisboa Ocidental, Lisbon, Portugal
| | - J Bernardo Barahona-Corrêa
- Champalimaud Clinical Centre, Champalimaud Foundation, Lisbon, Portugal.,Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,NOVA Medical School, NMS, Universidade Nova de Lisboa, Lisbon, Portugal
| | - Peter Dayan
- Max Planck Institute for Biological Cybernetics, Tübingen, Germany.,The University of Tübingen, Tübingen, Germany
| | - H Blair Simpson
- Center for Obsessive-Compulsive & Related Disorders, New York State Psychiatric Institute, New York, NY, USA.,Department of Psychiatry, Columbia University, New York, NY, USA
| | - Rui M Costa
- Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal.,NOVA Medical School, NMS, Universidade Nova de Lisboa, Lisbon, Portugal.,Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY, USA
| | - Albino J Oliveira-Maia
- Champalimaud Clinical Centre, Champalimaud Foundation, Lisbon, Portugal. .,Champalimaud Research, Champalimaud Foundation, Lisbon, Portugal. .,NOVA Medical School, NMS, Universidade Nova de Lisboa, Lisbon, Portugal.
| |
Collapse
|
11
|
Rybicki AJ, Sowden SL, Schuster B, Cook JL. Dopaminergic challenge dissociates learning from primary versus secondary sources of information. eLife 2022; 11:74893. [PMID: 35289748 PMCID: PMC9023054 DOI: 10.7554/elife.74893] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2021] [Accepted: 03/14/2022] [Indexed: 11/13/2022] Open
Abstract
Some theories of human cultural evolution posit that humans have social-specific learning mechanisms that are adaptive specialisations moulded by natural selection to cope with the pressures of group living. However, the existence of neurochemical pathways that are specialised for learning from social information and individual experience is widely debated. Cognitive neuroscientific studies present mixed evidence for social-specific learning mechanisms: some studies find dissociable neural correlates for social and individual learning, whereas others find the same brain areas and, dopamine-mediated, computations involved in both. Here, we demonstrate that, like individual learning, social learning is modulated by the dopamine D2 receptor antagonist haloperidol when social information is the primary learning source, but not when it comprises a secondary, additional element. Two groups (total N = 43) completed a decision-making task which required primary learning, from own experience, and secondary learning from an additional source. For one group, the primary source was social, and secondary was individual; for the other group this was reversed. Haloperidol affected primary learning irrespective of social/individual nature, with no effect on learning from the secondary source. Thus, we illustrate that dopaminergic mechanisms underpinning learning can be dissociated along a primary-secondary but not a social-individual axis. These results resolve conflict in the literature and support an expanding field showing that, rather than being specialised for particular inputs, neurochemical pathways in the human brain can process both social and non-social cues and arbitrate between the two depending upon which cue is primarily relevant for the task at hand.
Collapse
|
12
|
Frydecka D, Piotrowski P, Bielawski T, Pawlak E, Kłosińska E, Krefft M, Al Noaimy K, Rymaszewska J, Moustafa AA, Drapała J, Misiak B. Confirmation Bias in the Course of Instructed Reinforcement Learning in Schizophrenia-Spectrum Disorders. Brain Sci 2022; 12:brainsci12010090. [PMID: 35053833 PMCID: PMC8773670 DOI: 10.3390/brainsci12010090] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2021] [Revised: 01/05/2022] [Accepted: 01/06/2022] [Indexed: 11/16/2022] Open
Abstract
A large body of research attributes learning deficits in schizophrenia (SZ) to the systems involved in value representation (prefrontal cortex, PFC) and reinforcement learning (basal ganglia, BG) as well as to the compromised connectivity of these regions. In this study, we employed learning tasks hypothesized to probe the function and interaction of the PFC and BG in patients with SZ-spectrum disorders in comparison to healthy control (HC) subjects. In the Instructed Probabilistic Selection task (IPST), participants received false instruction about one of the stimuli used in the course of probabilistic learning which creates confirmation bias, whereby the instructed stimulus is overvalued in comparison to its real experienced value. The IPST was administered to 102 patients with SZ and 120 HC subjects. We have shown that SZ patients and HC subjects were equally influenced by false instruction in reinforcement learning (RL) probabilistic task (IPST) (p-value = 0.441); however, HC subjects had significantly higher learning rates associated with the process of overcoming cognitive bias in comparison to SZ patients (p-value = 0.018). The behavioral results of our study could be hypothesized to provide further evidence for impairments in the SZ-BG circuitry; however, this should be verified by neurofunctional imaging studies.
Collapse
Affiliation(s)
- Dorota Frydecka
- Department of Psychiatry, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (T.B.); (M.K.); (K.A.N.); (J.R.)
- Correspondence:
| | - Patryk Piotrowski
- Department of Psychiatry, Division of Consultation Psychiatry and Neuroscience, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (P.P.); (B.M.)
| | - Tomasz Bielawski
- Department of Psychiatry, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (T.B.); (M.K.); (K.A.N.); (J.R.)
| | - Edyta Pawlak
- Department of Experimental Therapy, Institute of Immunology and Experimental Therapy, Polish Academy of Sciences, Weigel Street 12, 53-114 Wroclaw, Poland;
| | - Ewa Kłosińska
- Day-Care Psychiatric Unit, University Clinical Hospital, Pasteur Street 10, 50-367 Wroclaw, Poland;
| | - Maja Krefft
- Department of Psychiatry, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (T.B.); (M.K.); (K.A.N.); (J.R.)
| | - Kamila Al Noaimy
- Department of Psychiatry, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (T.B.); (M.K.); (K.A.N.); (J.R.)
| | - Joanna Rymaszewska
- Department of Psychiatry, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (T.B.); (M.K.); (K.A.N.); (J.R.)
| | - Ahmed A. Moustafa
- School of Psychology, Marcs Institute for Brain and Behaviour, Western Sydney University, Locked Bag 1797, Penrith, NSW 2751, Australia;
- Department of Human Anatomy and Physiology, Faculty of Health Sciences, University of Johannesburg, Johannesburg 2006, South Africa
| | - Jarosław Drapała
- Department of Computer Science and Systems Engineering, Faculty of Information and Communication Technology, Wroclaw University of Science and Technology, Wybrzeze Wyspianskiego Street 27, 50-370 Wroclaw, Poland;
| | - Błażej Misiak
- Department of Psychiatry, Division of Consultation Psychiatry and Neuroscience, Wroclaw Medical University, Pasteur Street 10, 50-367 Wroclaw, Poland; (P.P.); (B.M.)
| |
Collapse
|
13
|
Atlas LY, Sandman CF, Phelps EA. Rating expectations can slow aversive reversal learning. Psychophysiology 2021; 59:e13979. [PMID: 34837385 PMCID: PMC8810599 DOI: 10.1111/psyp.13979] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2021] [Revised: 10/24/2021] [Accepted: 11/03/2021] [Indexed: 12/01/2022]
Abstract
The process of learning allows organisms to develop predictions about outcomes in the environment, and learning is sensitive to both simple associations and higher order knowledge. However, it is unknown whether consciously attending to expectations shapes the learning process itself. Here, we directly tested whether rating expectations shapes arousal during classical conditioning. Participants performed an aversive learning paradigm wherein one image (CS+) was paired with shock on 50% of trials, while a second image (CS-) was never paired with shock. Halfway through the task, contingencies reversed. One group of participants rated the probability of upcoming shock on each trial, while the other group made no online ratings. We measured skin conductance response (SCR) evoked in response to the CS and used traditional analyses as well as quantitative models of reinforcement learning to test whether rating expectations influenced arousal and aversive reversal learning. Participants who provided online expectancy ratings displayed slower learning based on a hybrid model of adaptive learning and reduced reversal of SCR relative to those who did not rate expectations. Mediation analysis revealed that the effect of associative learning on SCR could be fully explained through its effects on subjective expectancy within the group who provided ratings. This suggests that the act of rating expectations reduces the speed of learning, likely through changes in attention, and that expectations directly influence arousal. Our findings indicate that higher order expectancy judgments can alter associative learning.
Collapse
Affiliation(s)
- Lauren Y Atlas
- National Center for Complementary and Integrative Health, National Institutes of Health, Bethesda, Maryland, USA.,National Institute of Mental Health, National Institutes of Health, Bethesda, Maryland, USA.,National Institutes on Drug Abuse, National Institutes of Health, Baltimore, Maryland, USA
| | - Christina F Sandman
- Department of Psychology, University of California, Los Angeles, California, USA
| | - Elizabeth A Phelps
- Department of Psychology, Harvard University, Cambridge, Massachusetts, USA
| |
Collapse
|
14
|
McDougle SD, Ballard IC, Baribault B, Bishop SJ, Collins AGE. Executive Function Assigns Value to Novel Goal-Congruent Outcomes. Cereb Cortex 2021; 32:231-247. [PMID: 34231854 PMCID: PMC8634563 DOI: 10.1093/cercor/bhab205] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 05/10/2021] [Accepted: 06/04/2021] [Indexed: 11/14/2022] Open
Abstract
People often learn from the outcomes of their actions, even when these outcomes do not involve material rewards or punishments. How does our brain provide this flexibility? We combined behavior, computational modeling, and functional neuroimaging to probe whether learning from abstract novel outcomes harnesses the same circuitry that supports learning from familiar secondary reinforcers. Behavior and neuroimaging revealed that novel images can act as a substitute for rewards during instrumental learning, producing reliable reward-like signals in dopaminergic circuits. Moreover, we found evidence that prefrontal correlates of executive control may play a role in shaping flexible responses in reward circuits. These results suggest that learning from novel outcomes is supported by an interplay between high-level representations in prefrontal cortex and low-level responses in subcortical reward circuits. This interaction may allow for human reinforcement learning over arbitrarily abstract reward functions.
Collapse
Affiliation(s)
| | - Ian C Ballard
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA 94720, USA
| | - Beth Baribault
- Department of Psychology, University of California, Berkeley, CA 94704, USA
| | - Sonia J Bishop
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA 94720, USA
- Department of Psychology, University of California, Berkeley, CA 94704, USA
| | - Anne G E Collins
- Helen Wills Neuroscience Institute, University of California, Berkeley, CA 94720, USA
- Department of Psychology, University of California, Berkeley, CA 94704, USA
| |
Collapse
|
15
|
Brady WJ, McLoughlin K, Doan TN, Crockett MJ. How social learning amplifies moral outrage expression in online social networks. SCIENCE ADVANCES 2021; 7:eabe5641. [PMID: 34389534 PMCID: PMC8363141 DOI: 10.1126/sciadv.abe5641] [Citation(s) in RCA: 28] [Impact Index Per Article: 9.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/30/2020] [Accepted: 06/24/2021] [Indexed: 05/26/2023]
Abstract
Moral outrage shapes fundamental aspects of social life and is now widespread in online social networks. Here, we show how social learning processes amplify online moral outrage expressions over time. In two preregistered observational studies on Twitter (7331 users and 12.7 million total tweets) and two preregistered behavioral experiments (N = 240), we find that positive social feedback for outrage expressions increases the likelihood of future outrage expressions, consistent with principles of reinforcement learning. In addition, users conform their outrage expressions to the expressive norms of their social networks, suggesting norm learning also guides online outrage expressions. Norm learning overshadows reinforcement learning when normative information is readily observable: in ideologically extreme networks, where outrage expression is more common, users are less sensitive to social feedback when deciding whether to express outrage. Our findings highlight how platform design interacts with human learning mechanisms to affect moral discourse in digital public spaces.
Collapse
Affiliation(s)
- William J Brady
- Department of Psychology, Yale University, New Haven, CT, USA.
| | | | - Tuan N Doan
- Department of Statistics and Data Science, Yale University, New Haven, CT, USA
| | | |
Collapse
|
16
|
Effects of category learning strategies on recognition memory. Mem Cognit 2021; 50:512-526. [PMID: 34282566 DOI: 10.3758/s13421-021-01207-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/07/2021] [Indexed: 11/08/2022]
Abstract
Extant research has shown that previously acquired categorical knowledge affects recognition memory, and that differences in category learning strategies impact classification accuracy. However, it is unknown whether different learning strategies also have downstream effects on subsequent recognition memory. The present study investigates the effect of two unidimensional rule-based category learning strategies - learning (a) with or without explicit instruction, and (b) with or without supervision - on subsequent recognition memory. Our findings suggest that acquiring categorical knowledge increased both hits (Experiments 1 and 2) and false-alarms (Experiment 1) for category-congruent items regardless of the particular strategy employed in initially learning these categories. There were, however, small processing speed advantages during recognition memory for both explicit instruction and supervised practice relative to neither (Experiment 2). We discuss these findings in the context of how prior knowledge influences recognition memory, and in relation to similar findings showing schematic effects on episodic memory.
Collapse
|
17
|
Jepma M, Schaaf JV, Visser I, Huizenga HM. Effects of advice on experienced-based learning in adolescents and adults. J Exp Child Psychol 2021; 211:105230. [PMID: 34256185 DOI: 10.1016/j.jecp.2021.105230] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Revised: 03/24/2021] [Accepted: 06/16/2021] [Indexed: 01/04/2023]
Abstract
Recent studies that compared effects of pre-learning advice on experience-based learning in adolescents and adults have yielded mixed results. Previous studies on this topic used choice tasks in which age-related differences in advice-related learning bias and exploratory choice behavior are difficult to dissociate. Moreover, these studies did not examine whether effects of advice depend on working memory load. In this preregistered study (in adolescents [13-15 years old] and adults [18-31 years old]), we addressed these issues by factorially combining advice and working memory load manipulations in an estimation task that does not require choices and hence eliminates the influence of known age-related differences in exploration. We found that advice guided participants' initial estimates in both age groups. When advice was correct, this improved estimation performance, especially in adolescents when working memory load was high. When advice was incorrect, it had a longer-lasting effect on adolescents' performance than on adults' performance. In contrast to previous findings in choice tasks, we found no evidence that advice biased learning in either age group. Taken together, our results suggest that learning in an estimation task improves between adolescence and adulthood but that the effects of advice on learning do not differ substantially between adolescents and adults.
Collapse
Affiliation(s)
- Marieke Jepma
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands.
| | - Jessica V Schaaf
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| | - Ingmar Visser
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| | - Hilde M Huizenga
- Department of Psychology, University of Amsterdam, 1012 WX Amsterdam, the Netherlands
| |
Collapse
|
18
|
Vélez N, Gweon H. Learning from other minds: an optimistic critique of reinforcement learning models of social learning. Curr Opin Behav Sci 2021; 38:110-115. [DOI: 10.1016/j.cobeha.2021.01.006] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/26/2023]
|
19
|
Palombo DJ, Patt VM, Hunsberger R, Verfaellie M. Probabilistic value learning in medial temporal lobe amnesia. Hippocampus 2021; 31:461-468. [PMID: 33638580 DOI: 10.1002/hipo.23317] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2020] [Revised: 02/04/2021] [Accepted: 02/06/2021] [Indexed: 11/06/2022]
Abstract
A prevailing view in cognitive neuroscience suggests that different forms of learning are mediated by dissociable memory systems, with a mesolimbic (i.e., midbrain and basal ganglia) system supporting incremental trial-and-error reinforcement learning and a hippocampal-based system supporting episodic memory. Yet, growing evidence suggests that the hippocampus may also be important for trial-and-error learning, particularly value or reward-based learning. In the present report, we use a lesion-based neuropsychological approach to clarify hippocampal contributions to such learning. Six amnesic patients with medial temporal lobe damage and a group of healthy controls were administered a simple value-based learning task involving probabilistic trial-and-error acquisition of stimulus-response-outcome (reward or none) contingencies modeled after Li et al. (Proceedings of the National Academy of Sciences , 2011, 108 (1), 55-60). As predicted, patients were significantly impaired on the task, demonstrating reduced learning of the contingencies. Our results provide further supportive evidence that the hippocampus' role in cognition extends beyond episodic memory tasks and call for further refinement of theoretical models of hippocampal functioning.
Collapse
Affiliation(s)
- Daniela J Palombo
- Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada.,VA Boston Healthcare System, Boston, Massachusetts, USA
| | - Virginie M Patt
- VA Boston Healthcare System, Boston, Massachusetts, USA.,Department of Psychiatry, Boston University School of Medicine, Boston, Massachusetts, USA
| | | | - Mieke Verfaellie
- VA Boston Healthcare System, Boston, Massachusetts, USA.,Department of Psychiatry, Boston University School of Medicine, Boston, Massachusetts, USA
| |
Collapse
|
20
|
Reward history impacts attentional orienting and inhibitory control on untrained tasks. Atten Percept Psychophys 2020; 82:3842-3862. [PMID: 32935290 DOI: 10.3758/s13414-020-02130-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/13/2023]
Abstract
It has been robustly shown that stimuli with reward history receive attentional priority. However, the majority of this research tests reward history effects on attentional bias using similar tasks for both the reward learning phase and the unrewarded testing phase, which limits our understanding of how the effects of reward history generalize beyond the trained tasks and mental sets. Across two new experiments, the current study addresses these issues by first associating reward with a stimulus in a visual search paradigm, and then testing value-driven effects of that stimulus in untrained and unrewarded tasks, including a cueing paradigm, a go/no-go task, and a delay discounting task. Results of Experiment 1 demonstrate that history of reward association in a visual search task generalizes to value-driven attentional bias in a different attention paradigm (i.e., cueing), indicating these effects are indeed attributable to imbued value that can transfer to other tasks beyond that in which the reward was trained. The results of Experiment 2 demonstrate that in addition to eliciting attentional orienting on untrained tasks, reward history can lead to better inhibitory control in the go/no-go task. We find no evidence for reward history effects in the delay discounting task. Together, these experiments demonstrate that when the reward association task is in the attention domain, reward history modulates attentional priority, and this effect generalizes to untrained and unrewarded tasks that utilize both spatial and nonspatial attention.
Collapse
|
21
|
Hippocampal contributions to value-based learning: Converging evidence from fMRI and amnesia. COGNITIVE AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2020; 19:523-536. [PMID: 30767129 DOI: 10.3758/s13415-018-00687-8] [Citation(s) in RCA: 17] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/13/2022]
Abstract
Recent evidence suggests that the human hippocampus-known primarily for its involvement in episodic memory-plays a role in a host of motivationally relevant behaviors, including some forms of value-based decision-making. However, less is known about the role of the hippocampus in value-based learning. Such learning is typically associated with a striatal system, yet a small number of studies, both in human and nonhuman species, suggest hippocampal engagement. It is not clear, however, whether this engagement is necessary for such learning. In the present study, we used both functional MRI (fMRI) and lesion-based neuropsychological methods to clarify hippocampal contributions to value-based learning. In Experiment 1, healthy participants were scanned while learning value-based contingencies (whether players in a "game" win money) in the context of a probabilistic learning task. Here, we observed recruitment of the hippocampus, in addition to the expected ventral striatal (nucleus accumbens) activation that typically accompanies such learning. In Experiment 2, we administered this task to amnesic patients with medial temporal lobe damage and to healthy controls. Amnesic patients, including those with damage circumscribed to the hippocampus, failed to acquire value-based contingencies, thus confirming that hippocampal engagement is necessary for task performance. Control experiments established that this impairment was not due to perceptual demands or memory load. Future research is needed to clarify the mechanisms by which the hippocampus contributes to value-based learning, but these findings point to a broader role for the hippocampus in goal-directed behaviors than previously appreciated.
Collapse
|
22
|
Olsson A, Knapska E, Lindström B. The neural and computational systems of social learning. Nat Rev Neurosci 2020; 21:197-212. [PMID: 32221497 DOI: 10.1038/s41583-020-0276-4] [Citation(s) in RCA: 110] [Impact Index Per Article: 27.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/07/2020] [Indexed: 01/10/2023]
Abstract
Learning the value of stimuli and actions from others - social learning - adaptively contributes to individual survival and plays a key role in cultural evolution. We review research across species targeting the neural and computational systems of social learning in both the aversive and appetitive domains. Social learning generally follows the same principles as self-experienced value-based learning, including computations of prediction errors and is implemented in brain circuits activated across task domains together with regions processing social information. We integrate neural and computational perspectives of social learning with an understanding of behaviour of varying complexity, from basic threat avoidance to complex social learning strategies and cultural phenomena.
Collapse
Affiliation(s)
- Andreas Olsson
- Department of Clinical Neuroscience, Division of Psychology, Karolinska Institutet, Solna, Sweden.
| | - Ewelina Knapska
- Laboratory of Emotions' Neurobiology, Centre of Excellence for Neural Plasticity and Brain Disorders (BRAINCITY), Nencki Institute of Experimental Biology, Polish Academy of Sciences, Warsaw, Poland
| | - Björn Lindström
- Department of Clinical Neuroscience, Division of Psychology, Karolinska Institutet, Solna, Sweden.,Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| |
Collapse
|
23
|
Schenk LA, Colloca L. The neural processes of acquiring placebo effects through observation. Neuroimage 2019; 209:116510. [PMID: 31899287 DOI: 10.1016/j.neuroimage.2019.116510] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2019] [Revised: 12/20/2019] [Accepted: 12/26/2019] [Indexed: 12/16/2022] Open
Abstract
Learning through social observation is critical for humans. The present study investigates the neural processes underlying the acquisition of placebo effects through observational learning. We created a new functional magnetic resonance imaging (fMRI) paradigm where participants (n = 38, healthy, both sexes) observed a demonstrator experiencing pain relief by a placebo treatment cream and experiencing pain without a treatment (control cream), and subsequently performed the same procedure themselves. Participants demonstrated placebo hypoalgesia while they performed the procedure themselves, confirming that observational learning can lead to placebo effects. During the observational learning phase, fMRI analysis showed a modulation of the amygdalae, periaqueductal grey, temporoparietal junctions (TPJ), and dorsolateral prefrontal cortex (DLPFC). Connectivity between the DLPFC and TPJ during the observational learning task was modulated by the placebo treatment and predicted subsequent placebo effects. Mediation analysis further confirmed that the DLPFC-TPJ connectivity formally mediated the effect of the observed treatment condition on subsequent placebo effects. Additionally, pre-recorded resting state connectivity between the DLPFC and TPJ also predicted observationally-learned placebo effects. Our findings provide an understanding of the neural processes during the acquisition of placebo effects through observation and indicate a critical role for DLPFC-TPJ integration processes during observational learning of therapeutic outcomes.
Collapse
Affiliation(s)
- Lieven A Schenk
- Department of Pain and Translational Symptom Science, School of Nursing, University of Maryland, Baltimore, USA.
| | - Luana Colloca
- Department of Pain and Translational Symptom Science, School of Nursing, University of Maryland, Baltimore, USA; Departments of Anesthesiology and Psychiatry, School of Medicine, University of Maryland, Baltimore, USA; Center to Advance Chronic Pain Research, University of Maryland, Baltimore, USA.
| |
Collapse
|
24
|
Phillips J, Morris A, Cushman F. How We Know What Not To Think. Trends Cogn Sci 2019; 23:1026-1040. [PMID: 31676214 DOI: 10.1016/j.tics.2019.09.007] [Citation(s) in RCA: 44] [Impact Index Per Article: 8.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2018] [Revised: 09/09/2019] [Accepted: 09/10/2019] [Indexed: 11/16/2022]
Abstract
Humans often represent and reason about unrealized possible actions - the vast infinity of things that were not (or have not yet been) chosen. This capacity is central to the most impressive of human abilities: causal reasoning, planning, linguistic communication, moral judgment, etc. Nevertheless, how do we select possible actions that are worth considering from the infinity of unrealized actions that are better left ignored? We review research across the cognitive sciences, and find that the possible actions considered by default are those that are both likely to occur and generally valuable. We then offer a unified theory of why. We propose that (i) across diverse cognitive tasks, the possible actions we consider are biased towards those of general practical utility, and (ii) a plausible primary function for this mechanism resides in decision making.
Collapse
Affiliation(s)
- Jonathan Phillips
- Program in Cognitive Science, Dartmouth College, 201 Reed Hall, Hanover, NH 03755, USA.
| | - Adam Morris
- Department of Psychology, Harvard University, 33 Kirkland Street, Cambridge, MA 02138, USA
| | - Fiery Cushman
- Department of Psychology, Harvard University, 33 Kirkland Street, Cambridge, MA 02138, USA
| |
Collapse
|
25
|
Koban L, Jepma M, López-Solà M, Wager TD. Different brain networks mediate the effects of social and conditioned expectations on pain. Nat Commun 2019; 10:4096. [PMID: 31506426 PMCID: PMC6736972 DOI: 10.1038/s41467-019-11934-y] [Citation(s) in RCA: 49] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2018] [Accepted: 08/06/2019] [Indexed: 01/31/2023] Open
Abstract
Information about others' experiences can strongly influence our own feelings and decisions. But how does such social information affect the neural generation of affective experience, and are the brain mechanisms involved distinct from those that mediate other types of expectation effects? Here, we used fMRI to dissociate the brain mediators of social influence and associative learning effects on pain. Participants viewed symbolic depictions of other participants' pain ratings (social information) and classically conditioned pain-predictive cues before experiencing painful heat. Social information and conditioned stimuli each had significant effects on pain ratings, and both effects were mediated by self-reported expectations. Yet, these effects were mediated by largely separable brain activity patterns, involving different large-scale functional networks. These results show that learned versus socially instructed expectations modulate pain via partially different mechanisms-a distinction that should be accounted for by theories of predictive coding and related top-down influences.
Collapse
Affiliation(s)
- Leonie Koban
- Institute of Cognitive Science, University of Colorado Boulder, Muenzinger D244, 345 UCB, Boulder, CO, 80302, USA.
- Department of Psychology and Neuroscience, University of Colorado Boulder, Muenzinger D244, 345 UCB, Boulder, CO, 80302, USA.
- Brain and Spine Institute (ICM), Control-Interoception-Attention Team, 47 Boulevard de l'Hôpital, 75013, Paris, France.
- Marketing Area, INSEAD, Boulevard de Constance, 77300, Fontainebleau, France.
| | - Marieke Jepma
- Department of Psychology, University of Amsterdam, Nieuwe Achtergracht 129B, 1018 WS, Amsterdam, The Netherlands
| | - Marina López-Solà
- Division of Behavioral Medicine and Clinical Psychology, Department of Pediatrics, Cincinnati Children's Hospital Medical Center, University of Cincinnati, 3333 Burnet Avenue, MLC2 7031 Pain Research Center, Cincinnati, OH, 45229, USA
| | - Tor D Wager
- Institute of Cognitive Science, University of Colorado Boulder, Muenzinger D244, 345 UCB, Boulder, CO, 80302, USA
- Department of Psychology and Neuroscience, University of Colorado Boulder, Muenzinger D244, 345 UCB, Boulder, CO, 80302, USA
- Department of Psychological and Brain Sciences, Dartmouth College, HB 6207, Moore Hall, Hanover, NH, 03755, USA
| |
Collapse
|
26
|
Di Gregorio F, Ernst B, Steinhauser M. Differential effects of instructed and objective feedback reliability on feedback-related brain activity. Psychophysiology 2019; 56:e13399. [PMID: 31131923 DOI: 10.1111/psyp.13399] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2018] [Revised: 03/22/2019] [Accepted: 03/26/2019] [Indexed: 12/26/2022]
Abstract
Feedback reliability refers to the probability that the same decision leads to the same positive or negative feedback in the future. Previous research has shown that unreliable feedback is associated with attenuated feedback-related brain activity in ERPs, represented by a reduced fronto-central valence effect (feedback-related negativity or reward positivity) and a reduced feedback-related P3. Here, we asked whether these effects reflect top-down mechanisms or whether they can be explained by implicit feedback-outcome contingency learning. In two experiments, participants performed a trial-and-error learning task while subjective or objective feedback reliability was varied across blocks. In Experiment 1, we manipulated instructed feedback reliability while holding objective feedback reliability constant. Low instructed feedback reliability led to an attenuation of the fronto-central valence effect and the P3. In Experiment 2, we manipulated objective feedback reliability while holding instructed feedback reliability constant. Here, no modulation of feedback-related brain activity was observed. These results suggest that effects of feedback reliability are driven by top-down mechanisms based on explicit knowledge. Specifically, effects on the fronto-central valence effect could indicate a devaluation of unreliable feedback or a bias on the generation or utilization of reward prediction errors.
Collapse
Affiliation(s)
- Francesco Di Gregorio
- Department of Psychology, Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany.,Casa Dei Risvegli Luca De Nigris, Centro Studi per la Ricerca sul Coma, Bologna, Italy
| | - Benjamin Ernst
- Department of Psychology, Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany
| | - Marco Steinhauser
- Department of Psychology, Catholic University of Eichstätt-Ingolstadt, Eichstätt, Germany
| |
Collapse
|
27
|
Effects of feedback reliability on feedback-related brain activity: A feedback valuation account. COGNITIVE AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2019; 18:596-608. [PMID: 29626297 DOI: 10.3758/s13415-018-0591-7] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Adaptive decision making relies on learning from feedback. Because feedback sometimes can be misleading, optimal learning requires that knowledge about the feedback's reliability be utilized to adjust feedback processing. Although previous research has shown that feedback reliability indeed influences feedback processing, the underlying mechanisms through which this is accomplished remain unclear. Here we propose that feedback processing is adjusted by the adaptive, top-down valuation of feedback. We assume that unreliable feedback is devalued relative to reliable feedback, thus reducing the reward prediction errors that underlie feedback-related brain activity and learning. A crucial prediction of this account is that the effects of feedback reliability are susceptible to contrast effects. That is, the effects of feedback reliability should be enhanced when both reliable and unreliable feedback are experienced within the same context, as compared to when only one level of feedback reliability is experienced. To evaluate this prediction, we measured the event-related potentials elicited by feedback in two experiments in which feedback reliability was varied either within or between blocks. We found that the fronto-central valence effect, a correlate of reward prediction errors during reinforcement learning, was reduced for unreliable feedback. But this result was obtained only when feedback reliability was varied within blocks, thus indicating a contrast effect. This suggests that the adaptive valuation of feedback is one mechanism underlying the effects of feedback reliability on feedback processing.
Collapse
|
28
|
How instructions shape aversive learning: higher order knowledge, reversal learning, and the role of the amygdala. Curr Opin Behav Sci 2019. [DOI: 10.1016/j.cobeha.2018.12.008] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/20/2022]
|
29
|
Wang JM, Zhu L, Brown VM, De La Garza R, Newton T, King-Casas B, Chiu PH. In Cocaine Dependence, Neural Prediction Errors During Loss Avoidance Are Increased With Cocaine Deprivation and Predict Drug Use. BIOLOGICAL PSYCHIATRY. COGNITIVE NEUROSCIENCE AND NEUROIMAGING 2019; 4:291-299. [PMID: 30297162 PMCID: PMC6857782 DOI: 10.1016/j.bpsc.2018.07.009] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/13/2018] [Revised: 07/02/2018] [Accepted: 07/11/2018] [Indexed: 02/06/2023]
Abstract
BACKGROUND In substance-dependent individuals, drug deprivation and drug use trigger divergent behavioral responses to environmental cues. These divergent responses are consonant with data showing that short- and long-term adaptations in dopamine signaling are similarly sensitive to state of drug use. The literature suggests a drug state-dependent role of learning in maintaining substance use; evidence linking dopamine to both reinforcement learning and addiction provides a framework to test this possibility. METHODS In a randomized crossover design, 22 participants with current cocaine use disorder completed a probabilistic loss-learning task during functional magnetic resonance imaging while on and off cocaine (44 sessions). Another 54 participants without Axis I psychopathology served as a secondary reference group. Within-drug state and paired-subjects' learning effects were assessed with computational model-derived individual learning parameters. Model-based neuroimaging analyses evaluated effects of drug use state on neural learning signals. Relationships among model-derived behavioral learning rates (α+, α-), neural prediction error signals (δ+, δ-), cocaine use, and desire to use were assessed. RESULTS During cocaine deprivation, cocaine-dependent individuals exhibited heightened positive learning rates (α+), heightened neural positive prediction error (δ+) responses, and heightened association of α+ with neural δ+ responses. The deprivation-enhanced neural learning signals were specific to successful loss avoidance, comparable to participants without psychiatric conditions, and mediated a relationship between chronicity of drug use and desire to use cocaine. CONCLUSIONS Neurocomputational learning signals are sensitive to drug use status and suggest that heightened reinforcement by successful avoidance of negative outcomes may contribute to drug seeking during deprivation. More generally, attention to drug use state is important for delineating substrates of addiction.
Collapse
Affiliation(s)
- John M Wang
- Virginia Tech Carilion Research Institute, Roanoke, Virginia; Department of Psychology, Virginia Tech, Virginia
| | - Lusha Zhu
- Virginia Tech Carilion Research Institute, Roanoke, Virginia
| | - Vanessa M Brown
- Virginia Tech Carilion Research Institute, Roanoke, Virginia; Department of Psychology, Virginia Tech, Virginia
| | | | | | - Brooks King-Casas
- Virginia Tech Carilion Research Institute, Roanoke, Virginia; Department of Psychology, Virginia Tech, Virginia; Virginia Tech-Wake Forest University School of Biomedical Engineering and Science, Blacksburg, Virginia.
| | - Pearl H Chiu
- Virginia Tech Carilion Research Institute, Roanoke, Virginia; Department of Psychology, Virginia Tech, Virginia.
| |
Collapse
|
30
|
Abstract
In today’s world, indirect exposure to threatening situations is more common than ever, as illustrated by footage of terror and disaster in social media. How do such social threat learning experiences shape our decisions? We found that learning about threats from both observation and verbal information strongly influenced decision making. As with learning from our own experience, this influence could be either adaptive or maladaptive depending on whether the social information provided accurate expectations about the environment. Our findings can help explain both adaptive and pathological behaviors resulting from the indirect exposure to threatening events. In today’s world, mass-media and online social networks present us with unprecedented exposure to second-hand, vicarious experiences and thereby the chance of forming associations between previously innocuous events (e.g., being in a subway station) and aversive outcomes (e.g., footage or verbal reports from a violent terrorist attack) without direct experience. Such social threat, or fear, learning can have dramatic consequences, as manifested in acute stress symptoms and maladaptive fears. However, most research has so far focused on socially acquired threat responses that are expressed as increased arousal rather than active behavior. In three experiments (n = 120), we examined the effect of indirect experiences on behaviors by establishing a link between social threat learning and instrumental decision making. We contrasted learning from direct experience (i.e., Pavlovian conditioning) (experiment 1) against two common forms of social threat learning—social observation (experiment 2) and verbal instruction (experiment 3)—and how this learning transferred to subsequent instrumental decision making using behavioral experiments and computational modeling. We found that both types of social threat learning transfer to decision making in a strong and surprisingly inflexible manner. Notably, computational modeling indicated that the transfer of observational and instructed threat learning involved different computational mechanisms. Our results demonstrate the strong influence of others’ expressions of fear on one’s own decisions and have important implications for understanding both healthy and pathological human behaviors resulting from the indirect exposure to threatening events.
Collapse
|
31
|
Tardiff N, Graves KN, Thompson-Schill SL. The Role of Frontostriatal Systems in Instructed Reinforcement Learning: Evidence From Genetic and Experimentally-Induced Variation. Front Hum Neurosci 2019; 12:472. [PMID: 30618672 PMCID: PMC6304395 DOI: 10.3389/fnhum.2018.00472] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2018] [Accepted: 11/12/2018] [Indexed: 01/16/2023] Open
Abstract
Instructions have a powerful effect on learning and decision-making, biasing choice even in the face of disconfirming feedback. Detrimental biasing effects have been reported in a number of studies in which instruction was given prior to trial-and-error learning. Previous work has attributed individual differences in instructional bias to variations in prefrontal and striatal dopaminergic genes, suggesting a role for prefrontally-mediated cognitive control processes in biasing learning. The current study replicates and extends these findings. Human subjects performed a probabilistic reinforcement learning task after receiving inaccurate instructions about the quality of one of the options. In order to establish a causal relationship between prefrontal cortical mechanisms and instructional bias, we applied transcranial direct current stimulation over dorsolateral prefrontal cortex (anodal, cathodal, or sham) while subjects performed the task. We additionally genotyped subjects for the COMT Val158Met genetic polymorphism, which influences the breakdown of prefrontal dopamine, and for the DAT1/SLC6A3 variable number tandem repeat, which affects expression of striatal dopamine transporter. We replicated the finding that the COMT Met allele is associated with increased instructional bias and further demonstrated that variation in DAT1 has similar effects to variation in COMT, with 9-repeat carriers demonstrating increased bias relative to 10-repeat homozygotes. Consistent with increased top-down regulation of reinforcement learning, anodal subjects demonstrated greater bias relative to sham, though this effect was present only early in training. In contrast, there was no effect of cathodal stimulation. Finally, we fit computational models to subjects' data to better characterize the mechanisms underlying instruction bias. A novel choice bias model, in which instructions influence decision-making rather than learning, was found to best account for subjects' behavior. Overall, these data provide further evidence for the role of frontostriatal interactions in biasing instructed reinforcement learning, which adds to the growing literature documenting both costs and benefits of cognitive control.
Collapse
Affiliation(s)
- Nathan Tardiff
- Department of Psychology, University of Pennsylvania Philadelphia, PA, United States
| | - Kathryn N Graves
- Department of Psychology, University of Pennsylvania Philadelphia, PA, United States
| | | |
Collapse
|
32
|
Jepma M, Koban L, van Doorn J, Jones M, Wager TD. Behavioural and neural evidence for self-reinforcing expectancy effects on pain. Nat Hum Behav 2018; 2:838-855. [PMID: 31558818 PMCID: PMC6768437 DOI: 10.1038/s41562-018-0455-8] [Citation(s) in RCA: 65] [Impact Index Per Article: 10.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2017] [Accepted: 09/19/2018] [Indexed: 01/30/2023]
Abstract
Beliefs and expectations often persist despite evidence to the contrary. Here we examine two potential mechanisms underlying such 'self-reinforcing' expectancy effects in the pain domain: modulation of perception and biased learning. In two experiments, cues previously associated with symbolic representations of high or low temperatures preceded painful heat. We examined trial-to-trial dynamics in participants' expected pain, reported pain and brain activity. Subjective and neural pain responses assimilated towards cue-based expectations, and pain responses in turn predicted subsequent expectations, creating a positive dynamic feedback loop. Furthermore, we found evidence for a confirmation bias in learning: higher- and lower-than-expected pain triggered greater expectation updating for high- and low-pain cues, respectively. Individual differences in this bias were reflected in the updating of pain-anticipatory brain activity. Computational modelling provided converging evidence that expectations influence both perception and learning. Together, perceptual assimilation and biased learning promote self-reinforcing expectations, helping to explain why beliefs can be resistant to change.
Collapse
Affiliation(s)
- Marieke Jepma
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands.
- Department of Psychology and Neuroscience and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA.
| | - Leonie Koban
- Department of Psychology and Neuroscience and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA
| | - Johnny van Doorn
- Department of Psychology, University of Amsterdam, Amsterdam, the Netherlands
| | - Matt Jones
- Department of Psychology and Neuroscience and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA
| | - Tor D Wager
- Department of Psychology and Neuroscience and Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA
| |
Collapse
|
33
|
Cavazza M. A Motivational Model of BCI-Controlled Heuristic Search. Brain Sci 2018; 8:E166. [PMID: 30200321 PMCID: PMC6162724 DOI: 10.3390/brainsci8090166] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/11/2018] [Revised: 08/12/2018] [Accepted: 08/17/2018] [Indexed: 02/04/2023] Open
Abstract
Several researchers have proposed a new application for human augmentation, which is to provide human supervision to autonomous artificial intelligence (AI) systems. In this paper, we introduce a framework to implement this proposal, which consists of using Brain⁻Computer Interfaces (BCI) to influence AI computation via some of their core algorithmic components, such as heuristic search. Our framework is based on a joint analysis of philosophical proposals characterising the behaviour of autonomous AI systems and recent research in cognitive neuroscience that support the design of appropriate BCI. Our framework is defined as a motivational approach, which, on the AI side, influences the shape of the solution produced by heuristic search using a BCI motivational signal reflecting the user's disposition towards the anticipated result. The actual mapping is based on a measure of prefrontal asymmetry, which is translated into a non-admissible variant of the heuristic function. Finally, we discuss results from a proof-of-concept experiment using functional near-infrared spectroscopy (fNIRS) to capture prefrontal asymmetry and control the progression of AI computation of traditional heuristic search problems.
Collapse
Affiliation(s)
- Marc Cavazza
- Department of Computing and Information Systems, University of Greenwich, London SE10 9LS, UK.
| |
Collapse
|
34
|
Talmi D, Slapkova M, Wieser MJ. Testing the Possibility of Model-based Pavlovian Control of Attention to Threat. J Cogn Neurosci 2018; 31:36-48. [PMID: 30156504 DOI: 10.1162/jocn_a_01329] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/23/2023]
Abstract
Signals for reward or punishment attract attention preferentially, a principle termed value-modulated attention capture (VMAC). The mechanisms that govern the allocation of attention can be described with a terminology that is more often applied to the control of overt behaviors, namely, the distinction between instrumental and Pavlovian control, and between model-free and model-based control. Although instrumental control of VMAC can be either model-free or model-based, it is not known whether Pavlovian control of VMAC can be model-based. To decide whether this is possible, we measured steady-state visual evoked potentials (SSVEPs) while 20 healthy adults took part in a novel task. During the learning stage, participants underwent aversive threat conditioning with two conditioned stimuli (CSs): one that predicted pain (CS+) and one that predicted safety (CS-). Instructions given before the test stage allowed participants to infer whether novel, ambiguous CSs (new_CS+/new_CS-) were threatening or safe. Correct inference required combining stored internal representations and new propositional information, the hallmark of model-based control. SSVEP amplitudes quantified the amount of attention allocated to novel CSs on their very first presentation, before they were ever reinforced. We found that SSVEPs were higher for new_CS+ than new_CS-. This result is potentially indicative of model-based Pavlovian control of VMAC, but additional controls are necessary to verify this conclusively. This result underlines the potential transformative role of information and inference in emotion regulation.
Collapse
|
35
|
Koban L, Jepma M, Geuter S, Wager TD. What's in a word? How instructions, suggestions, and social information change pain and emotion. Neurosci Biobehav Rev 2018; 81:29-42. [PMID: 29173508 DOI: 10.1016/j.neubiorev.2017.02.014] [Citation(s) in RCA: 66] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2016] [Revised: 02/06/2017] [Accepted: 02/14/2017] [Indexed: 01/10/2023]
Abstract
Instructions, suggestions, and other types of social information can have powerful effects on pain and emotion. Prominent examples include observational learning, social influence, placebo, and hypnosis. These different phenomena and their underlying brain mechanisms have been studied in partially separate literatures, which we discuss, compare, and integrate in this review. Converging findings from these literatures suggest that (1) instructions and social information affect brain systems associated with the generation of pain and emotion, and with reinforcement learning, and that (2) these changes are mediated by alterations in prefrontal systems responsible for top-down control and the generation of affective meaning. We argue that changes in expectation and appraisal, a process of assessing personal meaning and implications for wellbeing, are two potential key mediators of the effects of instructions and social information on affective experience. Finally, we propose a tentative model of how prefrontal regions, especially dorsolateral and ventromedial prefrontal cortex may regulate affective processing based on instructions and socially transmitted expectations more broadly.
Collapse
Affiliation(s)
- Leonie Koban
- Institute of Cognitive Science, University of Colorado Boulder, United States; Department of Psychology and Neuroscience, University of Colorado Boulder, United States.
| | - Marieke Jepma
- Cognitive Psychology Unit, Institute of Psychology, Leiden University, The Netherlands; Leiden Institute for Brain and Cognition, Leiden University, The Netherlands
| | - Stephan Geuter
- Institute of Cognitive Science, University of Colorado Boulder, United States; Department of Psychology and Neuroscience, University of Colorado Boulder, United States
| | - Tor D Wager
- Institute of Cognitive Science, University of Colorado Boulder, United States; Department of Psychology and Neuroscience, University of Colorado Boulder, United States
| |
Collapse
|
36
|
Wegman J, van Loon I, Smeets PA, Cools R, Aarts E. Top-down expectation effects of food labels on motivation. Neuroimage 2018; 173:13-24. [DOI: 10.1016/j.neuroimage.2018.02.011] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2017] [Revised: 02/03/2018] [Accepted: 02/05/2018] [Indexed: 01/02/2023] Open
|
37
|
Credit Assignment in a Motor Decision Making Task Is Influenced by Agency and Not Sensory Prediction Errors. J Neurosci 2018; 38:4521-4530. [PMID: 29650698 DOI: 10.1523/jneurosci.3601-17.2018] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2017] [Revised: 03/28/2018] [Accepted: 04/07/2018] [Indexed: 11/21/2022] Open
Abstract
Failures to obtain reward can occur from errors in action selection or action execution. Recently, we observed marked differences in choice behavior when the failure to obtain a reward was attributed to errors in action execution compared with errors in action selection (McDougle et al., 2016). Specifically, participants appeared to solve this credit assignment problem by discounting outcomes in which the absence of reward was attributed to errors in action execution. Building on recent evidence indicating relatively direct communication between the cerebellum and basal ganglia, we hypothesized that cerebellar-dependent sensory prediction errors (SPEs), a signal indicating execution failure, could attenuate value updating within a basal ganglia-dependent reinforcement learning system. Here we compared the SPE hypothesis to an alternative, "top-down" hypothesis in which changes in choice behavior reflect participants' sense of agency. In two experiments with male and female human participants, we manipulated the strength of SPEs, along with the participants' sense of agency in the second experiment. The results showed that, whereas the strength of SPE had no effect on choice behavior, participants were much more likely to discount the absence of rewards under conditions in which they believed the reward outcome depended on their ability to produce accurate movements. These results provide strong evidence that SPEs do not directly influence reinforcement learning. Instead, a participant's sense of agency appears to play a significant role in modulating choice behavior when unexpected outcomes can arise from errors in action execution.SIGNIFICANCE STATEMENT When learning from the outcome of actions, the brain faces a credit assignment problem: Failures of reward can be attributed to poor choice selection or poor action execution. Here, we test a specific hypothesis that execution errors are implicitly signaled by cerebellar-based sensory prediction errors. We evaluate this hypothesis and compare it with a more "top-down" hypothesis in which the modulation of choice behavior from execution errors reflects participants' sense of agency. We find that sensory prediction errors have no significant effect on reinforcement learning. Instead, instructions influencing participants' belief of causal outcomes appear to be the main factor influencing their choice behavior.
Collapse
|
38
|
Targeting RIG-I or STING promotes epithelial regeneration. Oncotarget 2017; 8:114418-114419. [PMID: 29383086 PMCID: PMC5777698 DOI: 10.18632/oncotarget.22994] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2017] [Accepted: 12/03/2017] [Indexed: 11/25/2022] Open
|
39
|
The Neural Basis of Changing Social Norms through Persuasion. Sci Rep 2017; 7:16295. [PMID: 29176682 PMCID: PMC5701130 DOI: 10.1038/s41598-017-16572-2] [Citation(s) in RCA: 6] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2017] [Accepted: 11/14/2017] [Indexed: 11/15/2022] Open
Abstract
Social norms regulate behavior, and changes in norms have a great impact on society. In most modern societies, norms change through interpersonal communication and persuasive messages found in media. Here, we examined the neural basis of persuasion-induced changes in attitude toward and away from norms using fMRI. We measured brain activity while human participants were exposed to persuasive messages directed toward specific norms. Persuasion directed toward social norms specifically activated a set of brain regions including temporal poles, temporo-parietal junction, and medial prefrontal cortex. Beyond these regions, when successful, persuasion away from an accepted norm specifically recruited the left middle temporal and supramarginal gyri. Furthermore, in combination with data from a separate attitude-rating task, we found that left supramarginal gyrus activity represented participant attitude toward norms and tracked the persuasion-induced attitude changes that were away from agreement.
Collapse
|
40
|
Bolenz F, Reiter AMF, Eppinger B. Developmental Changes in Learning: Computational Mechanisms and Social Influences. Front Psychol 2017; 8:2048. [PMID: 29250006 PMCID: PMC5715389 DOI: 10.3389/fpsyg.2017.02048] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2017] [Accepted: 11/09/2017] [Indexed: 11/13/2022] Open
Abstract
Our ability to learn from the outcomes of our actions and to adapt our decisions accordingly changes over the course of the human lifespan. In recent years, there has been an increasing interest in using computational models to understand developmental changes in learning and decision-making. Moreover, extensions of these models are currently applied to study socio-emotional influences on learning in different age groups, a topic that is of great relevance for applications in education and health psychology. In this article, we aim to provide an introduction to basic ideas underlying computational models of reinforcement learning and focus on parameters and model variants that might be of interest to developmental scientists. We then highlight recent attempts to use reinforcement learning models to study the influence of social information on learning across development. The aim of this review is to illustrate how computational models can be applied in developmental science, what they can add to our understanding of developmental mechanisms and how they can be used to bridge the gap between psychological and neurobiological theories of development.
Collapse
Affiliation(s)
- Florian Bolenz
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany
| | - Andrea M F Reiter
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany.,Department of Neurology, Max-Planck-Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
| | - Ben Eppinger
- Chair of Lifespan Developmental Neuroscience, Department of Psychology, Technische Universität Dresden, Dresden, Germany.,Department of Psychology, Concordia University, Montreal, QC, Canada.,PERFORM Centre, Concordia University, Montreal, QC, Canada
| |
Collapse
|
41
|
Suppression of Striatal Prediction Errors by the Prefrontal Cortex in Placebo Hypoalgesia. J Neurosci 2017; 37:9715-9723. [PMID: 28883019 DOI: 10.1523/jneurosci.1101-17.2017] [Citation(s) in RCA: 38] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2017] [Revised: 08/03/2017] [Accepted: 08/21/2017] [Indexed: 11/21/2022] Open
Abstract
Classical learning theories predict extinction after the discontinuation of reinforcement through prediction errors. However, placebo hypoalgesia, although mediated by associative learning, has been shown to be resistant to extinction. We tested the hypothesis that this is mediated by the suppression of prediction error processing through the prefrontal cortex (PFC). We compared pain modulation through treatment cues (placebo hypoalgesia, treatment context) with pain modulation through stimulus intensity cues (stimulus context) during functional magnetic resonance imaging in 48 male and female healthy volunteers. During acquisition, our data show that expectations are correctly learned and that this is associated with prediction error signals in the ventral striatum (VS) in both contexts. However, in the nonreinforced test phase, pain modulation and expectations of pain relief persisted to a larger degree in the treatment context, indicating that the expectations were not correctly updated in the treatment context. Consistently, we observed significantly stronger neural prediction error signals in the VS in the stimulus context compared with the treatment context. A connectivity analysis revealed negative coupling between the anterior PFC and the VS in the treatment context, suggesting that the PFC can suppress the expression of prediction errors in the VS. Consistent with this, a participant's conceptual views and beliefs about treatments influenced the pain modulation only in the treatment context. Our results indicate that in placebo hypoalgesia contextual treatment information engages prefrontal conceptual processes, which can suppress prediction error processing in the VS and lead to reduced updating of treatment expectancies, resulting in less extinction of placebo hypoalgesia.SIGNIFICANCE STATEMENT In aversive and appetitive reinforcement learning, learned effects show extinction when reinforcement is discontinued. This is thought to be mediated by prediction errors (i.e., the difference between expectations and outcome). Although reinforcement learning has been central in explaining placebo hypoalgesia, placebo hypoalgesic effects show little extinction and persist after the discontinuation of reinforcement. Our results support the idea that conceptual treatment beliefs bias the neural processing of expectations in a treatment context compared with a more stimulus-driven processing of expectations with stimulus intensity cues. We provide evidence that this is associated with the suppression of prediction error processing in the ventral striatum by the prefrontal cortex. This provides a neural basis for persisting effects in reinforcement learning and placebo hypoalgesia.
Collapse
|
42
|
Wang Y, Ma N, He X, Li N, Wei Z, Yang L, Zha R, Han L, Li X, Zhang D, Liu Y, Zhang X. Neural substrates of updating the prediction through prediction error during decision making. Neuroimage 2017; 157:1-12. [PMID: 28536046 DOI: 10.1016/j.neuroimage.2017.05.041] [Citation(s) in RCA: 13] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2016] [Revised: 05/13/2017] [Accepted: 05/17/2017] [Indexed: 11/30/2022] Open
Abstract
Learning of prediction error (PE), including reward PE and risk PE, is crucial for updating the prediction in reinforcement learning (RL). Neurobiological and computational models of RL have reported extensive brain activations related to PE. However, the occurrence of PE does not necessarily predict updating the prediction, e.g., in a probability-known event. Therefore, the brain regions specifically engaged in updating the prediction remain unknown. Here, we conducted two functional magnetic resonance imaging (fMRI) experiments, the probability-unknown Iowa Gambling Task (IGT) and the probability-known risk decision task (RDT). Behavioral analyses confirmed that PEs occurred in both tasks but were only used for updating the prediction in the IGT. By comparing PE-related brain activations between the two tasks, we found that the rostral anterior cingulate cortex/ventral medial prefrontal cortex (rACC/vmPFC) and the posterior cingulate cortex (PCC) activated only during the IGT and were related to both reward and risk PE. Moreover, the responses in the rACC/vmPFC and the PCC were modulated by uncertainty and were associated with reward prediction-related brain regions. Electric brain stimulation over these regions lowered the performance in the IGT but not in the RDT. Our findings of a distributed neural circuit of PE processing suggest that the rACC/vmPFC and the PCC play a key role in updating the prediction through PE processing during decision making.
Collapse
Affiliation(s)
- Ying Wang
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Ning Ma
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Xiaosong He
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Nan Li
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Zhengde Wei
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Lizhuang Yang
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Rujing Zha
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Long Han
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Xiaoming Li
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China; Department of Medical Psychology, Anhui Medical University, Hefei, Anhui 230032, China
| | - Daren Zhang
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China
| | - Ying Liu
- Provincial Hospital Affiliated to Anhui Medical University, Hefei, Anhui 230001, China.
| | - Xiaochu Zhang
- Key Laboratory of Brain Function and Disease, School of Life Sciences, University of Science and Technology of China, Chinese Academy of Sciences, Hefei, Anhui 230026, China; School of Humanities & Social Science, University of Science and Technology of China Hefei, Anhui 230026, China; Centers for Biomedical Engineering, University of Science and Technology of China Hefei, Anhui 230027, China.
| |
Collapse
|
43
|
Ernst B, Steinhauser M. Top-down control over feedback processing: The probability of valid feedback affects feedback-related brain activity. Brain Cogn 2017; 115:33-40. [PMID: 28407527 DOI: 10.1016/j.bandc.2017.03.008] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2016] [Revised: 02/15/2017] [Accepted: 03/21/2017] [Indexed: 01/20/2023]
Abstract
Adaptive decision-making requires that feedback about decision outcomes is adequately processed. Recent studies have shown that fronto-central event-related potentials (ERPs) are sensitive to feedback valence and can be used as an index of feedback processing. The present study investigated whether the processes involved in feedback evaluation are affected by top-down mechanisms driven by knowledge about feedback validity. In a simple decision task, participants had to make use of feedback to learn which one of two stimuli was associated with a reward in a later test phase. Feedback stimuli were followed by a cue indicating whether feedback was valid or invalid. Prior to each block, participants were informed about the frequency of valid feedback in this block. An effect of feedback validity was obtained not only for learning but also for fronto-central ERPs. While high-validity feedback was associated with a fronto-central valence effect, this effect was absent for low-validity feedback. This indicates that processes involved in feedback evaluation are affected by prior knowledge about feedback validity via top-down processes.
Collapse
|
44
|
Ruge H, Wolfensteller U. Towards an understanding of the neural dynamics of intentional learning: Considering the timescale. Neuroimage 2016; 142:668-673. [DOI: 10.1016/j.neuroimage.2016.06.006] [Citation(s) in RCA: 12] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2016] [Revised: 05/07/2016] [Accepted: 06/05/2016] [Indexed: 11/26/2022] Open
|
45
|
Neural Activities Underlying the Feedback Express Salience Prediction Errors for Appetitive and Aversive Stimuli. Sci Rep 2016; 6:34032. [PMID: 27694920 PMCID: PMC5046116 DOI: 10.1038/srep34032] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2016] [Accepted: 09/05/2016] [Indexed: 01/06/2023] Open
Abstract
Feedback information is essential for us to adapt appropriately to the environment. The feedback-related negativity (FRN), a frontocentral negative deflection after the delivery of feedback, has been found to be larger for outcomes that are worse than expected, and it reflects a reward prediction error derived from the midbrain dopaminergic projections to the anterior cingulate cortex (ACC), as stated in reinforcement learning theory. In contrast, the prediction of response-outcome (PRO) model claims that the neural activity in the mediofrontal cortex (mPFC), especially the ACC, is sensitive to the violation of expectancy, irrespective of the valence of feedback. Additionally, increasing evidence has demonstrated significant activities in the striatum, anterior insula and occipital lobe for unexpected outcomes independently of their valence. Thus, the neural mechanism of the feedback remains under dispute. Here, we investigated the feedback with monetary reward and electrical pain shock in one task via functional magnetic resonance imaging. The results revealed significant prediction-error-related activities in the bilateral fusiform gyrus, right middle frontal gyrus and left cingulate gyrus for both money and pain. This implies that some regions underlying the feedback may signal a salience prediction error rather than a reward prediction error.
Collapse
|
46
|
Valentin VV, Maddox WT, Ashby FG. Dopamine dependence in aggregate feedback learning: A computational cognitive neuroscience approach. Brain Cogn 2016; 109:1-18. [PMID: 27596541 DOI: 10.1016/j.bandc.2016.06.002] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2015] [Revised: 06/07/2016] [Accepted: 06/13/2016] [Indexed: 01/10/2023]
Abstract
Procedural learning of skills depends on dopamine-mediated striatal plasticity. Most prior work investigated single stimulus-response procedural learning followed by feedback. However, many skills include several actions that must be performed before feedback is available. A new procedural-learning task is developed in which three independent and successive unsupervised categorization responses receive aggregate feedback indicating either that all three responses were correct, or at least one response was incorrect. Experiment 1 showed superior learning of stimuli in position 3, and that learning in the first two positions was initially compromised, and then recovered. An extensive theoretical analysis that used parameter space partitioning found that a large class of procedural-learning models, which predict propagation of dopamine release from feedback to stimuli, and/or an eligibility trace, fail to fully account for these data. The analysis also suggested that any dopamine released to the second or third stimulus impaired categorization learning in the first and second positions. A second experiment tested and confirmed a novel prediction of this large class of procedural-learning models that if the to-be-learned actions are introduced one-by-one in succession then learning is much better if training begins with the first action (and works forwards) than if it begins with the last action (and works backwards).
Collapse
Affiliation(s)
- Vivian V Valentin
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, United States.
| | - W Todd Maddox
- Department of Psychology, University of Texas, 108 E. Dean Keeton, Stop A8000, Austin, TX 78712-1043, United States.
| | - F Gregory Ashby
- Department of Psychological & Brain Sciences, University of California, Santa Barbara, United States.
| |
Collapse
|
47
|
The fate of memory: Reconsolidation and the case of Prediction Error. Neurosci Biobehav Rev 2016; 68:423-441. [DOI: 10.1016/j.neubiorev.2016.06.004] [Citation(s) in RCA: 105] [Impact Index Per Article: 13.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2015] [Revised: 05/07/2016] [Accepted: 06/06/2016] [Indexed: 11/22/2022]
|
48
|
Schiffer AM, Siletti K, Waszak F, Yeung N. Adaptive behaviour and feedback processing integrate experience and instruction in reinforcement learning. Neuroimage 2016; 146:626-641. [PMID: 27577720 PMCID: PMC5312784 DOI: 10.1016/j.neuroimage.2016.08.057] [Citation(s) in RCA: 20] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/10/2016] [Revised: 08/25/2016] [Accepted: 08/26/2016] [Indexed: 12/03/2022] Open
Abstract
In any non-deterministic environment, unexpected events can indicate true changes in the world (and require behavioural adaptation) or reflect chance occurrence (and must be discounted). Adaptive behaviour requires distinguishing these possibilities. We investigated how humans achieve this by integrating high-level information from instruction and experience. In a series of EEG experiments, instructions modulated the perceived informativeness of feedback: Participants performed a novel probabilistic reinforcement learning task, receiving instructions about reliability of feedback or volatility of the environment. Importantly, our designs de-confound informativeness from surprise, which typically co-vary. Behavioural results indicate that participants used instructions to adapt their behaviour faster to changes in the environment when instructions indicated that negative feedback was more informative, even if it was simultaneously less surprising. This study is the first to show that neural markers of feedback anticipation (stimulus-preceding negativity) and of feedback processing (feedback-related negativity; FRN) reflect informativeness of unexpected feedback. Meanwhile, changes in P3 amplitude indicated imminent adjustments in behaviour. Collectively, our findings provide new evidence that high-level information interacts with experience-driven learning in a flexible manner, enabling human learners to make informed decisions about whether to persevere or explore new options, a pivotal ability in our complex environment. Study used instructions to modulate beliefs about informativeness of feedback. Reversal learning performance improved with perceived informativeness. Instruction-sensitive Hidden Markov Model provides good fit of behaviour. EEG recordings of feedback-related negativity (FRN) show modulation by instructions. Findings suggest reinforcement learning integrates experience with high-level beliefs.
Collapse
Affiliation(s)
- Anne-Marike Schiffer
- Department of Experimental Psychology, University of Oxford, OX13UD Oxford, UK; Université Paris Descartes, Sorbonne Paris Cité, Paris, France; CNRS (Laboratoire Psychologie de la Perception, UMR 8158), Paris, France.
| | - Kayla Siletti
- Department of Experimental Psychology, University of Oxford, OX13UD Oxford, UK
| | - Florian Waszak
- Université Paris Descartes, Sorbonne Paris Cité, Paris, France; CNRS (Laboratoire Psychologie de la Perception, UMR 8158), Paris, France
| | - Nick Yeung
- Department of Experimental Psychology, University of Oxford, OX13UD Oxford, UK
| |
Collapse
|
49
|
Contributions of the hippocampus to feedback learning. COGNITIVE AFFECTIVE & BEHAVIORAL NEUROSCIENCE 2016; 15:861-77. [PMID: 26055632 DOI: 10.3758/s13415-015-0364-5] [Citation(s) in RCA: 15] [Impact Index Per Article: 1.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/02/2023]
Abstract
Humans learn about the world in a variety of manners, including by observation, by associating cues in the environment, and via feedback. Across species, two brain structures have been predominantly involved in these learning processes: the hippocampus--supporting learning via observation and paired association--and the striatum--critical for feedback learning. This simple dichotomy, however, has recently been challenged by reports of hippocampal engagement in feedback learning, although the role of the hippocampus is not fully understood. The purpose of this experiment was to characterize the hippocampal response during feedback learning by manipulating varying levels of memory interference. Consistent with prior reports, feedback learning recruited the striatum and midbrain. Notably, feedback learning also engaged the hippocampus. The level of activity in these regions was modulated by the degree of memory interference, such that the greatest activation occurred during the highest level of memory interference. Importantly, the accuracy of information learned via feedback correlated with hippocampal activation and was reduced by the presence of high memory interference. Taken together, these findings provide evidence of hippocampal involvement in feedback learning by demonstrating both its relevance for the accuracy of information learned via feedback and its susceptibility to interference.
Collapse
|
50
|
Atlas LY, Doll BB, Li J, Daw ND, Phelps EA. Instructed knowledge shapes feedback-driven aversive learning in striatum and orbitofrontal cortex, but not the amygdala. eLife 2016; 5. [PMID: 27171199 PMCID: PMC4907691 DOI: 10.7554/elife.15192] [Citation(s) in RCA: 62] [Impact Index Per Article: 7.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2016] [Accepted: 05/08/2016] [Indexed: 11/16/2022] Open
Abstract
Socially-conveyed rules and instructions strongly shape expectations and emotions. Yet most neuroscientific studies of learning consider reinforcement history alone, irrespective of knowledge acquired through other means. We examined fear conditioning and reversal in humans to test whether instructed knowledge modulates the neural mechanisms of feedback-driven learning. One group was informed about contingencies and reversals. A second group learned only from reinforcement. We combined quantitative models with functional magnetic resonance imaging and found that instructions induced dissociations in the neural systems of aversive learning. Responses in striatum and orbitofrontal cortex updated with instructions and correlated with prefrontal responses to instructions. Amygdala responses were influenced by reinforcement similarly in both groups and did not update with instructions. Results extend work on instructed reward learning and reveal novel dissociations that have not been observed with punishments or rewards. Findings support theories of specialized threat-detection and may have implications for fear maintenance in anxiety. DOI:http://dx.doi.org/10.7554/eLife.15192.001 Around the start of the twentieth century, Pavlov discovered that dogs salivate upon hearing a bell that has previously signaled that food is available. This phenomenon, in which a neutral stimulus (the bell) becomes associated with a particular outcome (such as food), is known as classical conditioning. The network of brain regions that supports this process – which includes the striatum, the amygdala and the prefrontal cortex – seems to work in a similar way across most animal species, including humans. However, humans don’t learn only through experience or trial-and-error. We do not need to burn our hands to learn not to touch a hot stove: a verbal warning from others is usually sufficient. Experiments have shown that giving people verbal instructions on how to obtain rewards alters the activity of the striatum and prefrontal cortex. That is, the instructions interact with the circuit that also supports learning through experience. But is this the case for learning how to avoid punishments? That process depends largely on the amygdala, and it is possible that systems designed to detect threats may be less sensitive to verbal warnings. To address this question, Atlas et al. taught people to associate one image with a mild electric shock, and another with the absence of a shock. After a number of trials, the relationships were reversed so that the previously neutral picture now predicted a shock and vice versa. Telling the participants about the reversal in advance triggered changes in the activity of the striatum and part of the prefrontal cortex. By contrast, such warnings had no effect on the amygdala. Instead, the activity of the amygdala changed only after the volunteers had experienced for themselves the new relationship between the pictures and the shocks. A key next step is to find out whether this distinction between the two types of learning signals (those that can be updated by instructions and those that cannot) is specific to humans. While the current study relied upon language, there are other methods that could be used to explore this issue in animals. Furthermore, knowing that the human brain has a specialized threat detection system that is less sensitive to instructions could help us to understand and treat anxiety disorders. Atlas et al. hope to test this possibility directly in the future. DOI:http://dx.doi.org/10.7554/eLife.15192.002
Collapse
Affiliation(s)
- Lauren Y Atlas
- National Center for Complementary and Integrative Health, National Institutes of Health, Bethesda, United States.,National Institute on Drug Abuse, National Institutes of Health, Bayview, United States
| | - Bradley B Doll
- Center for Neural Sciences, New York University, New York, United States.,Department of Psychology, Columbia University, New York, United States
| | - Jian Li
- Department of Psychology, Beijing Key Laboratory of Behavior and Mental Health, Peking University, Beijing, China.,PKU-IDG/McGovern Institute for Brain Research, Peking University, Beijing, China
| | - Nathaniel D Daw
- Department of Psychology, Princeton Neuroscience Institute, Princeton University, Princeton, United States
| | - Elizabeth A Phelps
- Center for Neural Sciences, New York University, New York, United States.,Department of Psychology, New York University, New York, United States.,Nathan Kline Institute, Orangeburg, United States
| |
Collapse
|