Early markers of ongoing action-effect learning

Parallel model-based and model-free reinforcement learning for card sorting performance

The Wisconsin Card Sorting Test (WCST) is considered a gold standard for the assessment of cognitive flexibility. On the WCST, repeating a sorting category following negative feedback is typically treated as indicating reduced cognitive flexibility. Therefore such responses are referred to as 'perseveration' errors. Recent research suggests that the propensity for perseveration errors is modulated by response demands: They occur less frequently when their commitment repeats the previously executed response. Here, we propose parallel reinforcement-learning models of card sorting performance, which assume that card sorting performance can be conceptualized as resulting from model-free reinforcement learning at the level of responses that occurs in parallel with model-based reinforcement learning at the categorical level. We compared parallel reinforcement-learning models with purely model-based reinforcement learning, and with the state-of-the-art attentional-updating model. We analyzed data from 375 participants who completed a computerized WCST. Parallel reinforcement-learning models showed best predictive accuracies for the majority of participants. Only parallel reinforcement-learning models accounted for the modulation of perseveration propensity by response demands. In conclusion, parallel reinforcement-learning models provide a new theoretical perspective on card sorting and it offers a suitable framework for discerning individual differences in latent processes that subserve behavioral flexibility.

On the efficiency of instruction-based rule encoding

Instructions have long been considered a highly efficient route to knowledge acquisition especially compared to trial-and-error learning. We aimed at substantiating this claim by identifying boundary conditions for such an efficiency gain, including the influence of active learning intention, repeated instructions, and working memory load and span. Our experimental design allowed us to not only assess how well the instructed stimulus-response (S-R) rules were implemented later on, but also to directly measure prior instruction encoding processes. This revealed that instruction encoding was boosted by an active learning intention which in turn entailed better subsequent rule implementation. As should be expected, instruction-based learning took fewer trials than trial-and-error learning to reach a similar performance level. But more importantly, even when performance was measured relative to the identical number of preceding correct implementation trials, this efficiency gain persisted both in accuracy and in speed. This suggests that the naturally greater number of failed attempts in the initial phase of trial-and-error learning also negatively impacted learning in subsequent trials due to the persistence of erroneous memory traces established beforehand. A single instruction trial was sufficient to establish the advantage over trial-and-error learning but repeated instructions were better. Strategic factors and inter-individual differences in WM span - the latter exclusively affecting trial-and-error learning presumably due to the considerably more demanding working memory operations - could reduce or even abolish this advantage, but only in error rates. The same was not true for response time gains suggesting generally more efficient task automatization in instruction-based learning.

Learning-Related Brain-Electrical Activity Dynamics Associated with the Subsequent Impact of Learnt Action-Outcome Associations

Goal-directed behavior relies on the integration of anticipated outcomes into action planning based on acquired knowledge about the current contingencies between behavioral responses (R) and desired outcomes (O) under specific stimulus conditions (S). According to ideomotor theory, bidirectional R-O associations are an integral part of this knowledge structure. Previous EEG studies have identified neural activity markers linked to the involvement of such associations, but the initial acquisition process has not yet been characterized. The present study thus examined brain-electrical activity dynamics during the rapid acquisition of novel bidirectional R-O associations during instructed S-R learning. Within a trial, we inspected response-locked and stimulus-locked activity dynamics in order to identify markers linked to the forward and backward activation of bidirectional R-O associations as they were being increasingly strengthened under forced choice conditions. We found that a post-response anterior negativity following auditory outcomes was increasingly attenuated as a function of the acquired association strength. This suggests that previously reported action-induced sensory attenuation effects under extensively trained free choice conditions can be established within few repetitions of specific R-O pairings under forced choice conditions. Furthermore, we observed the even more rapid development of a post-response but pre-outcome fronto-central positivity which was reduced for high R-O learners which might indicate the rapid deployment of preparatory attention towards predictable outcomes. Finally, we identified a learning-related stimulus-locked activity modulation within the visual P1-N1 latency range which might reflect the multi-sensory integration of the perceived antecedent visual stimulus the anticipated auditory outcome.

The neural basis of integrating pre- and post-response information for goal-directed actions

A fundamental prerequisite for goal-directed action is to encode the contingencies between responses (R) producing specific outcomes (O) in specific stimulus conditions (S). The present study aimed to characterize the functional neuroanatomy of different associational sub-components of such S-R-O contingencies during the first few trials of exposure. We devised a novel paradigm that was suited to distinguish BOLD activation patterns related to S-R, R-O, and the full S-R-O contingency. Different from previous studies our experimental design ensured that stimulus-related processes and outcome-related processes were maximally comparable, as both were learned incidentally and lacked intrinsic incentive value, and different from trial-and-error learning situations, outcomes did not serve a special role as performance feedback. We observed contingency-related dissociations between SMA, lateral OFC, and large parts of the reward system including central OFC, anterior striatum and midbrain areas. While the lateral OFC was involved in processing differential outcomes irrespective of a predictive stimulus context, the SMA was specifically engaged when differential outcomes could be predicted by the stimulus. By contrast, the activation pattern of reward system areas suggested that these regions serve a role in integrating non-incentive differential outcome information and incentive common outcome information. Together, these results support the notion that striatal and orbitofrontal regions are involved in outcome-related processes beyond trial-and-error S-R learning, that is, when outcomes are non-incentive and do not serve as reinforcing feedback that drives learning. Furthermore, our results clarify the role of the SMA in outcome-related processes thereby supporting current versions of ideomotor theory.

Distinct fronto-striatal couplings reveal the double-faced nature of response-outcome relations in instruction-based learning

Higher species commonly learn novel behaviors by evaluating retrospectively whether actions have yielded desirable outcomes. By relying on explicit behavioral instructions, only humans can use an acquisition shortcut that prospectively specifies how to yield intended outcomes under the appropriate stimulus conditions. A recent and largely unexplored hypothesis suggests that striatal areas interact with lateral prefrontal cortex (LPFC) when novel behaviors are learned via explicit instruction, and that regional subspecialization exists for the integration of differential response-outcome contingencies into the current task model. Behaviorally, outcome integration during instruction-based learning has been linked to functionally distinct performance indices. This includes (1) compatibility effects, measured in a postlearning test procedure probing the encoding strength of outcome-response (O-R) associations, and (2) increasing response slowing across learning, putatively indicating active usage of O-R associations for the online control of goal-directed action. In the present fMRI study, we examined correlations between these behavioral indices and the dynamics of fronto-striatal couplings in order to mutually constrain and refine the interpretation of neural and behavioral measures in terms of separable subprocesses during outcome integration. We found that O-R encoding strength correlated with LPFC-putamen coupling, suggesting that the putamen is relevant for the formation of both S-R habits and habit-like O-R associations. By contrast, response slowing as a putative index of active usage of O-R associations correlated with LPFC-caudate coupling. This finding highlights the relevance of the caudate for the online control of goal-directed action also under instruction-based learning conditions, and in turn clarifies the functional relevance of the behavioral slowing effect.

Response selection difficulty modulates the behavioral impact of rapidly learnt action effects

It is well-established that we can pick up action effect associations when acting in a free-choice intentional mode. However, it is less clear whether and when action effect associations are learnt and actually affect behavior if we are acting in a forced-choice mode, applying a specific stimulus-response (S-R) rule. In the present study, we investigated whether response selection difficulty imposed by S-R rules influences the initial rapid learning and the behavioral expression of previously learnt but weakly practiced action effect associations when those are re-activated by effect exposure. Experiment 1 showed that the rapid learning of action effect associations is not directly influenced by response selection difficulty. By contrast, the behavioral expression of re-activated action effect associations is prevented when actions are directly activated by highly over-learnt response cues and thus response selection difficulty is low. However, all three experiments showed that if response selection difficulty is sufficiently high during re-activation, the same action effect associations do influence behavior. Experiments 2 and 3 revealed that the effect of response selection difficulty cannot be fully reduced to giving action effects more time to prime an action, but seems to reflect competition during response selection. Finally, the present data suggest that when multiple novel rules are rapidly learnt in succession, which requires a lot of flexibility, action effect associations continue to influence behavior only if response selection difficulty is sufficiently high. Thus, response selection difficulty might modulate the impact of experiencing multiple learning episodes on action effect expression and learning, possibly via inducing different strategies.