Navigating complex decision spaces: Problems and paradigms in sequential choice

State-transition-free reinforcement learning in chimpanzees (Pan troglodytes)

The outcome of an action often occurs after a delay. One solution for learning appropriate actions from delayed outcomes is to rely on a chain of state transitions. Another solution, which does not rest on state transitions, is to use an eligibility trace (ET) that directly bridges a current outcome and multiple past actions via transient memories. Previous studies revealed that humans (Homo sapiens) learned appropriate actions in a behavioral task in which solutions based on the ET were effective but transition-based solutions were ineffective. This suggests that ET may be used in human learning systems. However, no studies have examined nonhuman animals with an equivalent behavioral task. We designed a task for nonhuman animals following a previous human study. In each trial, participants chose one of two stimuli that were randomly selected from three stimulus types: a stimulus associated with a food reward delivered immediately, a stimulus associated with a reward delivered after a few trials, and a stimulus associated with no reward. The presented stimuli did not vary according to the participants' choices. To maximize the total reward, participants had to learn the value of the stimulus associated with a delayed reward. Five chimpanzees (Pan troglodytes) performed the task using a touchscreen. Two chimpanzees were able to learn successfully, indicating that learning mechanisms that do not depend on state transitions were involved in the learning processes. The current study extends previous ET research by proposing a behavioral task and providing empirical data from chimpanzees.

Building Human-Like Artificial Agents: A General Cognitive Algorithm for Emulating Human Decision-Making in Dynamic Environments

One of the early goals of artificial intelligence (AI) was to create algorithms that exhibited behavior indistinguishable from human behavior (i.e., human-like behavior). Today, AI has diverged, often aiming to excel in tasks inspired by human capabilities and outperform humans, rather than replicating human cogntion and action. In this paper, I explore the overarching question of whether computational algorithms have achieved this initial goal of AI. I focus on dynamic decision-making, approaching the question from the perspective of computational cognitive science. I present a general cognitive algorithm that intends to emulate human decision-making in dynamic environments, as defined in instance-based learning theory (IBLT). I use the cognitive steps proposed in IBLT to organize and discuss current evidence that supports some of the human-likeness of the decision-making mechanisms. I also highlight the significant gaps in research that are required to improve current models and to create higher fidelity in computational algorithms to represent human decision processes. I conclude with concrete steps toward advancing the construction of algorithms that exhibit human-like behavior with the ultimate goal of supporting human dynamic decision-making.

Learning to deal with delayed outcomes: EEG oscillatory and slow potentials during the prefeedback interval

It is well established that the stimulus-preceding negativity (SPN) decreases in amplitude as a task is mastered, a phenomenon generally attributed to the reduction in anticipatory attention as feedback becomes less needed. Typically, the experiments supporting this assumption have used relatively short delays (<3 s). However, we found in a previous study that this decline in amplitude, although present during the 2.5-s prefeedback delay of a patterned key-pressing task, was absent with an 8-s delay. We reexamined this finding using a 6-s delay and found that the SPN diminished at frontal sites as participants learned a sequence of four keypress durations, but that this modulation was limited to the early half of the delay (maximum at 2 s). Decline of lateralized sensorimotor theta activity across trials was also limited to early portions of the delay. These findings suggest that processes other than anticipatory attention to feedback may be more relevant for explaining SPN diminution. Such processes could include adjustment and maintenance of action-outcome expectancies (e.g., forward models) during the prefeedback interval.

Reinforcement Learning based Decoding Using Internal Reward for Time Delayed Task in Brain Machine Interfaces

Reinforcement learning (RL) algorithm interprets neural signals into movement intentions with the guidance of the reward in Brain-machine interfaces (BMIs). Current RL algorithms generally work for the tasks with immediate rewards delivery, and lack of efficiency in delayed reward task. Prefrontal cortex, including medial prefrontal cortex(mPFC), has been demonstrated to assign credit to intermediate steps, which reinforces preceding action more efficiently. In this paper, we propose to simulate the functionality of mPFC activities as intermediate rewards to train a RL based decoder in a two-step movement task. A support vector machine (SVM) is adopted to verify if the subject expects a reward due to the accomplishment of a subtask from mPFC activity. Then this discrimination result will be utilized to guide the training of the RL decoder for each step respectively. Here, we apply the Sarsa-style attention-gated reinforcement learning (SAGREL) as the decoder to interpret motor cortex(M1) activity to action states. We test on in vivo primary motor cortex (M1) and mPFC data collected from rats, where the rats need to first trigger the start and then press lever for rewards using M1 signals. SAGREL using intermediate rewards from mPFC activities achieves a prediction accuracy of 66.8% ± 2.0.% (mean ± std) %, which is significantly better than the one using the reward by the end of trial (45.9.% ± 1.2%). This reveals the potentials of modelling mPFC activities as intermediate rewards for the delayed reward tasks.

The research of human individual’s conformity behavior in emergency situations

Purpose

The purpose of this paper (including three studies) is to investigate the idea that individuals in an emergency situation experience significantly a higher level of emotional activation, lower performance of task, and change in decision making, escaping behavior and conformity as compared to being in a non-emergency situation. It is also suggested that the level of emotional activation mediates the association between situation and the performance of task.

Design/methodology/approach

Study 1 was an experimental study with 43 university students and revealed that different situations induced different levels of emotional activation and they are positively and significantly related. Study 2 was a further exploration of study 1, in which 49 participants were asked to watch a recomposed video telling a reasonable story about escaping from the emergency and complete several tasks associated with two kinds of situations (non-emergency vs emergency). In study 3, 168 participants, randomly assigned to three groups, were asked to work on judgment tasks with different numbers of options.

Findings

Results revealed that individuals in an emergency situation experience significantly lower performance of task and higher conformity tendency than in a non-emergency situation. Also, the causal effect of the situation on the performance of task is mediated by the level of emotional activation. Moreover, results found that the performance of task is also a mediator between the level of emotional activation and conformity. Result showed that the number of choices is negatively related to conformity, and performance is a mediator between the number of choices and conformity.

Originality/value

This paper suggested that the level of emotional activation mediates the association between situation and the performance of task.

Impulsive reactivity to emotion and vulnerability to psychopathology

Impulsiveness has been studied as an aspect of personality and psychopathology for generations. There are longstanding disagreements about how to define it and whether it should be viewed as one construct or several. This article begins by briefly reviewing some earlier and some more recent work on impulsiveness. Several approaches have recently converged to focus on a distinction between impulsive reactions to emotion and impulsive properties that are not initiated by emotion. From this review, we turn to psychopathology. It is well known that impulsiveness is related to externalizing psychopathology, but some have concluded that a similar relation does not exist for internalizing psychopathology. A recent literature is described that challenges the latter conclusion, linking impulsive reactivity to emotion to both externalizing and internalizing aspects of psychopathology. Discussion then turns to emotion-related impulsiveness and other constructs to which it is conceptually and empirically related, reexamining whether other conceptual targets should be added to the discussion. The article closes with a consideration of how important it is to continue to remain open to new conceptual perspectives. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Computational model for behavior shaping as an adaptive health intervention strategy

Adaptive behavioral interventions that automatically adjust in real-time to participants' changing behavior, environmental contexts, and individual history are becoming more feasible as the use of real-time sensing technology expands. This development is expected to improve shortcomings associated with traditional behavioral interventions, such as the reliance on imprecise intervention procedures and limited/short-lived effects. JITAI adaptation strategies often lack a theoretical foundation. Increasing the theoretical fidelity of a trial has been shown to increase effectiveness. This research explores the use of shaping, a well-known process from behavioral theory for engendering or maintaining a target behavior, as a JITAI adaptation strategy. A computational model of behavior dynamics and operant conditioning was modified to incorporate the construct of behavior shaping by adding the ability to vary, over time, the range of behaviors that were reinforced when emitted. Digital experiments were performed with this updated model for a range of parameters in order to identify the behavior shaping features that optimally generated target behavior. Narrowing the range of reinforced behaviors continuously in time led to better outcomes compared with a discrete narrowing of the reinforcement window. Rapid narrowing followed by more moderate decreases in window size was more effective in generating target behavior than the inverse scenario. The computational shaping model represents an effective tool for investigating JITAI adaptation strategies. Model parameters must now be translated from the digital domain to real-world experiments so that model findings can be validated.

Solving the Credit Assignment Problem With the Prefrontal Cortex

In naturalistic multi-cue and multi-step learning tasks, where outcomes of behavior are delayed in time, discovering which choices are responsible for rewards can present a challenge, known as the credit assignment problem. In this review, I summarize recent work that highlighted a critical role for the prefrontal cortex (PFC) in assigning credit where it is due in tasks where only a few of the multitude of cues or choices are relevant to the final outcome of behavior. Collectively, these investigations have provided compelling support for specialized roles of the orbitofrontal (OFC), anterior cingulate (ACC), and dorsolateral prefrontal (dlPFC) cortices in contingent learning. However, recent work has similarly revealed shared contributions and emphasized rich and heterogeneous response properties of neurons in these brain regions. Such functional overlap is not surprising given the complexity of reciprocal projections spanning the PFC. In the concluding section, I overview the evidence suggesting that the OFC, ACC and dlPFC communicate extensively, sharing the information about presented options, executed decisions and received rewards, which enables them to assign credit for outcomes to choices on which they are contingent. This account suggests that lesion or inactivation/inhibition experiments targeting a localized PFC subregion will be insufficient to gain a fine-grained understanding of credit assignment during learning and instead poses refined questions for future research, shifting the focus from focal manipulations to experimental techniques targeting cortico-cortical projections.

Evaluating the Theoretic Adequacy and Applied Potential of Computational Models of the Spacing Effect

The spacing effect is among the most widely replicated empirical phenomena in the learning sciences, and its relevance to education and training is readily apparent. Yet successful applications of spacing effect research to education and training is rare. Computational modeling can provide the crucial link between a century of accumulated experimental data on the spacing effect and the emerging interest in using that research to enable adaptive instruction. In this paper, we review relevant literature and identify 10 criteria for rigorously evaluating computational models of the spacing effect. Five relate to evaluating the theoretic adequacy of a model, and five relate to evaluating its application potential. We use these criteria to evaluate a novel computational model of the spacing effect called the Predictive Performance Equation (PPE). Predictive Performance Equation combines elements of earlier models of learning and memory including the General Performance Equation, Adaptive Control of Thought-Rational, and the New Theory of Disuse, giving rise to a novel computational account of the spacing effect that performs favorably across the complete sets of theoretic and applied criteria. We implemented two other previously published computational models of the spacing effect and compare them to PPE using the theoretic and applied criteria as guides.

The detour problem in a stochastic environment: Tolman revisited

We designed a grid world task to study human planning and re-planning behavior in an unknown stochastic environment. In our grid world, participants were asked to travel from a random starting point to a random goal position while maximizing their reward. Because they were not familiar with the environment, they needed to learn its characteristics from experience to plan optimally. Later in the task, we randomly blocked the optimal path to investigate whether and how people adjust their original plans to find a detour. To this end, we developed and compared 12 different models. These models were different on how they learned and represented the environment and how they planned to catch the goal. The majority of our participants were able to plan optimally. We also showed that people were capable of revising their plans when an unexpected event occurred. The result from the model comparison showed that the model-based reinforcement learning approach provided the best account for the data and outperformed heuristics in explaining the behavioral data in the re-planning trials.

Modeling Search Behaviors during the Acquisition of Expertise in a Sequential Decision-Making Task

Our daily interaction with the world is plagued of situations in which we develop expertise through self-motivated repetition of the same task. In many of these interactions, and especially when dealing with computer and machine interfaces, we must deal with sequences of decisions and actions. For instance, when drawing cash from an ATM machine, choices are presented in a step-by-step fashion and a specific sequence of choices must be performed in order to produce the expected outcome. But, as we become experts in the use of such interfaces, is it possible to identify specific search and learning strategies? And if so, can we use this information to predict future actions? In addition to better understanding the cognitive processes underlying sequential decision making, this could allow building adaptive interfaces that can facilitate interaction at different moments of the learning curve. Here we tackle the question of modeling sequential decision-making behavior in a simple human-computer interface that instantiates a 4-level binary decision tree (BDT) task. We record behavioral data from voluntary participants while they attempt to solve the task. Using a Hidden Markov Model-based approach that capitalizes on the hierarchical structure of behavior, we then model their performance during the interaction. Our results show that partitioning the problem space into a small set of hierarchically related stereotyped strategies can potentially capture a host of individual decision making policies. This allows us to follow how participants learn and develop expertise in the use of the interface. Moreover, using a Mixture of Experts based on these stereotyped strategies, the model is able to predict the behavior of participants that master the task.

Toward a Functional View of the P Factor in Psychopathology

Evidence indicates the existence of a superordinate factor of general psychopathology, which has been termed p. Among the issues raised by this discovery is whether this factor has substantive meaning or not. This article suggests a functional interpretation of the p factor, based in part on a family of dual process models, in which an associative system and a deliberative system compete for influence over action. The associative system is frequently said to be impulsively responsive to emotions. We hypothesize that this impulsive responsivity to emotion underlies the p factor. One benefit of this view is to use the same underlying process variable to account for both internalizing and externalizing vulnerabilities, as well as aspects of thought disorder. Evidence is reviewed linking impulsive reactivity to emotion to the p factor, and (separately) to internalizing, externalizing, and thought-disorder symptoms. Alternative interpretations are considered.

VTA neurons coordinate with the hippocampal reactivation of spatial experience

Spatial learning requires the hippocampus, and the replay of spatial sequences during hippocampal sharp wave-ripple (SPW-R) events of quiet wakefulness and sleep is believed to play a crucial role. To test whether the coordination of VTA reward prediction error signals with these replayed spatial sequences could contribute to this process, we recorded from neuronal ensembles of the hippocampus and VTA as rats performed appetitive spatial tasks and subsequently slept. We found that many reward responsive (RR) VTA neurons coordinated with quiet wakefulness-associated hippocampal SPW-R events that replayed recent experience. In contrast, coordination between RR neurons and SPW-R events in subsequent slow wave sleep was diminished. Together, these results indicate distinct contributions of VTA reinforcement activity associated with hippocampal spatial replay to the processing of wake and SWS-associated spatial memory.

Dissociable functions of reward inference in the lateral prefrontal cortex and the striatum

In a complex and uncertain world, how do we select appropriate behavior? One possibility is that we choose actions that are highly reinforced by their probabilistic consequences (model-free processing). However, we may instead plan actions prior to their actual execution by predicting their consequences (model-based processing). It has been suggested that the brain contains multiple yet distinct systems involved in reward prediction. Several studies have tried to allocate model-free and model-based systems to the striatum and the lateral prefrontal cortex (LPFC), respectively. Although there is much support for this hypothesis, recent research has revealed discrepancies. To understand the nature of the reward prediction systems in the LPFC and the striatum, a series of single-unit recording experiments were conducted. LPFC neurons were found to infer the reward associated with the stimuli even when the monkeys had not yet learned the stimulus-reward (SR) associations directly. Striatal neurons seemed to predict the reward for each stimulus only after directly experiencing the SR contingency. However, the one exception was "Exclusive Or" situations in which striatal neurons could predict the reward without direct experience. Previous single-unit studies in monkeys have reported that neurons in the LPFC encode category information, and represent reward information specific to a group of stimuli. Here, as an extension of these, we review recent evidence that a group of LPFC neurons can predict reward specific to a category of visual stimuli defined by relevant behavioral responses. We suggest that the functional difference in reward prediction between the LPFC and the striatum is that while LPFC neurons can utilize abstract code, striatal neurons can code individual associations between stimuli and reward but cannot utilize abstract code.

Effects of positive mood on probabilistic learning: behavioral and electrophysiological correlates

Whether positive mood can change reinforcement learning or not remains an open question. In this study, we used a probabilistic learning task and explored whether positive mood could alter the way positive versus negative feedback was used to guide learning. This process was characterized both at the behavioral and electro-encephalographic levels. Thirty two participants were randomly allocated either to a positive or a neutral (control) mood condition. Behavioral results showed that while learning performance was balanced between the two groups, participants in the positive mood group had a higher learning rate than participants in the neutral mood group. At the electrophysiological level, we found that positive mood increased the error-related negativity when the stimulus-response associations were deterministic, selectively (as opposed to random or probabilistic). However, it did not influence the feedback-related negativity. These new findings are discussed in terms of an enhanced internal reward prediction error signal after the induction of positive mood when the probability of getting a reward is high.

Higher-order conditioning is impaired by hippocampal lesions

Behavior in the real world is rarely motivated by primary conditioned stimuli that have been directly associated with potent unconditioned reinforcers. Instead, motivation and choice behavior are driven by complex chains of higher-order associations that are only indirectly linked to intrinsic reward and often exert their influence outside awareness. Second-order conditioning (SOC) [1] is a basic associative-learning mechanism whereby stimuli acquire motivational salience by proxy, in the absence of primary incentives [2, 3]. Memory-systems theories consider first-order conditioning (FOC) and SOC to be prime examples of hippocampal-independent nondeclarative memory [4, 5]. Accordingly, neurobiological models of SOC focus almost exclusively on nondeclarative neural systems that support motivational salience and reward value. Transfer of value from a conditioned stimulus to a neutral stimulus is thought to require the basolateral amygdala [6, 7] and the ventral striatum [2, 3], but not the hippocampus. We developed a new paradigm to measure appetitive SOC of tones in rats. Hippocampal lesions severely impaired both acquisition and expression of SOC despite normal FOC. Unlike controls, rats with hippocampal lesions could not discriminate between positive and negative secondary conditioned tones, although they exhibited general familiarity with previously presented tones compared with new tones. Importantly, normal rats' behavior, in contrast to that of hippocampal groups, also revealed different confidence levels as indexed by effort, a central characteristic of hippocampal relational memory. The results indicate, contrary to current systems models, that representations of intrinsic relationships between reward value, stimulus identity, and motivation require hippocampal mediation when these relationships are of a higher order.

Electrophysiological responses to feedback during the application of abstract rules

Much research focuses on how people acquire concrete stimulus-response associations from experience; however, few neuroscientific studies have examined how people learn about and select among abstract rules. To address this issue, we recorded ERPs as participants performed an abstract rule-learning task. In each trial, they viewed a sample number and two test numbers. Participants then chose a test number using one of three abstract mathematical rules they freely selected from: greater than the sample number, less than the sample number, or equal to the sample number. No one rule was always rewarded, but some rules were rewarded more frequently than others. To maximize their earnings, participants needed to learn which rules were rewarded most frequently. All participants learned to select the best rules for repeating and novel stimulus sets that obeyed the overall reward probabilities. Participants differed, however, in the extent to which they overgeneralized those rules to repeating stimulus sets that deviated from the overall reward probabilities. The feedback-related negativity (FRN), an ERP component thought to reflect reward prediction error, paralleled behavior. The FRN was sensitive to item-specific reward probabilities in participants who detected the deviant stimulus set, and the FRN was sensitive to overall reward probabilities in participants who did not. These results show that the FRN is sensitive to the utility of abstract rules and that the individual's representation of a task's states and actions shapes behavior as well as the FRN.