Temporal discounting in adolescents and adults with Tourette syndrome

Tourette syndrome is a neurodevelopmental disorder associated with hyperactivity in dopaminergic networks. Dopaminergic hyperactivity in the basal ganglia has previously been linked to increased sensitivity to positive reinforcement and increases in choice impulsivity. In this study, we examine whether this extends to changes in temporal discounting, where impulsivity is operationalized as an increased preference for smaller-but-sooner over larger-but-later rewards. We assessed intertemporal choice in two studies including nineteen adolescents (age: mean[sd] = 14.21[±2.37], 13 male subjects) and twenty-five adult patients (age: mean[sd] = 29.88 [±9.03]; 19 male subjects) with Tourette syndrome and healthy age- and education matched controls. Computational modeling using exponential and hyperbolic discounting models via hierarchical Bayesian parameter estimation revealed reduced temporal discounting in adolescent patients, and no evidence for differences in adult patients. Results are discussed with respect to neural models of temporal discounting, dopaminergic alterations in Tourette syndrome and the developmental trajectory of temporal discounting. Specifically, adolescents might show attenuated discounting due to improved inhibitory functions that also affect choice impulsivity and/or the developmental trajectory of executive control functions. Future studies would benefit from a longitudinal approach to further elucidate the developmental trajectory of these effects.

Perception and prediction of the putting distance of robot putting movements under different visual/viewing conditions

The purpose of this paper is to examine, whether and under which conditions humans are able to predict the putting distance of a robotic device. Based on the “flash-lag effect” (FLE) it was expected that the prediction errors increase with increasing putting velocity. Furthermore, we hypothesized that the predictions are more accurate and more confident if human observers operate under full vision (F-RCHB) compared to either temporal occlusion (I-RCHB) or spatial occlusion (invisible ball, F-RHC, or club, F-B). In two experiments, 48 video sequences of putt movements performed by a BioRob robot arm were presented to thirty-nine students (age: 24.49±3.20 years). In the experiments, video sequences included six putting distances (1.5, 2.0, 2.5, 3.0, 3.5, and 4.0 m; experiment 1) under full versus incomplete vision (F-RCHB versus I-RCHB) and three putting distances (2. 0, 3.0, and 4.0 m; experiment 2) under the four visual conditions (F-RCHB, I-RCHB, F-RCH, and F-B). After the presentation of each video sequence, the participants estimated the putting distance on a scale from 0 to 6 m and provided their confidence of prediction on a 5-point scale. Both experiments show comparable results for the respective dependent variables (error and confidence measures). The participants consistently overestimated the putting distance under the full vision conditions; however, the experiments did not show a pattern that was consistent with the FLE. Under the temporal occlusion condition, a prediction was not possible; rather a random estimation pattern was found around the centre of the prediction scale (3 m). Spatial occlusion did not affect errors and confidence of prediction. The experiments indicate that temporal constraints seem to be more critical than spatial constraints. The FLE may not apply to distance prediction compared to location estimation.

Magnitude of Diffusion- and Transverse Dispersion-Induced Isotope Fractionation of Organic Compounds in Aqueous Systems

Determining whether aqueous diffusion and dispersion lead to significant isotope fractionation is important for interpreting the isotope ratios of organic contaminants in groundwater. We performed diffusion experiments with modified Stokes diaphragm cells and transverse-dispersion experiments in quasi-two-dimensional flow-through sediment tank systems to explore isotope fractionation for benzene, toluene, ethylbenzene, 2,6-dichlorobenzamide, and metolachlor at natural isotopic abundance. We observed very small to negligible diffusion- and transverse-dispersion-induced isotope enrichment factors (ε < -0.4 ‰), with changes in carbon and nitrogen isotope values within ±0.5‰ and ±1‰, respectively. Isotope effects of diffusion did not show a clear correlation with isotopologue mass with calculated power-law exponents β close to zero (0.007 < β < 0.1). In comparison to ions, noble gases, and labeled compounds, three aspects stand out. (i) If a mass dependence is derived from collision theory, then isotopologue masses of polyatomic molecules would be affected by isotopes of multiple elements resulting in very small expected effects. (ii) However, collisions do not necessarily lead to translational movement but can excite molecular vibrations or rotations minimizing the mass dependence. (iii) Solute-solvent interactions like H-bonds can further minimize the effect of collisions. Modeling scenarios showed that an inadequate model choice, or erroneous choice of β, can greatly overestimate the isotope fractionation by diffusion and, consequently, transverse dispersion. In contrast, available data for chlorinated solvent and gasoline contaminants at natural isotopic abundance suggest that in field scenarios, a potential additional uncertainty from aqueous diffusion or dispersion would add to current instrumental uncertainties on carbon or nitrogen isotope values (±1‰) with an additional ±1‰ at most.

Assessing Transferability From Simulation to Reality for Reinforcement Learning

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the 'Simulation Optimization Bias' (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results on two different second order nonlinear systems show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real systems without any additional training.

Attenuated Directed Exploration during Reinforcement Learning in Gambling Disorder

Gambling disorder (GD) is a behavioral addiction associated with impairments in value-based decision-making and behavioral flexibility and might be linked to changes in the dopamine system. Maximizing long-term rewards requires a flexible trade-off between the exploitation of known options and the exploration of novel options for information gain. This exploration-exploitation trade-off is thought to depend on dopamine neurotransmission. We hypothesized that human gamblers would show a reduction in directed (uncertainty-based) exploration, accompanied by changes in brain activity in a fronto-parietal exploration-related network. Twenty-three frequent, non-treatment seeking gamblers and twenty-three healthy matched controls (all male) performed a four-armed bandit task during functional magnetic resonance imaging (fMRI). Computational modeling using hierarchical Bayesian parameter estimation revealed signatures of directed exploration, random exploration, and perseveration in both groups. Gamblers showed a reduction in directed exploration, whereas random exploration and perseveration were similar between groups. Neuroimaging revealed no evidence for group differences in neural representations of basic task variables (expected value, prediction errors). Our hypothesis of reduced frontal pole (FP) recruitment in gamblers was not supported. Exploratory analyses showed that during directed exploration, gamblers showed reduced parietal cortex and substantia-nigra/ventral-tegmental-area activity. Cross-validated classification analyses revealed that connectivity in an exploration-related network was predictive of group status, suggesting that connectivity patterns might be more predictive of problem gambling than univariate effects. Findings reveal specific reductions of strategic exploration in gamblers that might be linked to altered processing in a fronto-parietal network and/or changes in dopamine neurotransmission implicated in GD.SIGNIFICANCE STATEMENT Wiehler et al. (2021) report that gamblers rely less on the strategic exploration of unknown, but potentially better rewards during reward learning. This is reflected in a related network of brain activity. Parameters of this network can be used to predict the presence of problem gambling behavior in participants.

Rewards that are near increase impulsive action

In modern society, the natural drive to behave impulsively in order to obtain rewards must often be curbed. A continued failure to do so is associated with a range of outcomes including drug abuse, pathological gambling, and obesity. Here, we used virtual reality technology to investigate whether spatial proximity to rewards has the power to exacerbate the drive to behave impulsively toward them. We embedded two behavioral tasks measuring distinct forms of impulsive behavior, impulsive action, and impulsive choice, within an environment rendered in virtual reality. Participants responded to three-dimensional cues representing food rewards located in either near or far space. Bayesian analyses revealed that participants were significantly less able to stop motor actions when rewarding cues were near compared with when they were far. Since factors normally associated with proximity were controlled for, these results suggest that proximity plays a distinctive role in driving impulsive actions for rewards.

Multi-Channel Interactive Reinforcement Learning for Sequential Tasks

The ability to learn new tasks by sequencing already known skills is an important requirement for future robots. Reinforcement learning is a powerful tool for this as it allows for a robot to learn and improve on how to combine skills for sequential tasks. However, in real robotic applications, the cost of sample collection and exploration prevent the application of reinforcement learning for a variety of tasks. To overcome these limitations, human input during reinforcement can be beneficial to speed up learning, guide the exploration and prevent the choice of disastrous actions. Nevertheless, there is a lack of experimental evaluations of multi-channel interactive reinforcement learning systems solving robotic tasks with input from inexperienced human users, in particular for cases where human input might be partially wrong. Therefore, in this paper, we present an approach that incorporates multiple human input channels for interactive reinforcement learning in a unified framework and evaluate it on two robotic tasks with 20 inexperienced human subjects. To enable the robot to also handle potentially incorrect human input we incorporate a novel concept for self-confidence, which allows the robot to question human input after an initial learning phase. The second robotic task is specifically designed to investigate if this self-confidence can enable the robot to achieve learning progress even if the human input is partially incorrect. Further, we evaluate how humans react to suggestions of the robot, once the robot notices human input might be wrong. Our experimental evaluations show that our approach can successfully incorporate human input to accelerate the learning process in both robotic tasks even if it is partially wrong. However, not all humans were willing to accept the robot's suggestions or its questioning of their input, particularly if they do not understand the learning process and the reasons behind the robot's suggestions. We believe that the findings from this experimental evaluation can be beneficial for the future design of algorithms and interfaces of interactive reinforcement learning systems used by inexperienced users.

Hierarchical Tactile-Based Control Decomposition of Dexterous In-Hand Manipulation Tasks

In-hand manipulation and grasp adjustment with dexterous robotic hands is a complex problem that not only requires highly coordinated finger movements but also deals with interaction variability. The control problem becomes even more complex when introducing tactile information into the feedback loop. Traditional approaches do not consider tactile feedback and attempt to solve the problem either by relying on complex models that are not always readily available or by constraining the problem in order to make it more tractable. In this paper, we propose a hierarchical control approach where a higher level policy is learned through reinforcement learning, while low level controllers ensure grip stability throughout the manipulation action. The low level controllers are independent grip stabilization controllers based on tactile feedback. The independent controllers allow reinforcement learning approaches to explore the manipulation tasks state-action space in a more structured manner. We show that this structure allows learning the unconstrained task with RL methods that cannot learn it in a non-hierarchical setting. The low level controllers also provide an abstraction to the tactile sensors input, allowing transfer to real robot platforms. We show preliminary results of the transfer of policies trained in simulation to the real robot hand.

Dopamine and Risky Decision-Making in Gambling Disorder

Gambling disorder is a behavioral addiction associated with impairments in value-based decision-making and cognitive control. These functions are thought to be regulated by dopamine within fronto-striatal circuits, but the role of altered dopamine neurotransmission in the etiology of gambling disorder remains controversial. Preliminary evidence suggests that increasing frontal dopamine tone might improve cognitive functioning in gambling disorder. We therefore examined whether increasing frontal dopamine tone via a single dose of the catechol-O-methyltransferase (COMT) inhibitor tolcapone would reduce risky choice in human gamblers (n = 14) in a randomized double-blind placebo-controlled crossover study. Data were analyzed using hierarchical Bayesian parameter estimation and a combined risky choice drift diffusion model (DDM). Model comparison revealed a nonlinear mapping from value differences to trial-wise drift rates, confirming recent findings. An increase in risk-taking under tolcapone versus placebo was about five times more likely, given the data, than a decrease [Bayes factor (BF) = 0.2]. Examination of drug effects on diffusion model parameters revealed that an increase in the value dependency of the drift rate under tolcapone was about thirteen times more likely than a decrease (BF = 0.073). In contrast, a reduction in the maximum drift rate under tolcapone was about seven times more likely than an increase (BF = 7.51). Results add to previous work on COMT inhibitors in behavioral addictions and to mounting evidence for the applicability of diffusion models in value-based decision-making. Future work should focus on individual genetic, clinical and cognitive factors that might account for heterogeneity in the effects of COMT inhibition.

Dopaminergic modulation of the exploration/exploitation trade-off in human decision-making

Involvement of dopamine in regulating exploration during decision-making has long been hypothesized, but direct causal evidence in humans is still lacking. Here, we use a combination of computational modeling, pharmacological intervention and functional magnetic resonance imaging to address this issue. Thirty-one healthy male participants performed a restless four-armed bandit task in a within-subjects design under three drug conditions: 150 mg of the dopamine precursor L-dopa, 2 mg of the D2 receptor antagonist haloperidol, and placebo. Choices were best explained by an extension of an established Bayesian learning model accounting for perseveration, directed exploration and random exploration. Modeling revealed attenuated directed exploration under L-dopa, while neural signatures of exploration, exploitation and prediction error were unaffected. Instead, L-dopa attenuated neural representations of overall uncertainty in insula and dorsal anterior cingulate cortex. Our results highlight the computational role of these regions in exploration and suggest that dopamine modulates how this circuit tracks accumulating uncertainty during decision-making.

The drift diffusion model as the choice rule in inter-temporal and risky choice: A case study in medial orbitofrontal cortex lesion patients and controls

Sequential sampling models such as the drift diffusion model (DDM) have a long tradition in research on perceptual decision-making, but mounting evidence suggests that these models can account for response time (RT) distributions that arise during reinforcement learning and value-based decision-making. Building on this previous work, we implemented the DDM as the choice rule in inter-temporal choice (temporal discounting) and risky choice (probability discounting) using hierarchical Bayesian parameter estimation. We validated our approach in data from nine patients with focal lesions to the ventromedial prefrontal cortex / medial orbitofrontal cortex (vmPFC/mOFC) and nineteen age- and education-matched controls. Model comparison revealed that, for both tasks, the data were best accounted for by a variant of the drift diffusion model including a non-linear mapping from value-differences to trial-wise drift rates. Posterior predictive checks confirmed that this model provided a superior account of the relationship between value and RT. We then applied this modeling framework and 1) reproduced our previous results regarding temporal discounting in vmPFC/mOFC patients and 2) showed in a previously unpublished data set on risky choice that vmPFC/mOFC patients exhibit increased risk-taking relative to controls. Analyses of DDM parameters revealed that patients showed substantially increased non-decision times and reduced response caution during risky choice. In contrast, vmPFC/mOFC damage abolished neither scaling nor asymptote of the drift rate. Relatively intact value processing was also confirmed using DDM mixture models, which revealed that in both groups >98% of trials were better accounted for by a DDM with value modulation than by a null model without value modulation. Our results highlight that novel insights can be gained from applying sequential sampling models in studies of inter-temporal and risky decision-making in cognitive neuroscience.