The catecholamine precursor Tyrosine reduces autonomic arousal and decreases decision thresholds in reinforcement learning and temporal discounting

Supplementation with the catecholamine precursor L-Tyrosine might enhance cognitive performance, but overall findings are mixed. Here, we investigate the effect of a single dose of tyrosine (2g) vs. placebo on two catecholamine-dependent trans-diagnostic traits: model-based control during reinforcement learning (2-step task) and temporal discounting, using a double-blind, placebo-controlled, within-subject design (n = 28 healthy male participants). We leveraged drift diffusion models in a hierarchical Bayesian framework to jointly model participants' choices and response times (RTS) in both tasks. Furthermore, comprehensive autonomic monitoring (heart rate, heart rate variability, pupillometry, spontaneous eye blink rate) was performed both pre- and post-supplementation, to explore potential physiological effects of supplementation. Across tasks, tyrosine consistently reduced participants' RTs without deteriorating task-performance. Diffusion modeling linked this effect to attenuated decision-thresholds in both tasks and further revealed increased model-based control (2-step task) and (if anything) attenuated temporal discounting. On the physiological level, participants' pupil dilation was predictive of the individual degree of temporal discounting. Tyrosine supplementation reduced physiological arousal as revealed by increases in pupil dilation variability and reductions in heart rate. Supplementation-related changes in physiological arousal predicted individual changes in temporal discounting. Our findings provide first evidence that tyrosine supplementation might impact psychophysiological parameters, and suggest that modeling approaches based on sequential sampling models can yield novel insights into latent cognitive processes modulated by amino-acid supplementation.

[Atraumatic rupture of the extensor hallucis longus tendon]

An atraumatic rupture of the extensor hallucis longus tendon is a rare injury. Chronic overload due to contact sports, bony anomalies, previous operations, and drug injections close to the tendon have been reported as a cause. We report the case of a young patient who presented with an acute loss of function in his extensor hallucis longus tendon for no apparent reason.

Batch Reinforcement Learning With a Nonparametric Off-Policy Policy Gradient

Off-policy reinforcement learning (RL) holds the promise of better data efficiency as it allows sample reuse and potentially enables safe interaction with the environment. Current off-policy policy gradient methods either suffer from high bias or high variance, delivering often unreliable estimates. The price of inefficiency becomes evident in real-world scenarios such as interaction-driven robot learning, where the success of RL has been rather limited, and a very high sample cost hinders straightforward application. In this paper, we propose a nonparametric Bellman equation, which can be solved in closed form. The solution is differentiable w.r.t the policy parameters and gives access to an estimation of the policy gradient. In this way, we avoid the high variance of importance sampling approaches, and the high bias of semi-gradient methods. We empirically analyze the quality of our gradient estimate against state-of-the-art methods, and show that it outperforms the baselines in terms of sample efficiency on classical control tasks.

Continuous Action Reinforcement Learning From a Mixture of Interpretable Experts

Reinforcement learning (RL) has demonstrated its ability to solve high dimensional tasks by leveraging non-linear function approximators. However, these successes are mostly achieved by 'black-box' policies in simulated domains. When deploying RL to the real world, several concerns regarding the use of a 'black-box' policy might be raised. In order to make the learned policies more transparent, we propose in this paper a policy iteration scheme that retains a complex function approximator for its internal value predictions but constrains the policy to have a concise, hierarchical, and human-readable structure, based on a mixture of interpretable experts. Each expert selects a primitive action according to a distance to a prototypical state. A key design decision to keep such experts interpretable is to select the prototypical states from trajectory data. The main technical contribution of the paper is to address the challenges introduced by this non-differentiable prototypical state selection procedure. Experimentally, we show that our proposed algorithm can learn compelling policies on continuous action deep RL benchmarks, matching the performance of neural network based policies, but returning policies that are more amenable to human inspection than neural network or linear-in-feature policies.

HEBO: An Empirical Study of Assumptions in Bayesian Optimisation

In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO’s empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multiobjective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation.

Gambling Environment Exposure Increases Temporal Discounting but Improves Model-Based Control in Regular Slot-Machine Gamblers

Gambling disorder is a behavioral addiction that negatively impacts personal finances, work, relationships and mental health. In this pre-registered study (https://osf.io/5ptz9/) we investigated the impact of real-life gambling environments on two computational markers of addiction, temporal discounting and model-based reinforcement learning. Gambling disorder is associated with increased temporal discounting and reduced model-based learning. Regular gamblers (n = 30, DSM-5 score range 3-9) performed both tasks in a neutral (café) and a gambling-related environment (slot-machine venue) in counterbalanced order. Data were modeled using drift diffusion models for temporal discounting and reinforcement learning via hierarchical Bayesian estimation. Replicating previous findings, gamblers discounted rewards more steeply in the gambling-related context. This effect was positively correlated with gambling related cognitive distortions (pre-registered analysis). In contrast to our pre-registered hypothesis, model-based reinforcement learning was improved in the gambling context. Here we show that temporal discounting and model-based reinforcement learning are modulated in opposite ways by real-life gambling cue exposure. Results challenge aspects of habit theories of addiction, and reveal that laboratory-based computational markers of psychopathology are under substantial contextual control.

Motor response vigour and visual fixation patterns reflect subjective valuation during intertemporal choice

Value-based decision-making is of central interest in cognitive neuroscience and psychology, as well as in the context of neuropsychiatric disorders characterised by decision-making impairments. Studies examining (neuro-)computational mechanisms underlying choice behaviour typically focus on participants’ decisions. However, there is increasing evidence that option valuation might also be reflected in motor response vigour and eye movements, implicit measures of subjective utility. To examine motor response vigour and visual fixation correlates of option valuation in intertemporal choice, we set up a task where the participants selected an option by pressing a grip force transducer, simultaneously tracking fixation shifts between options. As outlined in our preregistration (https://osf.io/k6jct), we used hierarchical Bayesian parameter estimation to model the choices assuming hyperbolic discounting, compared variants of the softmax and drift diffusion model, and assessed the relationship between response vigour and the estimated model parameters. The behavioural data were best explained by a drift diffusion model specifying a non-linear scaling of the drift rate by the subjective value differences. Replicating previous findings, we found a magnitude effect for temporal discounting, such that higher rewards were discounted less. This magnitude effect was further reflected in motor response vigour, such that stronger forces were exerted in the high vs. the low magnitude condition. Bayesian hierarchical linear regression further revealed higher grip forces, faster response times and a lower number of fixation shifts for trials with higher subjective value differences. An exploratory analysis revealed that subjective value sums across options showed an even more pronounced association with trial-wise grip force amplitudes. Our data suggest that subjective utility or implicit valuation is reflected in motor response vigour and visual fixation patterns during intertemporal choice. Taking into account response vigour might thus provide deeper insight into decision-making, reward valuation and maladaptive changes in these processes, e.g. in the context of neuropsychiatric disorders.

POS0774 ANTI-Sm AUTOANTIBODIES IDENTIFY A PHENOTYPE OF SEVERE SLE WITH AN ASSOCIATED SERUM BIOMARKER PROFILE

Background

Antibodies to Smith (Sm) have been described as one of the most specific autoantibodies for systemic lupus erythematosus (SLE). Other than its association with lupus nephritis, there is, however, limited understanding of its clinical significance1,2.

Objectives

To describe clinical associations and serum protein profiles of anti-Sm positivity in SLE.

Methods

Patients fulfilling SLE classification criteria who were followed longitudinally in a prospective multicentre cohort were studied according to their baseline anti-Sm antibody status. Comparison between Sm+ and Sm- patients was made using descriptive statistics. Clinical associations of Sm positivity with patient disease characteristics were studied using logistic regression. In a subset, 211 serum analytes were measured using Quantibody, Luminex and ELISA assays. Associations between serum proteins and Sm positivity were studied using Least Absolute Shrinkage and Selection Operator (LASSO) penalised regression, adjusting for demographics (age, sex, ethnicity) and medication use

Results

383 patients were studied with median (IQR) follow-up of 4.9 (2,9) years; 65 (17%) had positive anti-Sm antibodies. Sm+ patients were significantly more likely to be of non-European ancestry (OR 2.73, 95% CI 1.55-4.82, p<0.001), and to be positive for anti-dsDNA antibodies (OR 2.8, 95% CI 2.3-3.4, p<0.001), anti-RNP antibodies (OR 15.7, 95% CI 13.9-17.8, p<0.001), direct anti-globulin test (OR 2.36, 95% CI 2.07-2.7, p<0.001) and hypocomplementemia (OR 7.73, 95% CI 5.1-11.7, p<0.001). Sm+ patients were significantly more likely to have active disease during the observation period in a range of organ domains, including mucocutaneous, renal, vasculitis and fever.

More Sm+ patients had episodes of High Disease Activity Status (HDAS, SLEDAI-2K ≧10)3 (OR 3.07, 95% CI 1.70-5.54, p<0.001) and persistent active disease (time-adjusted mean SLEDAI-2K > 4) (OR 3.23. 95% CI 1.84-5.70, p<0.001). Conversely, fewer Sm+ patients attained LLDAS for ≥50% observed time (19.7% vs 41.8%, p=0.002). Sm+ patients were more likely to be treated with glucocorticoids, immunosuppressants, and rituximab. There was no significant difference in damage accrual between Sm + and Sm - patients.

In serum protein analysis (n=197, 29 Sm+), LASSO modelling retained 3 proteins associated with Sm+ status, CXCL13, IL1RL1 and FLT1, along with Asian ethnicity and age. In analysis including pairwise interaction between predictors, 28 Sm+ associated proteins were identified, including CCL4, VCAM1, IL1RL1, Fcg R IIB/C, TDGF1, CEACAM1, TIMP1, BMP5, GDF15, and TNFRSF17.

Conclusion

Anti-Sm autoantibodies, present in 17% of SLE patients, were strongly associated with classical disease manifestations, more severe disease activity, and a specific serological and proteomic profile. These findings suggest anti-Sm+ SLE as a specific disease subset.

References

[1]Barada, FA., B.S. Andrews, J.S. Davis, R.P. Taylor, Antibodies to Sm in patients with systemic lupus erythematosus. Correlation of Sm antibody titers with disease activity and other laboratory parameters. Arthritis Rheum, 1981. 24:1236-1244

[2]Arroyo-Avilla, M, Y. Santiago-Casas, G.McGwin, R.S. Cantor, M. Petri, R. Ramsey-Goldman, J.D. Reveille, R.P.Kimberly, G.S. Alarcon, L.M.Vila, E.E. Brown. Clinical Associations of anti-Smith antibodies in PROFILE: a multi-ethnic lupus cohort. 2015. 34:1217-1223

[3]Koelmeyer, R., H.T. Nim, M. Nikpour, Y.B. Sun, A. Kao, O. Guenther, E. Morand, and A. Hoi, High disease activity status suggests more severe disease and damage accrual in systemic lupus erythematosus. Lupus Sci Med, 2020. 7(1).

Acknowledgements

I would like to acknowledge participants and clinicians involved with the Australian Lupus Registry & Biobank

Disclosure of Interests

None declared

Category-sensitive incidental reinstatement in medial temporal lobe subregions during word recognition

During associative retrieval, the brain reinstates neural representations that were present during encoding. The human medial temporal lobe (MTL), with its subregions hippocampus (HC), perirhinal cortex (PRC), and parahippocampal cortex (PHC), plays a central role in neural reinstatement. Previous studies have given compelling evidence for reinstatement in the MTL during explicitly instructed associative retrieval. High-confident recognition may be similarly accompanied by recollection of associated information from the encoding context. It is unclear, however, whether high-confident recognition memory elicits reinstatement in the MTL even in the absence of an explicit instruction to retrieve associated information. Here, we addressed this open question using high-resolution fMRI. Twenty-eight male and female human volunteers engaged in a recognition memory task for words that they had previously encoded together with faces and scenes. Using complementary univariate and multivariate approaches, we show that MTL subregions including the PRC, PHC, and HC differentially reinstate category-sensitive representations during high-confident word recognition, even though no explicit instruction to retrieve the associated category was given. This constitutes novel evidence that high-confident recognition memory is accompanied by incidental reinstatement of associated category information in MTL subregions, and supports a functional model of the MTL that emphasizes content-sensitive representations during both encoding and retrieval.

Fixing a Subtrochanteric Femoral Fracture with a Humerus Nail

A 17-year-old paraplegic patient sustained a subtrochanteric femoral fracture due to inadequate trauma. The unusual anatomical conditions associated with his congenital paraplegia did not allow treatment with a standard intramedullary implant for the femur. Because his soft tissues were already compromised, alternative options like plate osteosynthesis were considered unfavourable as a salvage procedure. Therefore, we used an implant designed for the humerus. A satisfactory result of osteosynthesis was achieved despite varus deformity, shortening and rotational error.

Robot Learning From Randomized Simulations: A Review

The rise of deep learning has caused a paradigm shift in robotics research, favoring methods that require large amounts of data. Unfortunately, it is prohibitively expensive to generate such data sets on a physical platform. Therefore, state-of-the-art approaches learn in simulation where data generation is fast as well as inexpensive and subsequently transfer the knowledge to the real robot (sim-to-real). Despite becoming increasingly realistic, all simulators are by construction based on models, hence inevitably imperfect. This raises the question of how simulators can be modified to facilitate learning robot control policies and overcome the mismatch between simulation and reality, often called the “reality gap.” We provide a comprehensive review of sim-to-real research for robotics, focusing on a technique named “domain randomization” which is a method for learning from randomized simulations.

Trial-wise exposure to visual emotional cues increases physiological arousal but not temporal discounting

Humans and many animals devalue future rewards as a function of time (temporal discounting). Increased discounting has been linked to various psychiatric conditions, including substance-use-disorders, behavioral addictions, and obesity. Despite its high intra-individual stability, temporal discounting is partly under contextual control. One prominent manipulation that has been linked to increases in discounting is the exposure to highly arousing appetitive cues. However, results from trial-wise cue exposure studies appear highly mixed, and changes in physiological arousal were not adequately controlled. Here we tested the effects of appetitive (erotic), aversive, and neutral visual cues on temporal discounting in 35 healthy male participants. The contribution of single-trial physiological arousal was assessed using comprehensive monitoring of autonomic activity (pupil size, heart rate, electrodermal activity). Physiological arousal was elevated following aversive and in particular erotic cues. In contrast to our pre-registered hypothesis, steepness of temporal discounting was not significantly affected by emotional cues of either valence. Aversive cues tended to increase decision noise. Computational modeling revealed that trial-wise arousal only accounted for minor variance over and above aversive and erotic condition effects, arguing against a general effect of physiological arousal on temporal discounting.

ExGenNet: Learning to Generate Robotic Facial Expression Using Facial Expression Recognition

The ability of a robot to generate appropriate facial expressions is a key aspect of perceived sociability in human-robot interaction. Yet many existing approaches rely on the use of a set of fixed, preprogrammed joint configurations for expression generation. Automating this process provides potential advantages to scale better to different robot types and various expressions. To this end, we introduce ExGenNet, a novel deep generative approach for facial expressions on humanoid robots. ExGenNets connect a generator network to reconstruct simplified facial images from robot joint configurations with a classifier network for state-of-the-art facial expression recognition. The robots' joint configurations are optimized for various expressions by backpropagating the loss between the predicted expression and intended expression through the classification network and the generator network. To improve the transfer between human training images and images of different robots, we propose to use extracted features in the classifier as well as in the generator network. Unlike most studies on facial expression generation, ExGenNets can produce multiple configurations for each facial expression and be transferred between robots. Experimental evaluations on two robots with highly human-like faces, Alfie (Furhat Robot) and the android robot Elenoide, show that ExGenNet can successfully generate sets of joint configurations for predefined facial expressions on both robots. This ability of ExGenNet to generate realistic facial expressions was further validated in a pilot study where the majority of human subjects could accurately recognize most of the generated facial expressions on both the robots.

The WHO 2016 diagnostic criteria for Acute Myeloid leukemia with myelodysplasia related changes (AML-MRC) produce a very heterogeneous entity: A retrospective analysis of the FAB subtype RAEB-T

We studied 79 patients with AML-MRC or RAEB-T, who were later reclassified according to the WHO classification. Marrow slides were examined cytomorphologically with regard to dysplasia. Patients were followed up until March 2020. Thirty-one patients underwent allogeneic stem cell transplantation (median survival (ms) 16 months), 14 were treated with induction chemotherapy (ms 8.4 months), 18 received hypomethylating agents (ms 9.2 months), 16 received low dose chemotherapy or best supportive care (ms 2.4 months). Only 30.4 % fulfilled the morphologic WHO criteria. 46.8 % were classified as AML-MRC by an antecedent MDS, 54.4 % of the pts were classified by MDS-related chromosomal abnormalities. 5 % did not fulfill any of the criteria and were entered based on 20-29 % medullary blasts. There was no difference in ms between pts presenting with > 50 % dysplasia as compared to pts with dysplasia between 10 % and 50 % (ms 9.1 vs 9.9 months, p = n.s.) or for pts with antecedent MDS (ms 9.1 vs 8.9 months, p = n.s.). Myelodysplasia-related cytogenetic abnormalities were associated with a worse outcome (ms 8.1 vs 13.5 months, p = 0.026). AML-MRC in its current definition is a heterogenous entity. Dysplasia of ≥ 50 % in ≥ two lineages is not helpful for diagnostics and prognostication and therefore should be deleted in future classifications. We recommend utilizing the WHO guidelines for defining dysplasia (10 % or greater in ≥ 1 of the three myeloid cell lines) assisting in establishing the diagnosis of MDS.

Heterogeneous Microscopic Dynamics of Intruded Water in a Superhydrophobic Nanoconfinement: Neutron Scattering and Molecular Modeling

With their strong confining porosity and versatile surface chemistry, zeolitic imidazolate frameworks-including the prototypical ZIF-8-display exceptional properties for various applications. In particular, the forced intrusion of water at high pressure (∼25 MPa) into ZIF-8 nanopores is of interest for energy storage. Such a system reveals also ideal to study experimentally water dynamics and thermodynamics in an ultrahydrophobic confinement. Here, we report on neutron scattering experiments to probe the molecular dynamics of water within ZIF-8 nanopores under high pressure up to 38 MPa. In addition to an overall confinement-induced slowing down, we provide evidence for strong dynamical heterogeneities with different underlying molecular dynamics. Using complementary molecular simulations, these heterogeneities are found to correspond to different microscopic mechanisms inherent to vicinal molecules located in strongly adsorbing sites (ligands) and other molecules nanoconfined in the cavity center. These findings unveil a complex microscopic dynamics, which results from the combination of surface residence times and exchanges between the cavity surface and center.