Automatic data augmentation for medical image segmentation using Adaptive Sequence-length based Deep Reinforcement Learning

Although existing deep reinforcement learning-based approaches have achieved some success in image augmentation tasks, their effectiveness and adequacy for data augmentation in intelligent medical image analysis are still unsatisfactory. Therefore, we propose a novel Adaptive Sequence-length based Deep Reinforcement Learning (ASDRL) model for Automatic Data Augmentation (AutoAug) in intelligent medical image analysis. The improvements of ASDRL-AutoAug are two-fold: (i) To remedy the problem of some augmented images being invalid, we construct a more accurate reward function based on different variations of the augmentation trajectories. This reward function assesses the validity of each augmentation transformation more accurately by introducing different information about the validity of the augmented images. (ii) Then, to alleviate the problem of insufficient augmentation, we further propose a more intelligent automatic stopping mechanism (ASM). ASM feeds a stop signal to the agent automatically by judging the adequacy of image augmentation. This ensures that each transformation before stopping the augmentation can smoothly improve the model performance. Extensive experimental results on three medical image segmentation datasets show that (i) ASDRL-AutoAug greatly outperforms the state-of-the-art data augmentation methods in medical image segmentation tasks, (ii) the proposed improvements are both effective and essential for ASDRL-AutoAug to achieve superior performance, and the new reward evaluates the transformations more accurately than existing reward functions, and (iii) we also demonstrate that ASDRL-AutoAug is adaptive for different images in terms of sequence length, as well as generalizable across different segmentation models.

AI-driven molecular generation of not-patented pharmaceutical compounds using world open patent data

Developing compounds with novel structures is important for the production of new drugs. From an intellectual perspective, confirming the patent status of newly developed compounds is essential, particularly for pharmaceutical companies. The generation of a large number of compounds has been made possible because of the recent advances in artificial intelligence (AI). However, confirming the patent status of these generated molecules has been a challenge because there are no free and easy-to-use tools that can be used to determine the novelty of the generated compounds in terms of patents in a timely manner; additionally, there are no appropriate reference databases for pharmaceutical patents in the world. In this study, two public databases, SureChEMBL and Google Patents Public Datasets, were used to create a reference database of drug-related patented compounds using international patent classification. An exact structure search system was constructed using InChIKey and a relational database system to rapidly search for compounds in the reference database. Because drug-related patented compounds are a good source for generative AI to learn useful chemical structures, they were used as the training data. Furthermore, molecule generation was successfully directed by increasing and decreasing the number of generated patented compounds through incorporation of patent status (i.e., patented or not) into learning. The use of patent status enabled generation of novel molecules with high drug-likeness. The generation using generative AI with patent information would help efficiently propose novel compounds in terms of pharmaceutical patents. Scientific contribution: In this study, a new molecule-generation method that takes into account the patent status of molecules, which has rarely been considered but is an important feature in drug discovery, was developed. The method enables the generation of novel molecules based on pharmaceutical patents with high drug-likeness and will help in the efficient development of effective drug compounds.

Cross-domain policy adaptation with dynamics alignment

The implementation of robotic reinforcement learning is hampered by problems such as an unspecified reward function and high training costs. Many previous works have used cross-domain policy transfer to obtain the policy of the problem domain. However, these researches require paired and aligned dynamics trajectories or other interactions with the environment. We propose a cross-domain dynamics alignment framework for the problem domain policy acquisition that can transfer the policy trained in the source domain to the problem domain. Our framework aims to learn dynamics alignment across two domains that differ in agents' physical parameters (armature, rotation range, or torso mass) or agents' morphologies (limbs). Most importantly, we learn dynamics alignment between two domains using unpaired and unaligned dynamics trajectories. For these two scenarios, we propose a cross-physics-domain policy adaptation algorithm (CPD) and a cross-morphology-domain policy adaptation algorithm (CMD) based on our cross-domain dynamics alignment framework. In order to improve the performance of policy in the source domain so that a better policy can be transferred to the problem domain, we propose the Boltzmann TD3 (BTD3) algorithm. We conduct diverse experiments on agent continuous control domains to demonstrate the performance of our approaches. Experimental results show that our approaches can obtain better policies and higher rewards for the agents in the problem domains even when the dataset of the problem domain is small.

Modeling pedestrian behavior in pedestrian-vehicle near misses: A continuous Gaussian Process Inverse Reinforcement Learning (GP-IRL) approach

Using simulation models to conduct safety assessments can have several advantages as it enables the evaluation of the safety of various design and traffic management options before actually making changes. However, limited studies have developed microsimulation models for the safety evaluation of active road users such as pedestrians. This can be attributed to the limited ability of simulation models to capture the heterogeneity in pedestrian behavior and their complex collision avoidance mechanisms. Therefore, the objective of this study is to develop an agent-based framework to realistically model pedestrian behavior in near misses and to improve the understanding of pedestrian evasive action mechanisms in interactions with vehicles. Pedestrian-vehicle conflicts are modeled using the Markov Decision Process (MDP) framework. A continuous Gaussian Process Inverse Reinforcement Learning (GP-IRL) approach is implemented to retrieve pedestrians' reward functions and infer their collision avoidance mechanisms in conflict situations. Video data from a congested intersection in Shanghai, China is used as a case study. Trajectories of pedestrians and vehicles involved in traffic conflicts were extracted with computer vision algorithms. A Deep Reinforcement Learning (DRL) model is used to estimate optimal pedestrian policies in traffic conflicts. Results show that the developed model predicted pedestrian trajectories and their evasive action mechanisms (i.e., swerving maneuver and speed changing) in conflict situations with high accuracy. As well, the model provided predictions of the post encroachment time (PET) conflict indicator that strongly correlated with the corresponding values of the field-measured conflicts. This study is a crucial step in developing a safety-oriented microsimulation tool for pedestrians in mixed traffic conditions.

Trace amine-associated receptor 1 (TAAR1): Potential application in mood disorders: A systematic review

There is a need for innovation with respect to therapeutics in psychiatry. Available evidence indicates that the trace amine-associated receptor 1 (TAAR1) agonist SEP-363856 is promising, as it improves measures of cognitive and reward function in schizophrenia. Hedonic and cognitive impairments are transdiagnostic and constitute major burdens in mood disorders. Herein, we systematically review the behavioural and genetic literature documenting the role of TAAR1 in reward and cognitive function, and propose a mechanistic model of TAAR1's functions in the brain. Notably, TAAR1 activity confers antidepressant-like effects, enhances attention and response inhibition, and reduces compulsive reward seeking without impairing normal function. Further characterization of the responsible mechanisms suggests ion-homeostatic, metabolic, neurotrophic, and anti-inflammatory enhancements in the limbic system. Multiple lines of evidence establish the viability of TAAR1 as a biological target for the treatment of mood disorders. Furthermore, the evidence suggests a role for TAAR1 in reward and cognitive function, which is attributed to a cascade of events that are relevant to the cellular integrity and function of the central nervous system.

Mice optimize timed decisions about probabilistic outcomes under deadlines

Optimal performance in temporal decisions requires the integration of timing uncertainty with environmental statistics such as probability or cost functions. Reward maximization under response deadlines constitutes one of the most stringent examples of these problems. The current study investigated whether and how mice can optimize their timing behavior in a complex experimental setting under a response deadline in which reward maximization required the integration of timing uncertainty with a geometrically increasing probability/decreasing cost function. Mice optimized their performance under seconds-long response deadlines when the underlying function was reward probability but approached this level of performance when the underlying function was reward cost, only under the assumption of logarithmically scaled subjective costs. The same subjects were then tested in a timed response inhibition task characterized by response rules that conflicted with the initial task, not responding earlier than a schedule as opposed to not missing the deadline. Irrespective of original test groups, mice optimized the timing of their inhibitory control in the second experiment. These results provide strong support for the ubiquity of optimal temporal risk assessment in mice.

'Proactive' use of cue-context congruence for building reinforcement learning's reward function

BACKGROUND

Reinforcement learning is a fundamental form of learning that may be formalized using the Bellman equation. Accordingly an agent determines the state value as the sum of immediate reward and of the discounted value of future states. Thus the value of state is determined by agent related attributes (action set, policy, discount factor) and the agent's knowledge of the environment embodied by the reward function and hidden environmental factors given by the transition probability. The central objective of reinforcement learning is to solve these two functions outside the agent's control either using, or not using a model.

RESULTS

In the present paper, using the proactive model of reinforcement learning we offer insight on how the brain creates simplified representations of the environment, and how these representations are organized to support the identification of relevant stimuli and action. Furthermore, we identify neurobiological correlates of our model by suggesting that the reward and policy functions, attributes of the Bellman equitation, are built by the orbitofrontal cortex (OFC) and the anterior cingulate cortex (ACC), respectively.

CONCLUSIONS

Based on this we propose that the OFC assesses cue-context congruence to activate the most context frame. Furthermore given the bidirectional neuroanatomical link between the OFC and model-free structures, we suggest that model-based input is incorporated into the reward prediction error (RPE) signal, and conversely RPE signal may be used to update the reward-related information of context frames and the policy underlying action selection in the OFC and ACC, respectively. Furthermore clinical implications for cognitive behavioral interventions are discussed.

Sleep and circadian contributions to adolescent alcohol use disorder

Adolescence is a time of marked changes across sleep, circadian rhythms, brain function, and alcohol use. Starting at puberty, adolescents' endogenous circadian rhythms and preferred sleep times shift later, often leading to a mismatch with the schedules imposed by secondary education. This mismatch induces circadian misalignment and sleep loss, which have been associated with affect dysregulation, increased drug and alcohol use, and other risk-taking behaviors in adolescents and adults. In parallel to developmental changes in sleep, adolescent brains are undergoing structural and functional changes in the circuits subserving the pursuit and processing of rewards. These developmental changes in reward processing likely contribute to the initiation of alcohol use during adolescence. Abundant evidence indicates that sleep and circadian rhythms modulate reward function, suggesting that adolescent sleep and circadian disturbance may contribute to altered reward function, and in turn, alcohol involvement. In this review, we summarize the relevant evidence and propose that these parallel developmental changes in sleep, circadian rhythms, and neural processing of reward interact to increase risk for alcohol use disorder (AUD).

Reduced reward anticipation in youth at high-risk for unipolar depression: a preliminary study

•

Depression is characterized by reduced neural response to reward, particularly in the ventral striatum.

•

Few studies have examined if alterations in reward functioning are present before the onset of depression.

•

Youth at high- and low-familial risk for depression completed a reward task during a fMRI scan.

•

High-risk youth had significantly less ventral striatal reactivity than low-risk youth during reward anticipation.

•

Reward functioning is altered in individuals at high-familial risk for depression before the onset of the disorder.

Offspring of depressed parents are at risk for depression and recent evidence suggests that reduced positive affect (PA) may be a marker of risk. We investigated whether self-reports of PA and fMRI-measured striatal response to reward, a neural correlate of PA, are reduced in adolescent youth at high familial risk for depression (HR) relative to youth at low familial risk for depression (LR). Functional magnetic resonance imaging assessments were conducted with 14 HR and 12 LR youth. All youth completed an ecological momentary assessment protocol to measure PA in natural settings and a self-report measure of depression symptomatology. Analyses found that HR youth demonstrated lower striatal response than LR youth during both reward anticipation and outcome. However, after controlling for youth self-reports of depression, HR youth demonstrated lower striatal response than LR youth only during reward anticipation. No significant differences were found between HR and LR youth on subjective ratings of PA or depressive symptoms. Results are consistent with previous findings that reduced reward response is a marker of risk for depression, particularly during reward anticipation, even in the absence of (or accounting for) disrupted subjective mood. Further examinations of prospective associations between reward response and depression onset are needed.