A Survey on Causal Reinforcement Learning

While reinforcement learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these causal RL (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide the existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov decision process (MDP), partially observed MDP (POMDP), multiarmed bandits (MABs), imitation learning (IL), and dynamic treatment regime (DTR). Each of them represents a distinct type of causal graphical illustration. Moreover, we summarize the evaluation matrices and open sources, while we discuss emerging applications, along with promising prospects for the future development of CRL.

NeuroLISP: High-level symbolic programming with attractor neural networks

Despite significant improvements in contemporary machine learning, symbolic methods currently outperform artificial neural networks on tasks that involve compositional reasoning, such as goal-directed planning and logical inference. This illustrates a computational explanatory gap between cognitive and neurocomputational algorithms that obscures the neurobiological mechanisms underlying cognition and impedes progress toward human-level artificial intelligence. Because of the strong relationship between cognition and working memory control, we suggest that the cognitive abilities of contemporary neural networks are limited by biologically-implausible working memory systems that rely on persistent activity maintenance and/or temporal nonlocality. Here we present NeuroLISP, an attractor neural network that can represent and execute programs written in the LISP programming language. Unlike previous approaches to high-level programming with neural networks, NeuroLISP features a temporally-local working memory based on itinerant attractor dynamics, top-down gating, and fast associative learning, and implements several high-level programming constructs such as compositional data structures, scoped variable binding, and the ability to manipulate and execute programmatic expressions in working memory (i.e., programs can be treated as data). Our computational experiments demonstrate the correctness of the NeuroLISP interpreter, and show that it can learn non-trivial programs that manipulate complex derived data structures (multiway trees), perform compositional string manipulation operations (PCFG SET task), and implement high-level symbolic AI algorithms (first-order unification). We conclude that NeuroLISP is an effective neurocognitive controller that can replace the symbolic components of hybrid models, and serves as a proof of concept for further development of high-level symbolic programming in neural networks.

A Novel Application of Levenshtein Distance for Assessment of High-Level Motor Planning Underlying Performance During Learning of Complex Motor Sequences

Few studies have examined high-level motor plans underlying cognitive-motor performance during practice of complex action sequences. These investigations have assessed performance through fairly simple metrics without examining how practice affects the structures of action sequences. By adapting the Levenshtein distance (LD) method to the motor domain, we propose a computational approach to accurately capture performance dynamics during practice of action sequences. Practice performance dynamics were assessed by computing the LD based on the number of insertions, deletions, and substitutions of actions needed to transform any sequence into a reference sequence (having a minimal number of actions to complete the task). Also, combining LD-based performance with mental workload metrics allowed assessment of cognitive-motor efficiency dynamics. This approach was tested on the Tower of Hanoi task. The findings revealed that throughout practice this method could capture: i) action sequence performance improvements as indexed by a reduced LD (decrease of insertions and substitutions), ii) structural modifications of the high-level plans, iii) an attenuation of mental workload, and iv) enhanced cognitive-motor efficiency. This effort complements prior work examining the practice of complex action sequences in healthy adults and has potential for probing cognitive-motor impairment in clinical populations as well as the development/assessment of cognitive robotic controllers.

Changes in motor performance and mental workload during practice of reaching movements: a team dynamics perspective

Few investigations have examined mental workload during motor practice or learning in a context of team dynamics. This study examines the underlying cognitive-motor processes of motor practice by assessing the changes in motor performance and mental workload during practice of reaching movements. Individuals moved a robotic arm to reach targets as fast and as straight as possible while satisfying the task requirement of avoiding a collision between the end-effector and the workspace limits. Individuals practiced the task either alone (HA group) or with a synthetic teammate (HRT group), which regulated the effector velocity to help satisfy the task requirements. The findings revealed that the performance of both groups improved similarly throughout practice. However, when compared to the individuals of the HA group, those in the HRT group (1) had a lower risk of collisions, (2) exhibited higher performance consistency, and (3) revealed a higher level of mental workload while generally perceiving the robotic teammate as interfering with their performance. As the synthetic teammate changed the effector velocity in specific regions near the workspace boundaries, individuals may have been constrained to learn a piecewise visuomotor map. This piecewise map made the task more challenging, which increased mental workload and perception of the synthetic teammate as a burden. The examination of both motor performance and mental workload revealed a combination of both adaptive and maladaptive team dynamics. This work is a first step to examine the human cognitive-motor processes underlying motor practice in a context of team dynamics and contributes to inform human-robot applications.