1
|
Wallace BA, Si J. Continuous-Time Reinforcement Learning Control: A Review of Theoretical Results, Insights on Performance, and Needs for New Designs. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:10199-10219. [PMID: 37027747 DOI: 10.1109/tnnls.2023.3245980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
This exposition discusses continuous-time reinforcement learning (CT-RL) for the control of affine nonlinear systems. We review four seminal methods that are the centerpieces of the most recent results on CT-RL control. We survey the theoretical results of the four methods, highlighting their fundamental importance and successes by including discussions on problem formulation, key assumptions, algorithm procedures, and theoretical guarantees. Subsequently, we evaluate the performance of the control designs to provide analyses and insights on the feasibility of these design methods for applications from a control designer's point of view. Through systematic evaluations, we point out when theory diverges from practical controller synthesis. We, furthermore, introduce a new quantitative analytical framework to diagnose the observed discrepancies. Based on the analyses and the insights gained through quantitative evaluations, we point out potential future research directions to unleash the potential of CT-RL control algorithms in addressing the identified challenges.
Collapse
|
2
|
Diaz MA, Vos M, Dillen A, Tassignon B, Flynn L, Geeroms J, Meeusen R, Verstraten T, Babic J, Beckerle P, De Pauw K. Human-in-the-Loop Optimization of Wearable Robotic Devices to Improve Human-Robot Interaction: A Systematic Review. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:7483-7496. [PMID: 37015459 DOI: 10.1109/tcyb.2022.3224895] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
This article presents a systematic review on wearable robotic devices that use human-in-the-loop optimization (HILO) strategies to improve human-robot interaction. A total of 46 HILO studies were identified and divided into upper and lower limb robotic devices. The main aspects from HILO were identified, reviewed, and classified in four areas: 1) human-machine systems; 2) optimization methods; 3) control strategies; and 4) experimental protocols. A variety of objective functions (physiological, biomechanical, and subjective), optimization strategies, and optimized control parameters configurations used in different control strategies are presented and analyzed. An overview of experimental protocols is provided, including metrics, tasks, and conditions tested. Moreover, the relevance given to training or adaptation periods was explored. We outline an HILO framework that includes current wearable robots, optimization strategies, objective functions, control strategies, and experimental protocols. We conclude by highlighting current research gaps and defining future directions to improve the development of advanced HILO strategies in upper and lower limb wearable robots.
Collapse
|
3
|
Jiang Y, Wang C, Zhou S. Artificial intelligence-based risk stratification, accurate diagnosis and treatment prediction in gynecologic oncology. Semin Cancer Biol 2023; 96:82-99. [PMID: 37783319 DOI: 10.1016/j.semcancer.2023.09.005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2022] [Revised: 08/27/2023] [Accepted: 09/25/2023] [Indexed: 10/04/2023]
Abstract
As data-driven science, artificial intelligence (AI) has paved a promising path toward an evolving health system teeming with thrilling opportunities for precision oncology. Notwithstanding the tremendous success of oncological AI in such fields as lung carcinoma, breast tumor and brain malignancy, less attention has been devoted to investigating the influence of AI on gynecologic oncology. Hereby, this review sheds light on the ever-increasing contribution of state-of-the-art AI techniques to the refined risk stratification and whole-course management of patients with gynecologic tumors, in particular, cervical, ovarian and endometrial cancer, centering on information and features extracted from clinical data (electronic health records), cancer imaging including radiological imaging, colposcopic images, cytological and histopathological digital images, and molecular profiling (genomics, transcriptomics, metabolomics and so forth). However, there are still noteworthy challenges beyond performance validation. Thus, this work further describes the limitations and challenges faced in the real-word implementation of AI models, as well as potential solutions to address these issues.
Collapse
Affiliation(s)
- Yuting Jiang
- Department of Obstetrics and Gynecology, Key Laboratory of Birth Defects and Related Diseases of Women and Children of MOE and State Key Laboratory of Biotherapy, West China Second Hospital, Sichuan University and Collaborative Innovation Center, Chengdu, Sichuan 610041, China; Department of Pulmonary and Critical Care Medicine, State Key Laboratory of Respiratory Health and Multimorbidity, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Chengdi Wang
- Department of Obstetrics and Gynecology, Key Laboratory of Birth Defects and Related Diseases of Women and Children of MOE and State Key Laboratory of Biotherapy, West China Second Hospital, Sichuan University and Collaborative Innovation Center, Chengdu, Sichuan 610041, China; Department of Pulmonary and Critical Care Medicine, State Key Laboratory of Respiratory Health and Multimorbidity, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China
| | - Shengtao Zhou
- Department of Obstetrics and Gynecology, Key Laboratory of Birth Defects and Related Diseases of Women and Children of MOE and State Key Laboratory of Biotherapy, West China Second Hospital, Sichuan University and Collaborative Innovation Center, Chengdu, Sichuan 610041, China; Department of Pulmonary and Critical Care Medicine, State Key Laboratory of Respiratory Health and Multimorbidity, Frontiers Science Center for Disease-related Molecular Network, West China Hospital, Sichuan University, Chengdu, Sichuan 610041, China.
| |
Collapse
|
4
|
Kim M, Hargrove LJ. Generating synthetic gait patterns based on benchmark datasets for controlling prosthetic legs. J Neuroeng Rehabil 2023; 20:115. [PMID: 37667313 PMCID: PMC10476332 DOI: 10.1186/s12984-023-01232-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 08/08/2023] [Indexed: 09/06/2023] Open
Abstract
BACKGROUND Prosthetic legs help individuals with an amputation regain locomotion. Recently, deep neural network (DNN)-based control methods, which take advantage of the end-to-end learning capability of the network, have been proposed. One prominent challenge for these learning-based approaches is obtaining data for the training, particularly for the training of a mid-level controller. In this study, we propose a method for generating synthetic gait patterns (vertical load and lower limb joint angles) using a generative adversarial network (GAN). This approach enables a mid-level controller to execute ambulation modes that are not included in the training datasets. METHODS The conditional GAN is trained on benchmark datasets that contain the gait data of individuals without amputation; synthetic gait patterns are generated from the user input. Further, a DNN-based controller for the generation of impedance parameters is trained using the synthetic gait pattern and the corresponding synthetic stiffness and damping coefficients. RESULTS The trained GAN generated synthetic gait patterns with a coefficient of determination of 0.97 and a structural similarity index of 0.94 relative to benchmark data that were not included in the training datasets. We trained a DNN-based controller using the GAN-generated synthetic gait patterns for level-ground walking, standing-to-sitting motion, and sitting-to-standing motion. Four individuals without amputation participated in bypass testing and demonstrated the ambulation modes. The model successfully generated control parameters for the knee and ankle based on thigh angle and vertical load. CONCLUSIONS This study demonstrates that synthetic gait patterns can be used to train DNN models for impedance control. We believe a conditional GAN trained on benchmark datasets can provide reliable gait data for ambulation modes that are not included in its training datasets. Thus, designing gait data using a conditional GAN could facilitate the efficient and effective training of controllers for prosthetic legs.
Collapse
Affiliation(s)
- Minjae Kim
- Department of Physical Medicine and Rehabilitation, Northwestern University, IL, Chicago, USA.
- Regenstein Center for Bionic Medicine, Shirley Ryan AbilityLab, IL, Chicago, USA.
| | - Levi J Hargrove
- Department of Physical Medicine and Rehabilitation, Northwestern University, IL, Chicago, USA
- Regenstein Center for Bionic Medicine, Shirley Ryan AbilityLab, IL, Chicago, USA
| |
Collapse
|
5
|
Luo S, Androwis G, Adamovich S, Nunez E, Su H, Zhou X. Robust walking control of a lower limb rehabilitation exoskeleton coupled with a musculoskeletal model via deep reinforcement learning. J Neuroeng Rehabil 2023; 20:34. [PMID: 36935514 PMCID: PMC10024861 DOI: 10.1186/s12984-023-01147-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2021] [Accepted: 02/14/2023] [Indexed: 03/21/2023] Open
Abstract
BACKGROUND Few studies have systematically investigated robust controllers for lower limb rehabilitation exoskeletons (LLREs) that can safely and effectively assist users with a variety of neuromuscular disorders to walk with full autonomy. One of the key challenges for developing such a robust controller is to handle different degrees of uncertain human-exoskeleton interaction forces from the patients. Consequently, conventional walking controllers either are patient-condition specific or involve tuning of many control parameters, which could behave unreliably and even fail to maintain balance. METHODS We present a novel, deep neural network, reinforcement learning-based robust controller for a LLRE based on a decoupled offline human-exoskeleton simulation training with three independent networks, which aims to provide reliable walking assistance against various and uncertain human-exoskeleton interaction forces. The exoskeleton controller is driven by a neural network control policy that acts on a stream of the LLRE's proprioceptive signals, including joint kinematic states, and subsequently predicts real-time position control targets for the actuated joints. To handle uncertain human interaction forces, the control policy is trained intentionally with an integrated human musculoskeletal model and realistic human-exoskeleton interaction forces. Two other neural networks are connected with the control policy network to predict the interaction forces and muscle coordination. To further increase the robustness of the control policy to different human conditions, we employ domain randomization during training that includes not only randomization of exoskeleton dynamics properties but, more importantly, randomization of human muscle strength to simulate the variability of the patient's disability. Through this decoupled deep reinforcement learning framework, the trained controller of LLREs is able to provide reliable walking assistance to patients with different degrees of neuromuscular disorders without any control parameter tuning. RESULTS AND CONCLUSION A universal, RL-based walking controller is trained and virtually tested on a LLRE system to verify its effectiveness and robustness in assisting users with different disabilities such as passive muscles (quadriplegic), muscle weakness, or hemiplegic conditions without any control parameter tuning. Analysis of the RMSE for joint tracking, CoP-based stability, and gait symmetry shows the effectiveness of the controller. An ablation study also demonstrates the strong robustness of the control policy under large exoskeleton dynamic property ranges and various human-exoskeleton interaction forces. The decoupled network structure allows us to isolate the LLRE control policy network for testing and sim-to-real transfer since it uses only proprioception information of the LLRE (joint sensory state) as the input. Furthermore, the controller is shown to be able to handle different patient conditions without the need for patient-specific control parameter tuning.
Collapse
Affiliation(s)
- Shuzhen Luo
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA
- Lab of Biomechatronics and Intelligent Robotics, Department of Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, 27695, NC, USA
| | - Ghaith Androwis
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA
- Kessler Foundation, West Orange, 07052, NJ, USA
| | - Sergei Adamovich
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA
| | - Erick Nunez
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA
| | - Hao Su
- Lab of Biomechatronics and Intelligent Robotics, Department of Mechanical and Aerospace Engineering, North Carolina State University, Raleigh, 27695, NC, USA
- Joint NCSU/UNC Department of Biomedical Engineering, University of North Carolina at Chapel Hill, Chapel Hill, 27599, NC, USA
| | - Xianlian Zhou
- Department of Biomedical Engineering, New Jersey Institute of Technology, Newark, 07102, NJ, USA.
| |
Collapse
|
6
|
Yang R, Zheng J, Song R. Continuous mode adaptation for cable-driven rehabilitation robot using reinforcement learning. Front Neurorobot 2022; 16:1068706. [PMID: 36620486 PMCID: PMC9813438 DOI: 10.3389/fnbot.2022.1068706] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Accepted: 11/28/2022] [Indexed: 12/24/2022] Open
Abstract
Continuous mode adaptation is very important and useful to satisfy the different user rehabilitation needs and improve human-robot interaction (HRI) performance for rehabilitation robots. Hence, we propose a reinforcement-learning-based optimal admittance control (RLOAC) strategy for a cable-driven rehabilitation robot (CDRR), which can realize continuous mode adaptation between passive and active working mode. To obviate the requirement of the knowledge of human and robot dynamics model, a reinforcement learning algorithm was employed to obtain the optimal admittance parameters by minimizing a cost function composed of trajectory error and human voluntary force. Secondly, the contribution weights of the cost function were modulated according to the human voluntary force, which enabled the CDRR to achieve continuous mode adaptation between passive and active working mode. Finally, simulation and experiments were conducted with 10 subjects to investigate the feasibility and effectiveness of the RLOAC strategy. The experimental results indicated that the desired performances could be obtained; further, the tracking error and energy per unit distance of the RLOAC strategy were notably lower than those of the traditional admittance control method. The RLOAC strategy is effective in improving the tracking accuracy and robot compliance. Based on its performance, we believe that the proposed RLOAC strategy has potential for use in rehabilitation robots.
Collapse
Affiliation(s)
- Renyu Yang
- Key Laboratory of Sensing Technology and Biomedical Instrument of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, China,School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
| | - Jianlin Zheng
- Key Laboratory of Sensing Technology and Biomedical Instrument of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, China,School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China
| | - Rong Song
- Key Laboratory of Sensing Technology and Biomedical Instrument of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, China,School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, Shenzhen, China,*Correspondence: Rong Song,
| |
Collapse
|
7
|
Wu R, Li M, Yao Z, Liu W, Si J, Huang H. Reinforcement Learning Impedance Control of a Robotic Prosthesis to Coordinate With Human Intact Knee Motion. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3179420] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Ruofan Wu
- School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA
| | - Minhan Li
- UNC/NCSU Department of Biomedical Engineering, NC State University, Raleigh, NC, USA
| | - Zhikai Yao
- School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA
| | - Wentao Liu
- UNC/NCSU Department of Biomedical Engineering, NC State University, Raleigh, NC, USA
| | - Jennie Si
- School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, USA
| | - He Huang
- UNC/NCSU Department of Biomedical Engineering, NC State University, Raleigh, NC, USA
| |
Collapse
|
8
|
Wu J, Huang Z, Huang W, Lv C. Prioritized Experience-Based Reinforcement Learning With Human Guidance for Autonomous Driving. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2022; PP:855-869. [PMID: 35687630 DOI: 10.1109/tnnls.2022.3177685] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Reinforcement learning (RL) requires skillful definition and remarkable computational efforts to solve optimization and control problems, which could impair its prospect. Introducing human guidance into RL is a promising way to improve learning performance. In this article, a comprehensive human guidance-based RL framework is established. A novel prioritized experience replay mechanism that adapts to human guidance in the RL process is proposed to boost the efficiency and performance of the RL algorithm. To relieve the heavy workload on human participants, a behavior model is established based on an incremental online learning method to mimic human actions. We design two challenging autonomous driving tasks for evaluating the proposed algorithm. Experiments are conducted to access the training and testing performance and learning mechanism of the proposed algorithm. Comparative results against the state-of-the-art methods suggest the advantages of our algorithm in terms of learning efficiency, performance, and robustness.
Collapse
|
9
|
Liu W, Wu R, Si J, Huang H. A New Robotic Knee Impedance Control Parameter Optimization Method Facilitated by Inverse Reinforcement Learning. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3194326] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Wentao Liu
- UNC/NCSU Department of Biomedical Engineering, North Carolina State University, Raleigh, NC, USA
| | - Ruofan Wu
- School of Electrical, Computer, and Energy Engineering, Arizona State University, Tempe, AZ, USA
| | - Jennie Si
- School of Electrical, Computer, and Energy Engineering, Arizona State University, Tempe, AZ, USA
| | - He Huang
- UNC/NCSU Department of Biomedical Engineering, North Carolina State University, Raleigh, NC, USA
| |
Collapse
|