Reinforcement Q-Learning Control With Reward Shaping Function for Swing Phase Control in a Semi-active Prosthetic Knee

Reinforcement learning based variable damping control of wearable robotic limbs for maintaining astronaut pose during extravehicular activity

As astronauts perform on-orbit servicing of extravehicular activity (EVA) without the help of the space station's robotic arms, it will be rather difficult and labor-consuming to maintain the appropriate position in case of impact. In order to solve this problem, we propose the development of a wearable robotic limb system for astronaut assistance and a variable damping control method for maintaining the astronaut's position. The requirements of the astronaut's impact-resisting ability during EVA were analyzed, including the capabilities of deviation resistance, fast return, oscillation resistance, and accurate return. To meet these needs, the system of the astronaut with robotic limbs was modeled and simplified. In combination with this simplified model and a reinforcement learning algorithm, a variable damping controller for the end of the robotic limb was obtained, which can regulate the dynamic performance of the robot end to resist oscillation after impact. A weightless simulation environment for the astronaut with robotic limbs was constructed. The simulation results demonstrate that the proposed method can meet the recommended requirements for maintaining an astronaut's position during EVA. No matter how the damping coefficient was set, the fixed damping control method failed to meet all four requirements at the same time. In comparison to the fixed damping control method, the variable damping controller proposed in this paper fully satisfied all the impact-resisting requirements by itself. It could prevent excessive deviation from the original position and was able to achieve a fast return to the starting point. The maximum deviation displacement was reduced by 39.3% and the recovery time was cut by 17.7%. Besides, it also had the ability to prevent reciprocating oscillation and return to the original position accurately.

Machine Learning-Based Surgical Planning for Neurosurgery: Artificial Intelligent Approaches to the Cranium

Objectives

Artificial intelligence (AI) applications in neurosurgery have an increasing momentum as well as the growing number of implementations in the medical literature. In recent years, AI research define a link between neuroscience and AI. It is a connection between knowing and understanding the brain and how to simulate the brain. The machine learning algorithms, as a subset of AI, are able to learn with experiences, perform big data analysis, and fulfill human-like tasks. Intracranial surgical approaches that have been defined, disciplined, and developed in the last century have become more effective with technological developments. We aimed to define individual-safe, intracranial approaches by introducing functional anatomical structures and pathological areas to artificial intelligence.

Methods

Preoperative MR images of patients with deeply located brain tumors were used for planning. Intracranial arteries, veins, and neural tracts are listed and numbered. Voxel values of these selected regions in cranial MR sequences were extracted and labeled. Tumor tissue was segmented as the target. Q-learning algorithm which is a model-free reinforcement learning algorithm was run on labeled voxel values (on optimal paths extracted from the new heuristic-based path planning algorithm), then the algorithm was assigned to list the cortico-tumoral pathways that aim to remove the maximum tumor tissue and in the meantime that functional anatomical tissues will be least affected.

Results

The most suitable cranial entry areas were found with the artificial intelligence algorithm. Cortico-tumoral pathways were revealed using Q-learning from these optimal points.

Conclusions

AI will make a significant contribution to the positive outcomes as its use in both preoperative surgical planning and intraoperative technique equipment assisted neurosurgery, its use increased.