A Continuum Robot and Control Interface for Surgical Assist in Fetoscopic Interventions

Improving the kinematic accuracy of a collaborative continuum robot by using flexure-hinges

Within various unstructured industrial environments, there is often the requirement to conduct remote engineering tasks, such as sampling the structure for analysis prior to decommissioning. Most existing tools are simply not dexterous enough to fulfil this task, and thus new technology is required. We describe here a simple, lightweight, and water-resistant collaborative dual-arm continuum robot system which can aid in this task. To improve the kinematic accuracy of the system, a class of flexible hinges have been combined with a conventional continuum robot configuration. The thickness and width of said flexible hinges can be adjusted to adapt to various tasks. Kinematic and stiffness models have further been developed, incorporating the influence of these flexible hinges. A set of experiments have been conducted to validate the proposed model and demonstrate the advantages of the platform. It was found that the kinematic accuracy of the continuum robot can be improved by a factor of around 10 with the aid of said hinges.

Robot assisted Fetoscopic Laser Coagulation: Improvements in navigation, re-location and coagulation

Fetoscopic Laser Coagulation (FLC) for Twin to Twin Transfusion Syndrome is a challenging intervention due to the working conditions: low quality images acquired from a 3 mm fetoscope inside a turbid liquid environment, local view of the placental surface, unstable surgical field and delicate tissue layers. FLC is based on locating, coagulating and reviewing anastomoses over the placenta's surface. The procedure demands the surgeons to generate a mental map of the placenta with the distribution of the anastomoses, maintaining, at the same time, precision in coagulation and protecting the placenta and amniotic sac from potential damages. This paper describes a teleoperated platform with a cognitive-based control that provides assistance to improve patient safety and surgery performance during fetoscope navigation, target re-location and coagulation processes. A comparative study between manual and teleoperated operation, executed in dry laboratory conditions, analyzes basic fetoscopic skills: fetoscope navigation and laser coagulation. Two exercises are proposed: first, fetoscope guidance and precise coagulation. Second, a resolved placenta (all anastomoses are indicated) to evaluate navigation, re-location and coagulation. The results are analyzed in terms of economy of movement, execution time, coagulation accuracy, amount of coagulated placental surface and risk of placenta puncture. In addition, new metrics, based on navigation and coagulation maps evaluate robotic performance. The results validate the developed platform, showing noticeable improvements in all the metrics.

Computer-assisted fetal laser surgery in the treatment of twin-to-twin transfusion syndrome recent trends and prospects

Fetal laser surgery has emerged as the preferred treatment of twin-to-twin transfusion syndrome (TTTS). However, the limited field of view of the fetoscope and the complexity of the procedure make the treatment challenging. Therefore, preoperative planning and intraoperative guidance solutions have been proposed to cope with these challenges. This review uncovers the literature on computer-assisted software solutions focused on TTTS. These solutions are classified by the pre- or intraoperative phase of the procedure and further categorized by discussed hardware and software approaches. In addition, it evaluates the current maturity of technologies by the technology readiness level and enumerates the necessary aspects to bring these new technologies to the clinical practice. This article is protected by copyright. All rights reserved.

Application of Intelligent Intravenous Drug Dispensing Robot in Clinical Nursing

In order to explore the application of intelligent intravenous drug dispensing robot in clinical nursing, the efficiency, residual amount, needle pushing, and pulling times, the incidence of accidental hand stab injury and the accuracy of drug dispensing were compared and observed between intelligent intravenous drug dispensing robot and manual dispensing. The study found that the sterile powder residual amount of injection is pointed out in the standard, the drug label is not equipped with ≤50 mg, and the residual limit is ≤ 15%; the lamination is 50–150 mg, the residual amount is limited to ≤10%; the marking is 150–500 mg, the residual amount is ≤ 7%; the marking is 500 mg, the residual amount is limited to ≤5%; 6 months of artificial deployment accidental stab ergonomics occur; without this phenomenon when using the drug-like robot, 6 months internal use of the pharmaceutical robot, 4 times of error occurrence, and errors occurred 60 times. During the clinical intravenous infusion dispensing process, the use of intelligent vein is used to configure the robot, improve the efficiency of the drug, reduce the incidence of drug residual, reduce the incidence of dispensing, increase the accuracy and safety of venous liquidation, and decrease to some extent professional injury caused by drug formulation for liquid nurses.

Fluoroscopic Navigation for a Surgical Robotic System Including a Continuum Manipulator

We present an image-based navigation solution for a surgical robotic system with a Continuum Manipulator (CM). Our navigation system uses only fluoroscopic images from a mobile C-arm to estimate the CM shape and pose with respect to the bone anatomy. The CM pose and shape estimation is achieved using image intensity-based 2D/3D registration. A learning-based framework is used to automatically detect the CM in X-ray images, identifying landmark features that are used to initialize and regularize image registration. We also propose a modified hand-eye calibration method that numerically optimizes the hand-eye matrix during image registration. The proposed navigation system for CM positioning was tested in simulation and cadaveric studies. In simulation, the proposed registration achieved a mean error of 1.10±0.72 mm between the CM tip and a target entry point on the femur. In cadaveric experiments, the mean CM tip position error was 2.86±0.80 mm after registration and repositioning of the CM. The results suggest that our proposed fluoroscopic navigation is feasible to guide the CM in orthopedic applications.

Miniature parallel robot with submillimeter positioning accuracy for minimally invasive laser osteotomy

To overcome the physical limitations of mechanical bone cutting in minimally invasive surgery, we are developing a miniature parallel robot that enables positioning of a pulsed laser with an accuracy below 0.25 mm and minimizes the required manipulation space above the target tissue. This paper presents the design, control, device characteristics, functional testing, and performance evaluation of the robot. The performance of the robot was evaluated within the scope of a path-following experiment. The required accuracy for continuous cuts was achieved and reached 0.176 mm on the test bench.

Design and Experimental Study of Space Continuous Robots Applied to Space Non-Cooperative Target Capture

Space capture actuators face problems such as insufficient flexibility and electrical components that are vulnerable to extreme space environments. To address these problems, a centralized-driven flexible continuous robot based on a multiple scissor mechanism units is proposed in this study. The continuous robot body is composed of two scissor mechanism units coupled in series, and the base container’s three motors to drive the robot. The two scissor mechanism units ensure a wide range of flexible operations and the light weight of the robot. The centralized drive with three motors not only reduces the number of driving sources, but also ensures temperature control and protection of electrical components in the space environment. The kinematics and dynamics of the robot are analyzed, and the workspace and deformation performance of the robot are verified through experiments. Compared with other continuous robots, the proposed continuous robot retains the characteristics of continuous robots in a wide range of flexible operations. At the same time, the configuration is light and a small number of driving sources are used, which is suitable for extreme temperatures, vacuum, radiation, and strict resource-constrained environments in space.

A Surgical Robotic System for Treatment of Pelvic Osteolysis Using an FBG-Equipped Continuum Manipulator and Flexible Instruments

This paper presents the development and experimental evaluation of a redundant robotic system for the less-invasive treatment of osteolysis (bone degradation) behind the acetabular implant during total hip replacement revision surgery. The system comprises a rigid-link positioning robot and a Continuum Dexterous Manipulator (CDM) equipped with highly flexible debriding tools and a Fiber Bragg Grating (FBG)-based sensor. The robot and the continuum manipulator are controlled concurrently via an optimization-based framework using the Tip Position Estimation (TPE) from the FBG sensor as feedback. Performance of the system is evaluated on a setup that consists of an acetabular cup and saw-bone phantom simulating the bone behind the cup. Experiments consist of performing the surgical procedure on the simulated phantom setup. CDM TPE using FBGs, target location placement, cutting performance, and the concurrent control algorithm capability in achieving the desired tasks are evaluated. Mean and standard deviation of the CDM TPE from the FBG sensor and the robotic system are 0.50 mm, and 0.18 mm, respectively. Using the developed surgical system, accurate positioning and successful cutting of desired straight-line and curvilinear paths on saw-bone phantoms behind the cup with different densities are demonstrated. Compared to the conventional rigid tools, the workspace reach behind the acetabular cup is 2.47 times greater when using the developed robotic system.

Recent Advances in Design and Actuation of Continuum Robots for Medical Applications

Traditional rigid robot application in the medical field is limited due to the limited degrees of freedom caused by their material and structure. Inspired by trunk, tentacles, and snakes, continuum robot (CR) could traverse confined space, manipulate objects in complex environment, and conform to curvilinear paths in space. The continuum robot has broad prospect in surgery due to its high dexterity, which can reach circuitous areas of the body and perform precision surgery. Recently, many efforts have been done by researchers to improve the design and actuation methods of continuum robots. Several continuum robots have been applied in clinic surgical interventions and demonstrated superiorities to conventional rigid-link robots. In this paper, we provide an overview of the current development of continuum robots, including the design principles, actuation methods, application prospect, limitations, and challenge. And we also provide perspective for the future development. We hope that with the development of material science, Engineering ethics, and manufacture technology, new methods can be applied to manufacture continuum robots for specific surgical procedures.

An Accelerated Finite-Time Convergent Neural Network for Visual Servoing of a Flexible Surgical Endoscope With Physical and RCM Constraints

This article designs and analyzes a recurrent neural network (RNN) for the visual servoing of a flexible surgical endoscope. The flexible surgical endoscope is based on a commercially available UR5 robot with a flexible endoscope attached as an end-effector. Most of the existing visual servo control frameworks of the robotic endoscopes or robot arms have not considered either the physical limits of the robot or the remote center of motion (RCM) constraints (i.e., the fulcrum effect). To tackle this issue, this article first conducts the kinematic modeling of the flexible robotic endoscope to achieve automation by visual servo control. The kinematic modeling results in a quadratic programming (QP) framework with physical limits and RCM constraints involved, making the UR5 robot applicable to surgical field. To solve the QP problem and accomplish the visual task, an RNN activated by a sign-bi-power activation function (AF) is proposed. The motivation of using the sign-bi-power AF is to enable the RNN to exhibit an accelerated finite-time convergence, which is more preferred in time-critical applications. Theoretically, the finite-time convergence of the RNN is rigorously proved using the Lyapunov theory. Compared with the previous AFs applied to the RNN, theoretical analysis shows that the RNN activated by the sign-bi-power AF delivers an accelerated convergence speed. Comparative validations are performed, showing that the proposed finite-time convergent neural network is effective to achieve visual servoing of the flexible endoscope with physical limits and RCM constraints handled simultaneously.

Deep learning-based fetoscopic mosaicking for field-of-view expansion

PURPOSE

Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure.

METHODS

We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos.

RESULTS

We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods.

CONCLUSION

The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.

Investigating exploration for deep reinforcement learning of concentric tube robot control

Purpose

Concentric tube robots are composed of multiple concentric, pre-curved, super-elastic, telescopic tubes that are compliant and have a small diameter suitable for interventions that must be minimally invasive like fetal surgery. Combinations of rotation and extension of the tubes can alter the robot’s shape but the inverse kinematics are complex to model due to the challenge of incorporating friction and other tube interactions or manufacturing imperfections. We propose a model-free reinforcement learning approach to form the inverse kinematics solution and directly obtain a control policy.

Method

Three exploration strategies are shown for deep deterministic policy gradient with hindsight experience replay for concentric tube robots in simulation environments. The aim is to overcome the joint to Cartesian sampling bias and be scalable with the number of robotic tubes. To compare strategies, evaluation of the trained policy network to selected Cartesian goals and associated errors are analyzed. The learned control policy is demonstrated with trajectory following tasks.

Results

Separation of extension and rotation joints for Gaussian exploration is required to overcome Cartesian sampling bias. Parameter noise and Ornstein–Uhlenbeck were found to be optimal strategies with less than 1 mm error in all simulation environments. Various trajectories can be followed with the optimal exploration strategy learned policy at high joint extension values. Our inverse kinematics solver in evaluation has 0.44 mm extension and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$0.3^{\circ }$$\end{document}0.3∘ rotation error.

Conclusion

We demonstrate the feasibility of effective model-free control for concentric tube robots. Directly using the control policy, arbitrary trajectories can be followed and this is an important step towards overcoming the challenge of concentric tube robot control for clinical use in minimally invasive interventions.

Combining Differential Kinematics and Optical Flow for Automatic Labeling of Continuum Robots in Minimally Invasive Surgery

The segmentation of continuum robots in medical images can be of interest for analyzing surgical procedures or for controlling them. However, the automatic segmentation of continuous and flexible shapes is not an easy task. On one hand conventional approaches are not adapted to the specificities of these instruments, such as imprecise kinematic models, and on the other hand techniques based on deep-learning showed interesting capabilities but need many manually labeled images. In this article we propose a novel approach for segmenting continuum robots on endoscopic images, which requires no prior on the instrument visual appearance and no manual annotation of images. The method relies on the use of the combination of kinematic models and differential kinematic models of the robot and the analysis of optical flow in the images. A cost function aggregating information from the acquired image, from optical flow and from robot encoders is optimized using particle swarm optimization and provides estimated parameters of the pose of the continuum instrument and a mask defining the instrument in the image. In addition a temporal consistency is assessed in order to improve stochastic optimization and reject outliers. The proposed approach has been tested for the robotic instruments of a flexible endoscopy platform both for benchtop acquisitions and an in vivo video. The results show the ability of the technique to correctly segment the instruments without a prior, and in challenging conditions. The obtained segmentation can be used for several applications, for instance for providing automatic labels for machine learning techniques.

Haptic Guidance Based on All-Optical Ultrasound Distance Sensing for Safer Minimally Invasive Fetal Surgery

By intervening during the early stage of gestation, fetal surgeons aim to correct or minimize the effects of congenital disorders. As compared to postnatal treatment of these disorders, such early interventions can often actually save the life of the fetus and also improve the quality of life of the newborn. However, fetal surgery is considered one of the most challenging disciplines within Minimally Invasive Surgery (MIS), owing to factors such as the fragility of the anatomic features, poor visibility, limited manoeuvrability, and extreme requirements in terms of instrument handling with precise positioning. This work is centered on a fetal laser surgery procedure treating placental disorders. It proposes the use of haptic guidance to enhance the overall safety of this procedure and to simplify instrument handling. A method is described that provides effective guidance by installing a forbidden region virtual fixture over the placenta, thereby safeguarding adequate clearance between the instrument tip and the placenta. With a novel application of all-optical ultrasound distance sensing in which transmission and reception are performed with fibre optics, this method can be used with a sole reliance on intraoperatively acquired data. The added value of the guidance approach, in terms of safety and performance, is demonstrated in a series of experiments with a robotic platform.