Semi-Autonomous Laparoscopic Robotic Electro-surgery with a Novel 3D Endoscope

Image-Based 3D Reconstruction in Laparoscopy: A Review Focusing on the Quantitative Evaluation by Applying the Reconstruction Error

Image-based 3D reconstruction enables laparoscopic applications as image-guided navigation and (autonomous) robot-assisted interventions, which require a high accuracy. The review's purpose is to present the accuracy of different techniques to label the most promising. A systematic literature search with PubMed and google scholar from 2015 to 2023 was applied by following the framework of "Review articles: purpose, process, and structure". Articles were considered when presenting a quantitative evaluation (root mean squared error and mean absolute error) of the reconstruction error (Euclidean distance between real and reconstructed surface). The search provides 995 articles, which were reduced to 48 articles after applying exclusion criteria. From these, a reconstruction error data set could be generated for the techniques of stereo vision, Shape-from-Motion, Simultaneous Localization and Mapping, deep-learning, and structured light. The reconstruction error varies from below one millimeter to higher than ten millimeters-with deep-learning and Simultaneous Localization and Mapping delivering the best results under intraoperative conditions. The high variance emerges from different experimental conditions. In conclusion, submillimeter accuracy is challenging, but promising image-based 3D reconstruction techniques could be identified. For future research, we recommend computing the reconstruction error for comparison purposes and use ex/in vivo organs as reference objects for realistic experiments.

Deep point cloud landmark localization for fringe projection profilometry

Point clouds have been widely used due to their information being richer than images. Fringe projection profilometry (FPP) is one of the camera-based point cloud acquisition techniques that is being developed as a vision system for robotic surgery. For semi-autonomous robotic suturing, fluorescent fiducials were previously used on a target tissue as suture landmarks. This not only increases system complexity but also imposes safety concerns. To address these problems, we propose a numerical landmark localization algorithm based on a convolutional neural network (CNN) and a conditional random field (CRF). A CNN is applied to regress landmark heatmaps from the four-channel image data generated by the FPP. A CRF leveraging both local and global shape constraints is developed to better tune the landmark coordinates, reject extra landmarks, and recover missing landmarks. The robustness of the proposed method is demonstrated through ex vivo porcine intestine landmark localization experiments.

A Confidence-Based Supervised-Autonomous Control Strategy for Robotic Vaginal Cuff Closure

Autonomous robotic suturing has the potential to improve surgery outcomes by leveraging accuracy, repeatability, and consistency compared to manual operations. However, achieving full autonomy in complex surgical environments is not practical and human supervision is required to guarantee safety. In this paper, we develop a confidence-based supervised autonomous suturing method to perform robotic suturing tasks via both Smart Tissue Autonomous Robot (STAR) and surgeon collaboratively with the highest possible degree of autonomy. Via the proposed method, STAR performs autonomous suturing when highly confident and otherwise asks the operator for possible assistance in suture positioning adjustments. We evaluate the accuracy of our proposed control method via robotic suturing tests on synthetic vaginal cuff tissues and compare them to the results of vaginal cuff closures performed by an experienced surgeon. Our test results indicate that by using the proposed confidence-based method, STAR can predict the success of pure autonomous suture placement with an accuracy of 94.74%. Moreover, via an additional 25% human intervention, STAR can achieve a 98.1% suture placement accuracy compared to an 85.4% accuracy of completely autonomous robotic suturing. Finally, our experiment results indicate that STAR using the proposed method achieves 1.6 times better consistency in suture spacing and 1.8 times better consistency in suture bite sizes than the manual results.

Supervised Autonomous Electrosurgery via Biocompatible Near-Infrared Tissue Tracking Techniques

Autonomous robotic surgery systems aim to improve patient outcomes by leveraging the repeatability and consistency of automation and also reducing human induced errors. However, intraoperative autonomous soft tissue tracking and robot control still remains a challenge due to the lack of structure, and high deformability of such tissues. In this paper, we take advantage of biocompatible Near-Infrared (NIR) marking methods and develop a supervised autonomous 3D path planning, filtering, and control strategy for our Smart Tissue Autonomous Robot (STAR) to enable precise and consistent incisions on complex 3D soft tissues. Our experimental results on cadaver porcine tongue samples indicate that the proposed strategy reduces surface incision error and depth incision error by 40.03% and 51.5%, respectively, compared to a teleoperation strategy via da Vinci. Furthermore, compared to an autonomous path planning method with linear interpolation between the NIR markers, the proposed strategy reduces the incision depth error by 48.58% by taking advantage of 3D tissue surface information.

Semi-autonomous Robotic Anastomoses of Vaginal Cuffs Using Marker Enhanced 3D Imaging and Path Planning

Autonomous robotic anastomosis has the potential to improve surgical outcomes by performing more consistent suture spacing and bite size compared to manual anastomosis. However, due to soft tissue's irregular shape and unpredictable deformation, performing autonomous robotic anastomosis without continuous tissue detection and three-dimensional path planning strategies remains a challenging task. In this paper, we present a novel three-dimensional path planning algorithm for Smart Tissue Autonomous Robot (STAR) to enable semi-autonomous robotic anastomosis on deformable tissue. The algorithm incorporates (i) continuous detection of 3D near infrared (NIR) markers manually placed on deformable tissue before the procedure, (ii) generating a uniform and consistent suture placement plan using 3D path planning methods based on the locations of the NIR markers, and (iii) updating the remaining suture plan after each completed stitch using a non-rigid registration technique to account for tissue deformation during anastomosis. We evaluate the path planning algorithm for accuracy and consistency by comparing the anastomosis of synthetic vaginal cuff tissue completed by STAR and a surgeon. Our test results indicate that STAR using the proposed method achieves 2.6 times better consistency in suture spacing and 2.4 times better consistency in suture bite sizes than the manual anastomosis.

Autonomous Laparoscopic Robotic Suturing with a Novel Actuated Suturing Tool and 3D Endoscope

Compared to open surgical techniques, laparoscopic surgical methods aim to reduce the collateral tissue damage and hence decrease the patient recovery time. However, constraints imposed by the laparoscopic surgery, i.e. the operation of surgical tools in limited spaces, turn simple surgical tasks such as suturing into time-consuming and inconsistent tasks for surgeons. In this paper, we develop an autonomous laparoscopic robotic suturing system. More specific, we expand our smart tissue anastomosis robot (STAR) by developing i) a new 3D imaging endoscope, ii) a novel actuated laparoscopic suturing tool, and iii) a suture planning strategy for the autonomous suturing. We experimentally test the accuracy and consistency of our developed system and compare it to sutures performed manually by surgeons. Our test results on suture pads indicate that STAR can reach 2.9 times better consistency in suture spacing compared to manual method and also eliminate suture repositioning and adjustments. Moreover, the consistency of suture bite sizes obtained by STAR matches with those obtained by manual suturing.