Toward image-guided robotic surgery: determining intrinsic accuracy of the da Vinci robot

Stretchable glove for accurate and robust hand pose reconstruction based on comprehensive motion data

We propose a compact wearable glove capable of estimating both the finger bone lengths and the joint angles of the wearer with a simple stretch-based sensing mechanism. The soft sensing glove is designed to easily stretch and to be one-size-fits-all, both measuring the size of the hand and estimating the finger joint motions of the thumb, index, and middle fingers. The system was calibrated and evaluated using comprehensive hand motion data that reflect the extensive range of natural human hand motions and various anatomical structures. The data were collected with a custom motion-capture setup and transformed into the joint angles through our post-processing method. The glove system is capable of reconstructing arbitrary and even unconventional hand poses with accuracy and robustness, confirmed by evaluations on the estimation of bone lengths (mean error: 2.1 mm), joint angles (mean error: 4.16°), and fingertip positions (mean 3D error: 4.02 mm), and on overall hand pose reconstructions in various applications. The proposed glove allows us to take advantage of the dexterity of the human hand with potential applications, including but not limited to teleoperation of anthropomorphic robot hands or surgical robots, virtual and augmented reality, and collection of human motion data.

Design and Performance Verification of a Novel RCM Mechanism for a Minimally Invasive Surgical Robot

Minimally invasive surgical robots have the advantages of high positioning accuracy, good stability, and flexible operation, which can effectively improve the quality of surgery and reduce the difficulty for doctors to operate. However, in order to realize the translation of the existing RCM mechanism, it is often necessary to add a mobile unit, which is often bulky and occupies most space above the patient's body, thus causing interference to the operation. In this paper, a new type of planar RCM mechanism is proposed. Based on this mechanism, a 3-DOF robotic arm is designed, which can complete the required motion for surgery without adding a mobile unit. In this paper, the geometric model of the mechanism is first introduced, and the RCM point of the mechanism is proven during the motion process. Then, based on the establishment of the geometric model of the mechanism, a kinematics analysis of the mechanism is carried out. The singularity, the Jacobian matrix, and the kinematic performance of the mechanism are analyzed, and the working space of the mechanism is verified according to the kinematic equations. Finally, a prototype of the RCM mechanism was built, and its functionality was tested using a master-slave control strategy.

Patient-specific, touch-based registration during robotic, image-guided partial nephrectomy

Image-guidance during partial nephrectomy enables navigation within the operative field alongside a 3-dimensional roadmap of renal anatomy generated from patient-specific imaging. Once a process is performed by the human mind, the technology will allow standardization of the task for the benefit of all patients undergoing robot-assisted partial nephrectomy. Any surgeon will be able to visualize the kidney and key subsurface landmarks in real-time within a 3-dimensional simulation, with the goals of improving operative efficiency, decreasing surgical complications, and improving oncologic outcomes. For similar purposes, image-guidance has already been adopted as a standard of care in other surgical fields; we are now at the brink of this in urology. This review summarizes touch-based approaches to image-guidance during partial nephrectomy, as the technology begins to enter in vivo human evaluation. The processes of segmentation, localization, registration, and re-registration are all described with seamless integration into the da Vinci surgical system; this will facilitate clinical adoption sooner.

Design and Integration of Electrical Bio-impedance Sensing in Surgical Robotic Tools for Tissue Identification and Display

The integration of intra-operative sensors into surgical robots is a hot research topic since this can significantly facilitate complex surgical procedures by enhancing surgical awareness with real-time tissue information. However, currently available intra-operative sensing technologies are mainly based on image processing and force feedback, which normally require heavy computation or complicated hardware modifications of existing surgical tools. This paper presents the design and integration of electrical bio-impedance sensing into a commercial surgical robot tool, leading to the creation of a novel smart instrument that allows the identification of tissues by simply touching them. In addition, an advanced user interface is designed to provide guidance during the use of the system and to allow augmented-reality visualization of the tissue identification results. The proposed system imposes minor hardware modifications to an existing surgical tool, but adds the capability to provide a wealth of data about the tissue being manipulated. This has great potential to allow the surgeon (or an autonomous robotic system) to better understand the surgical environment. To evaluate the system, a series of ex-vivo experiments were conducted. The experimental results demonstrate that the proposed sensing system can successfully identify different tissue types with 100% classification accuracy. In addition, the user interface was shown to effectively and intuitively guide the user to measure the electrical impedance of the target tissue, presenting the identification results as augmented-reality markers for simple and immediate recognition.

Toward real-time remote processing of laparoscopic video

Laparoscopic surgery is a minimally invasive surgical technique where surgeons insert a small video camera into the patient's body to visualize internal organs and use small tools to perform surgical procedures. However, the benefit of small incisions has a drawback of limited visualization of subsurface tissues, which can lead to navigational challenges in the delivering of therapy. Image-guided surgery uses the images to map subsurface structures and can reduce the limitations of laparoscopic surgery. One particular laparoscopic camera system of interest is the vision system of the daVinci-Si robotic surgical system (Intuitive Surgical, Sunnyvale, California). The video streams generate approximately 360 MB of data per second, demonstrating a trend toward increased data sizes in medicine, primarily due to higher-resolution video cameras and imaging equipment. Processing this data on a bedside PC has become challenging and a high-performance computing (HPC) environment may not always be available at the point of care. To process this data on remote HPC clusters at the typical 30 frames per second (fps) rate, it is required that each 11.9 MB video frame be processed by a server and returned within 1/30th of a second. The ability to acquire, process, and visualize data in real time is essential for the performance of complex tasks as well as minimizing risk to the patient. As a result, utilizing high-speed networks to access computing clusters will lead to real-time medical image processing and improve surgical experiences by providing real-time augmented laparoscopic data. We have performed image processing algorithms on a high-definition head phantom video (1920 × 1080 pixels) and transferred the video using a message passing interface. The total transfer time is around 53 ms or 19 fps. We will optimize and parallelize these algorithms to reduce the total time to 30 ms.

Localizing ECoG electrodes on the cortical anatomy without post-implantation imaging

INTRODUCTION

Electrocorticographic (ECoG) grids are placed subdurally on the cortex in people undergoing cortical resection to delineate eloquent cortex. ECoG signals have high spatial and temporal resolution and thus can be valuable for neuroscientific research. The value of these data is highest when they can be related to the cortical anatomy. Existing methods that establish this relationship rely either on post-implantation imaging using computed tomography (CT), magnetic resonance imaging (MRI) or X-Rays, or on intra-operative photographs. For research purposes, it is desirable to localize ECoG electrodes on the brain anatomy even when post-operative imaging is not available or when intra-operative photographs do not readily identify anatomical landmarks.

METHODS

We developed a method to co-register ECoG electrodes to the underlying cortical anatomy using only a pre-operative MRI, a clinical neuronavigation device (such as BrainLab VectorVision), and fiducial markers. To validate our technique, we compared our results to data collected from six subjects who also had post-grid implantation imaging available. We compared the electrode coordinates obtained by our fiducial-based method to those obtained using existing methods, which are based on co-registering pre- and post-grid implantation images.

RESULTS

Our fiducial-based method agreed with the MRI-CT method to within an average of 8.24 mm (mean, median = 7.10 mm) across 6 subjects in 3 dimensions. It showed an average discrepancy of 2.7 mm when compared to the results of the intra-operative photograph method in a 2D coordinate system. As this method does not require post-operative imaging such as CTs, our technique should prove useful for research in intra-operative single-stage surgery scenarios. To demonstrate the use of our method, we applied our method during real-time mapping of eloquent cortex during a single-stage surgery. The results demonstrated that our method can be applied intra-operatively in the absence of post-operative imaging to acquire ECoG signals that can be valuable for neuroscientific investigations.

Augmented reality partial nephrectomy: examining the current status and future perspectives

A minimal access approach to partial nephrectomy has historically been under-utilized, but is now becoming more popular with the growth of robot-assisted laparoscopy. One of the criticisms of minimal access partial nephrectomy is the loss of haptic feedback. Augmented reality operating environments are forecast to play a major enabling role in the future of minimal access partial nephrectomy by integrating enhanced visual information to supplement this loss of haptic sensation. In this article, we systematically examine the current status of augmented reality in partial nephrectomy by identifying existing research challenges and exploring future agendas for this technology to achieve wider clinical translation.

Surgical Robots

Information technology and robotics have been integrated into interventional medicine for over 25 years. Their primary aim has always been to provide patient benefits through increased precision, safety, and minimal invasiveness. Nevertheless, robotic devices should allow for sophisticated treatment methods that are not possible by other means. Several hundreds of different surgical robot prototypes have been developed, while only a handful passed clearance procedures, and was released to the market. This is mostly due to the difficulties associated with medical device development and approval, especially in those cases when some form of manipulation and automation is involved. This chapter is intended to present major aspects of surgical robotic prototyping and current trends through the analysis of various international projects. It spans across the phases from system planning, to development, validation, and clearance.

Registration of 3D ultrasound through an air-tissue boundary

In this study we evaluated a new method for registering three-dimensional ultrasound (3DUS) data to external coordinate systems. First, 3DUS was registered to the stereo endoscope of a da Vinci Surgical System by placing a registration tool against an air-tissue boundary so that the 3DUS could image ultrasound fiducials while the stereo endoscope could image camera markers on the same tool. The common points were used to solve the registration between the 3DUS and camera coordinate systems. The target registration error (TRE) when imaging through a PVC tissue phantom ranged from 3.85 1.76 mm to 1.82 1.03 mm using one to four registration tool positions. TRE when imaging through an ex-vivo liver tissue sample ranged from 2.36 1.01 mm to 1.51 0.70 mm using one to four registration tool positions. Second, using a similar method, 3DUS was registered to the kinematic coordinate system of a da Vinci Surgical System by using the da Vinci surgical manipulators to identify common points on an air-tissue boundary. TRE when imaging through a PVC tissue phantom was 0.95 0.38 mm. This registration method is simpler and potentially more accurate than methods using commercial motion tracking systems. This method may be useful in the future in augmented reality systems for laparoscopic and robotic-assisted surgery.

Image-guided robotic surgery: update on research and potential applications in urologic surgery

PURPOSE OF REVIEW

New methods of imaging and image-guidance technology have the potential to provide surgeons with spatially accurate three-dimensional information about the location and anatomical relationships of critical subsurface structures and instrument position updated and displayed during the performance of surgery. Robotic platforms and technology in various forms continues to revolutionize surgery and will soon incorporate image guidance.

RECENT RESEARCH

Image-guided surgery (IGS) for abdominal and urologic interventions presents complex engineering and surgical challenges along with potential benefits to surgeons and patients. Key concepts such as registration, localization, accuracy, and targeting error are necessary for surgeons to understand and utilize the potential of IGS. Standard robotic surgeries, such as partial nephrectomy and radical prostatectomy may soon incorporate IGS.

SUMMARY

Research continues to explore the potential for combining image guidance and robotics to augment and improve a variety of surgical interventions.

Surgical Robots

Information technology and robotics have been integrated into interventional medicine for over 25 years. Their primary aim has always been to provide patient benefits through increased precision, safety, and minimal invasiveness. Nevertheless, robotic devices should allow for sophisticated treatment methods that are not possible by other means. Several hundreds of different surgical robot prototypes have been developed, while only a handful passed clearance procedures, and was released to the market. This is mostly due to the difficulties associated with medical device development and approval, especially in those cases when some form of manipulation and automation is involved. This chapter is intended to present major aspects of surgical robotic prototyping and current trends through the analysis of various international projects. It spans across the phases from system planning, to development, validation, and clearance.

Development of subject-specific geometric spine model through use of automated active contour segmentation and kinematic constraint-limited registration

This paper describes the development of a patient-specific spine model through use of active contour segmentation and registration of intraoperative imaging of porcine vertebra augmented with kinematic constraints. The geometric active contours are fully automated and lead to a discrete representation of the image segmentation results. After determining errors within the segmentations, application of reliability theory allows the selection of active contour parameters to obtain best-fit segmentations from a stack of 2D images. The segmented images are then used in conjunction with C-arm fluoroscope images to simulate the result of intraoperative patient-specific model registration including patient and/or structure motion between preoperative and intraoperative scans. The results are validated through comparison of the error within the patient-specific model generated through use of the C-arm images with a model acquired directly from MRI images of the spine after motion. The results are applicable to the development of a wide variety of patient-specific geometric and biomechanical models.

Design and in vitro evaluation of a real-time catheter localization system using time of flight measurements from seven 3.5 MHz single element ultrasound transducers towards abdominal aortic aneurysm procedures

Interventional surgical instrument localization is a crucial component of minimally invasive surgery. Image guided surgery researchers are investigating devices broadly categorized as surgical localizers to provide real-time information on the instrument's 3D location and orientation only. This paper describes the implementation and in vitro evaluation of a prototype real-time nonimaging ultrasound-based catheter localizer system towards use in abdominal aortic aneurysm procedures. The catheter-tip is equipped with a single element ultrasound transducer which is tracked with an array of seven external single element transducers. The performance of the system was evaluated in a water tank and additionally in the presence of pork belly tissue and also a nitinol-dacron stent graft. The mean root mean square errors were respectively 1.94±0.06, 2.54±0.31 and 3.33±0.06 mm. In addition, this paper illustrates errors induced by transducer aperture size and suggests a method for aperture error compensation. Aperture compensation applied to the same experimental data yielded mean root mean square errors of 1.05±0.07, 2.42±0.33 and 3.23±0.07mm respectively for water; water and pork; and water, pork and stent experiments. Lastly, this paper presents a video showing free-hand movement of the catheter within the water tank with data capture at 25 frames per second.

Towards image guided robotic surgery: multi-arm tracking through hybrid localization

OBJECTIVE

Use of the robotic assisted surgery has been increasing in recent years, due both the continuous increase in the number of applications and the clinical benefits that surgical robots can provide. Currently robotic assisted surgery relies on endoscopic video for navigation, providing only surface visualization, thus limiting subsurface vision. To be able to visualize and identify subsurface information, techniques in image-guidance can be used. As part of designing an image guidance system, all arms of the robot need to be co-localized in a common coordinate system.

METHODS

In order to track multiple arms in a common coordinate space, intrinsic and extrinsic tracking methods can be used. First, the intrinsic tracking of the daVinci, specifically of the setup joints is analyzed. Because of the inadequacy of the setup joints for co-localization a hybrid tracking method is designed and implemented to mitigate the inaccuracy of the setup joints. Different both optical and magnetic tracking methods are examined for setup joint localization.

RESULTS

The hybrid localization method improved the localization accuracy of the setup joints. The inter-arm accuracy in hybrid localization was improved to 3.02 mm. This inter-arm error value was shown to be further reduced when the arms are co-registered, thus reducing common error.

Toward image guided robotic surgery: system validation

PURPOSE

Navigation for current robotic assisted surgical techniques is primarily accomplished through a stereo pair of laparoscopic camera images. These images provide standard optical visualization of the surface but provide no subsurface information. Image guidance methods allow the visualization of subsurface information to determine the current position in relationship to that of tracked tools.

MATERIALS AND METHODS

A robotic image guided surgical system was designed and implemented based on our previous laboratory studies. A series of experiments using tissue mimicking phantoms with injected target lesions was performed. The surgeon was asked to resect "tumor" tissue with and without the augmentation of image guidance using the da Vinci robotic surgical system. Resections were performed and compared to an ideal resection based on the radius of the tumor measured from preoperative computerized tomography. A quantity called the resection ratio, that is the ratio of resected tissue compared to the ideal resection, was calculated for each of 13 trials and compared.

RESULTS

The mean +/- SD resection ratio of procedures augmented with image guidance was smaller than that of procedures without image guidance (3.26 +/- 1.38 vs 9.01 +/- 1.81, p <0.01). Additionally, procedures using image guidance were shorter (average 8 vs 13 minutes).

CONCLUSIONS

It was demonstrated that there is a benefit from the augmentation of laparoscopic video with updated preoperative images. Incorporating our image guided system into the da Vinci robotic system improved overall tissue resection, as measured by our metric. Adding image guidance to the da Vinci robotic surgery system may result in the potential for improvements such as the decreased removal of benign tissue while maintaining an appropriate surgical margin.