Autonomous 3-d positioning of surgical instruments in robotized laparoscopic surgery using visual servoing

HydraMarker: Efficient, Flexible, and Multifold Marker Field Generation

An n-order marker field is a special binary matrix whose n×n subregions are all distinct from each other in four orientations. It is commonly used to guide the composing process of position-sensing markers, which can be detected and identified in a camera image with very limited scope or severe visibility problems. Despite the advantages, position-sensing markers are rare and overlooked because generating marker fields is difficult. In this article, we broaden the definition of marker field, making it more powerful and flexible. Then, we propose bWFC (binary wave function collapse) and its high-speed version, fast-bWFC, to solve the generation problem. The methods are packaged into an open-sourced toolkit named HydraMarker, with which, users not only can generate marker fields on laptops within a short period of time, but also can highly customize them: preset values; fields and subregions in any shape; multifold local uniqueness. Comparative results indicate that the proposed method has superior efficiency, quality, and capability. It makes marker field generation accessible to common marker designers, opening up more possibilities for fiducial markers.

Deep Instance Segmentation and Visual Servoing to Play Jenga with a Cost-Effective Robotic System

The game of Jenga is a benchmark used for developing innovative manipulation solutions for complex tasks. Indeed, it encourages the study of novel robotics methods to successfully extract blocks from a tower. A Jenga game involves many traits of complex industrial and surgical manipulation tasks, requiring a multi-step strategy, the combination of visual and tactile data, and the highly precise motion of a robotic arm to perform a single block extraction. In this work, we propose a novel, cost-effective architecture for playing Jenga with e.Do, a 6DOF anthropomorphic manipulator manufactured by Comau, a standard depth camera, and an inexpensive monodirectional force sensor. Our solution focuses on a visual-based control strategy to accurately align the end-effector with the desired block, enabling block extraction by pushing. To this aim, we trained an instance segmentation deep learning model on a synthetic custom dataset to segment each piece of the Jenga tower, allowing for visual tracking of the desired block's pose during the motion of the manipulator. We integrated the visual-based strategy with a 1D force sensor to detect whether the block could be safely removed by identifying a force threshold value. Our experimentation shows that our low-cost solution allows e.DO to precisely reach removable blocks and perform up to 14 consecutive extractions in a row.

Planning and visual-servoing for robotic manipulators in ROS

This article presents a probabilistic road map (PRM) and visual servo control (visual-servoing) based path planning strategy that allows a Motoman HP20D industrial robot to move from an initial positional to a random final position in the presence of fixed obstacles. The process begins with an application of the PRM algorithm to take the robot from an initial position to a point in space where it has a free line of sight to the target, to then apply visual servoing and end up, finally, at the desired position, where an image captured by a camera located at the robot’s end effector matches a reference image, located on the upper surface of a rectangular prismatic object. Algorithms and experiments were developed in simulation, specifically, the visual servo control that includes the dynamic model of the robot and the image sensor subject to realistic lighting were developed in robot operating system (ROS) environment.

Robust trocar identification and its application in robotic minimally invasive surgery

BACKGROUND

In minimally invasive surgery (MIS), instruments pass through trocars which are installed through the incision points. This forms a fulcrum effect and imposes significant constraint. For robotic manipulative operations, the real-time trocar information is a prerequisite. Systems acquire this knowledge either with a prior registration procedure or through coordinated control of their joints.

METHODS

A robust and real-time trocar identification algorithm based on least square (LS) algorithm was proposed in the context of human-robot co-manipulation scenario.

RESULTS

Both in vitro and in vivo experiments were performed to verify the effectiveness of the proposed algorithm. The estimated trocar coordinates expressed in the robot base frame were further leveraged to implement an instrument gravity compensation function.

CONCLUSIONS

An LS based approach can be employed to robustly estimate the real-time trocar information so as to implement more practical robotic functions.

Microsurgical Tool Detection and Characterization in Intra-operative Neurosurgical Videos

Brain surgery is complex and has evolved as a separate surgical specialty. Surgical procedures on the brain are performed using dedicated micro-instruments which are designed specifically for the requirements of operating with finesse in a confined space. The usage of these microsurgical tools in an operating environment defines the surgical skill of a surgeon. Video recordings of micro-surgical procedures are a rich source of information to develop automated surgical assessment tools that can offer continuous feedback for surgeons to improve their skills, effectively increase the outcome of the surgery, and make a positive impact on their patients. This work presents a novel deep learning system based on the Yolov5 algorithm to automatically detect, localize and characterize microsurgical tools from recorded intra-operative neurosurgical videos. The tool detection achieves a high 93.2% mean average precision. The detected tools are then characterized by their on-off time, motion trajectory and usage time. Tool characterization from neurosurgical videos offers useful insight into the surgical methods employed by a surgeon and can aid in their improvement. Additionally, a new dataset of annotated neurosurgical videos is used to develop the robust model and is made available for the research community.Clinical relevance- Tool detection and characterization in neurosurgery has several online and offline applications including skill assessment and outcome of the surgery. The development of automated tool characterization systems for intra-operative neurosurgery is expected to not only improve the surgical skills of the surgeon, but also leverage in training the neurosurgical workforce. Additionally, dedicated neurosurgical video based datasets will, in general, aid the research community to explore more automation in this field.

Oriented Localization of Surgical Tools by Location Encoding

Surgical tool localization is the foundation to a series of advanced surgical functions e.g. image guided surgical navigation. For precise scenarios like surgical tool localization, sophisticated tools and sensitive tissues can be quite close. This requires a higher localization accuracy than general object localization. And it is also meaningful to know the orientation of tools. To achieve these, this paper proposes a Compressive Sensing based Location Encoding scheme, which formulates the task of surgical tool localization in pixel space into a task of vector regression in encoding space. Furthermore with this scheme, the method is able to capture orientation of surgical tools rather than simply outputting horizontal bounding boxes. To prevent gradient vanishing, a novel back-propagation rule for sparse reconstruction is derived. The back-propagation rule is applicable to different implementations of sparse reconstruction and renders the entire network end-to-end trainable. Finally, the proposed approach gives more accurate bounding boxes as well as capturing the orientation of tools, and achieves state-of-the-art performance compared with 9 competitive both oriented and non-oriented localization methods (RRD, RefineDet, etc) on a mainstream surgical image dataset: m2cai16-tool-locations. A range of experiments support our claim that regression in CSLE space performs better than traditionally detecting bounding boxes in pixel space.

StereoCNC: A Stereovision-guided Robotic Laser System

This paper proposes an End-to-End stereovision-guided laser surgery system that can conduct laser ablation on targets selected by human operators in the color image, referred as StereoCNC. Two digital cameras are integrated into a previously developed robotic laser system to add a color sensing modality and formulate the stereovision. A calibration method is implemented to register the coordinate frames between stereo cameras and the laser system, modelled as a 3D-to-3D least-squares problem. The calibration reprojection errors are used to characterize a 3D error field by Gaussian Process Regression (GPR). This error field can make predictions for new point cloud data to identify an optimal position with lower calibration errors. A stereovision-guided laser ablation pipeline is proposed to optimize the positioning of the surgical site within the error field, which is achieved with a Genetic Algorithm search; mechanical stages move the site to the low-error region. The pipeline is validated by the experiments on phantoms with color texture and various geometric shapes. The overall targeting accuracy of the system achieved an average RMSE of 0.13 ± 0.02 mm and maximum error of 0.34 ± 0.06 mm, as measured by pre- and post-laser ablation images. The results show potential applications of using the developed stereovision-guided robotic system for superficial laser surgery, including dermatologic applications or removal of exposed tumorous tissue in neurosurgery.

Detection, segmentation, and 3D pose estimation of surgical tools using convolutional neural networks and algebraic geometry

BACKGROUND AND OBJECTIVE

Surgical tool detection, segmentation, and 3D pose estimation are crucial components in Computer-Assisted Laparoscopy (CAL). The existing frameworks have two main limitations. First, they do not integrate all three components. Integration is critical; for instance, one should not attempt computing pose if detection is negative. Second, they have highly specific requirements, such as the availability of a CAD model. We propose an integrated and generic framework whose sole requirement for the 3D pose is that the tool shaft is cylindrical. Our framework makes the most of deep learning and geometric 3D vision by combining a proposed Convolutional Neural Network (CNN) with algebraic geometry. We show two applications of our framework in CAL: tool-aware rendering in Augmented Reality (AR) and tool-based 3D measurement.

METHODS

We name our CNN as ART-Net (Augmented Reality Tool Network). It has a Single Input Multiple Output (SIMO) architecture with one encoder and multiple decoders to achieve detection, segmentation, and geometric primitive extraction. These primitives are the tool edge-lines, mid-line, and tip. They allow the tool's 3D pose to be estimated by a fast algebraic procedure. The framework only proceeds if a tool is detected. The accuracy of segmentation and geometric primitive extraction is boosted by a new Full resolution feature map Generator (FrG). We extensively evaluate the proposed framework with the EndoVis and new proposed datasets. We compare the segmentation results against several variants of the Fully Convolutional Network (FCN) and U-Net. Several ablation studies are provided for detection, segmentation, and geometric primitive extraction. The proposed datasets are surgery videos of different patients.

RESULTS

In detection, ART-Net achieves 100.0% in both average precision and accuracy. In segmentation, it achieves 81.0% in mean Intersection over Union (mIoU) on the robotic EndoVis dataset (articulated tool), where it outperforms both FCN and U-Net, by 4.5pp and 2.9pp, respectively. It achieves 88.2% in mIoU on the remaining datasets (non-articulated tool). In geometric primitive extraction, ART-Net achieves 2.45^∘ and 2.23^∘ in mean Arc Length (mAL) error for the edge-lines and mid-line, respectively, and 9.3 pixels in mean Euclidean distance error for the tool-tip. Finally, in terms of 3D pose evaluated on animal data, our framework achieves 1.87 mm, 0.70 mm, and 4.80 mm mean absolute errors on the X,Y, and Z coordinates, respectively, and 5.94^∘ angular error on the shaft orientation. It achieves 2.59 mm and 1.99 mm in mean and median location error of the tool head evaluated on patient data.

CONCLUSIONS

The proposed framework outperforms existing ones in detection and segmentation. Compared to separate networks, integrating the tasks in a single network preserves accuracy in detection and segmentation but substantially improves accuracy in geometric primitive extraction. Overall, our framework has similar or better accuracy in 3D pose estimation while largely improving robustness against the very challenging imaging conditions of laparoscopy. The source code of our framework and our annotated dataset will be made publicly available at https://github.com/kamruleee51/ART-Net.

An Optimization-Based Algorithm for Trajectory Planning of an Under-Actuated Robotic Arm to Perform Autonomous Suturing

In single-port access surgeries, robot size is crucial due to the limited workspace. Thus, a robot may be designed under-actuated. Suturing, in contrast, is a complicated task and requires full actuation. This study aims to overcome this shortcoming by implementing an optimization-based algorithm for autonomous suturing for an under-actuated robot. The proposed algorithm approximates the ideal suturing trajectory by slightly reorienting the needle while deviating as little as possible from the ideal, full degree-of-freedom suturing case. The deviation of the path taken by a custom robot with respect to the ideal trajectory varies depending on the suturing starting location within the workspace as well as the needle size. A quantitative analysis reveals that in 13% of the investigated workspace, the accumulative deviation was less than 10 mm. In the remaining workspace, the accumulative deviation was less than 30 mm. Likewise, the accumulative deviation of a needle with a radius of 10 mm was 2.2 mm as opposed to 8 mm when the radius was 20 mm. The optimization-based algorithm maximized the accuracy of a four-DOF robot to perform a path-constrained trajectory and illustrates the accuracy-workspace correlation.

Foot-Controlled Robot-Enabled EnDOscope Manipulator (FREEDOM) for Sinus Surgery: Design, Control, and Evaluation

Despite successful clinical applications, teleoperated robotic surgical systems face particular limitations in the functional endoscopic sinus surgery (FESS) in terms of incompatible instrument dimensions and robot set-up. The endoscope remains manually handled by an assistant when the surgeon performs bimanual operations. This paper introduces the development of the Foot-controlled Robot-Enabled EnDOscope Manipulator (FREEDOM) designed for FESS. The system features clinical considerations that inform the design for providing reliable and safe endoscope positioning with minimal obstruction to the routine practice. The robot structure is modular and compact to ensure coaxial instrument manipulation through the nostril for manual procedures. To avoid rigid endoscope motions, a new compliant endoscope holder is proposed that passively limits the lens-tissue contact forces under collisions for patient-side protection. To facilitate hands-free endoscope manipulation that imposes minimal distractions to the surgeon, a foot-wearable interface is further designed to relieve the assistant's workload. The foot control method owns a short learning curve (mean 3.4 mins), and leads the task to be more ergonomic and surgeon-centered. Cadaver and clinical studies were both conducted to evaluate the surgical applicability of the FREEDOM to assist endoscope manipulation in FESS. The system was validated to be safe (IEC-60601-1) and easy for set up (mean 3.6 mins), from which the surgeon could perform various three-handed procedures alone in FESS without disrupting the routine practice.

CATARACTS: Challenge on automatic tool annotation for cataRACT surgery

Surgical tool detection is attracting increasing attention from the medical image analysis community. The goal generally is not to precisely locate tools in images, but rather to indicate which tools are being used by the surgeon at each instant. The main motivation for annotating tool usage is to design efficient solutions for surgical workflow analysis, with potential applications in report generation, surgical training and even real-time decision support. Most existing tool annotation algorithms focus on laparoscopic surgeries. However, with 19 million interventions per year, the most common surgical procedure in the world is cataract surgery. The CATARACTS challenge was organized in 2017 to evaluate tool annotation algorithms in the specific context of cataract surgery. It relies on more than nine hours of videos, from 50 cataract surgeries, in which the presence of 21 surgical tools was manually annotated by two experts. With 14 participating teams, this challenge can be considered a success. As might be expected, the submitted solutions are based on deep learning. This paper thoroughly evaluates these solutions: in particular, the quality of their annotations are compared to that of human interpretations. Next, lessons learnt from the differential analysis of these solutions are discussed. We expect that they will guide the design of efficient surgery monitoring tools in the near future.

Vision-Based Surgical Tool Pose Estimation for the da Vinci® Robotic Surgical System

This paper presents an approach to surgical tool tracking using stereo vision for the da Vinci® Surgical Robotic System. The proposed method is based on robot kinematics, computer vision techniques and Bayesian state estimation. The proposed method employs a silhouette rendering algorithm to create virtual images of the surgical tool by generating the silhouette of the defined tool geometry under the da Vinci® robot endoscopes. The virtual rendering method provides the tool representation in image form, which makes it possible to measure the distance between the rendered tool and real tool from endoscopic stereo image streams. Particle Filter algorithm employing the virtual rendering method is then used for surgical tool tracking. The tracking performance is evaluated on an actual da Vinci® surgical robotic system and a ROS/Gazebo-based simulation of the da Vinci® system.

Wavelet and shearlet-based image representations for visual servoing

A visual servoing scheme consists of a closed-loop control approach that uses visual information feedback to control the motion of a robotic system. Probably the most popular visual servoing method is image-based visual servoing (IBVS). This kind of method uses geometric visual features extracted from the image to design the control law. However, extracting, matching, and tracking geometric visual features over time significantly limits the versatility of visual servoing controllers in various industrial and medical applications, in particular for “low-structured” medical images, e.g. ultrasounds and optical coherence tomography modalities. To overcome the limits of conventional IBVS, one can consider novel visual servoing paradigms known as “ direct” or “ featureless” approaches. This paper deals with the development of a new generation of direct visual servoing methods in which the signal control inputs are the coefficients of a multiscale image representation. In particular, we consider the use of multiscale image representations that are based on discrete wavelet and shearlet transforms. Up to now, one of the main obstacles in the investigation of multiscale image representations for visual servoing schemes was the issue of obtaining an analytical formulation of the interaction matrix that links the variation of wavelet and shearlet coefficients to the spatial velocity of the camera and the robot. In this paper, we derive four direct visual servoing controllers: two that are based on subsampled respectively non-subsampled wavelet coefficients and two that are based on the coefficients of subsampled respectively non-subsampled discrete shearlet transforms. All proposed controllers were tested in both simulation and experimental scenarios (using a six-degree-of-freedom Cartesian robot in an eye-in-hand configuration). The objective of this paper is to provide an analysis of the respective strengths and weaknesses of wavelet- and shearlet-based visual servoing controllers.

Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection

Video understanding of robot-assisted surgery (RAS) videos is an active research area. Modeling the gestures and skill level of surgeons presents an interesting problem. The insights drawn may be applied in effective skill acquisition, objective skill assessment, real-time feedback, and human-robot collaborative surgeries. We propose a solution to the tool detection and localization open problem in RAS video understanding, using a strictly computer vision approach and the recent advances of deep learning. We propose an architecture using multimodal convolutional neural networks for fast detection and localization of tools in RAS videos. To the best of our knowledge, this approach will be the first to incorporate deep neural networks for tool detection and localization in RAS videos. Our architecture applies a region proposal network (RPN) and a multimodal two stream convolutional network for object detection to jointly predict objectness and localization on a fusion of image and temporal motion cues. Our results with an average precision of 91% and a mean computation time of 0.1 s per test frame detection indicate that our study is superior to conventionally used methods for medical imaging while also emphasizing the benefits of using RPN for precision and efficiency. We also introduce a new data set, ATLAS Dione, for RAS video understanding. Our data set provides video data of ten surgeons from Roswell Park Cancer Institute, Buffalo, NY, USA, performing six different surgical tasks on the daVinci Surgical System (dVSS) with annotations of robotic tools per frame.

Robust Visual Control of Parallel Robots under Uncertain Camera Orientation

This work presents a stability analysis and experimental assessment of a visual control algorithm applied to a redundant planar parallel robot under uncertainty in relation to camera orientation. The key feature of the analysis is a strict Lyapunov function that allows the conclusion of asymptotic stability without invoking the Barbashin-Krassovsky-LaSalle invariance theorem. The controller does not rely on velocity measurements and has a structure similar to a classic Proportional Derivative control algorithm. Experiments in a laboratory prototype show that uncertainty in camera orientation does not significantly degrade closed-loop performance.

System Integration and In Vivo Testing of a Robot for Ultrasound Guidance and Monitoring During Radiotherapy

We are developing a cooperatively controlled robot system for image-guided radiation therapy (IGRT) in which a clinician and robot share control of a 3-D ultrasound (US) probe. IGRT involves two main steps: 1) planning/simulation and 2) treatment delivery. The goals of the system are to provide guidance for patient setup and real-time target monitoring during fractionated radiotherapy of soft tissue targets, especially in the upper abdomen. To compensate for soft tissue deformations created by the probe, we present a novel workflow where the robot holds the US probe on the patient during acquisition of the planning computerized tomography image, thereby ensuring that planning is performed on the deformed tissue. The robot system introduces constraints (virtual fixtures) to help to produce consistent soft tissue deformation between simulation and treatment days, based on the robot position, contact force, and reference US image recorded during simulation. This paper presents the system integration and the proposed clinical workflow, validated by an in vivo canine study. The results show that the virtual fixtures enable the clinician to deviate from the recorded position to better reproduce the reference US image, which correlates with more consistent soft tissue deformation and the possibility for more accurate patient setup and radiation delivery.

Vision-based and marker-less surgical tool detection and tracking: a review of the literature

In recent years, tremendous progress has been made in surgical practice for example with Minimally Invasive Surgery (MIS). To overcome challenges coming from deported eye-to-hand manipulation, robotic and computer-assisted systems have been developed. Having real-time knowledge of the pose of surgical tools with respect to the surgical camera and underlying anatomy is a key ingredient for such systems. In this paper, we present a review of the literature dealing with vision-based and marker-less surgical tool detection. This paper includes three primary contributions: (1) identification and analysis of data-sets used for developing and testing detection algorithms, (2) in-depth comparison of surgical tool detection methods from the feature extraction process to the model learning strategy and highlight existing shortcomings, and (3) analysis of validation techniques employed to obtain detection performance results and establish comparison between surgical tool detectors. The papers included in the review were selected through PubMed and Google Scholar searches using the keywords: "surgical tool detection", "surgical tool tracking", "surgical instrument detection" and "surgical instrument tracking" limiting results to the year range 2000 2015. Our study shows that despite significant progress over the years, the lack of established surgical tool data-sets, and reference format for performance assessment and method ranking is preventing faster improvement.

Automatic Detection of Instruments in Laparoscopic Images: A First Step Towards High-level Command of Robotic Endoscopic Holders

The tracking of surgical instruments o fers interesting possibilities for the development of high-level commands for robotic camera holders in laparoscopic surgery. We have developed a new method to detect instruments in laparoscopic images which uses information on the 3D position of the insertion point of an instrument into the abdominal cavity. This information strongly constrains the search for the instrument in each endoscopic image. Hence, the instrument can be detected in near real-time using shape considerations. Early results on laparoscopic images show that the method is rapid and robust in the presence of partial occlusion and smoke. Our first experiment on a cadaver validates our approach and shows encouraging results.

Detecting Surgical Tools by Modelling Local Appearance and Global Shape

Detecting tools in surgical videos is an important ingredient for context-aware computer-assisted surgical systems. To this end, we present a new surgical tool detection dataset and a method for joint tool detection and pose estimation in 2d images. Our two-stage pipeline is data-driven and relaxes strong assumptions made by previous works regarding the geometry, number, and position of tools in the image. The first stage classifies each pixel based on local appearance only, while the second stage evaluates a tool-specific shape template to enforce global shape. Both local appearance and global shape are learned from training data. Our method is validated on a new surgical tool dataset of 2 476 images from neurosurgical microscopes, which is made freely available. It improves over existing datasets in size, diversity and detail of annotation. We show that our method significantly improves over competitive baselines from the computer vision field. We achieve 15% detection miss-rate at 10(-1) false positives per image (for the suction tube) over our surgical tool dataset. Results indicate that performing semantic labelling as an intermediate task is key for high quality detection.

Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions

PURPOSE

Advances in technology and computing play an increasingly important role in the evolution of modern surgical techniques and paradigms. This article reviews the current role of machine learning (ML) techniques in the context of surgery with a focus on surgical robotics (SR). Also, we provide a perspective on the future possibilities for enhancing the effectiveness of procedures by integrating ML in the operating room.

METHODS

The review is focused on ML techniques directly applied to surgery, surgical robotics, surgical training and assessment. The widespread use of ML methods in diagnosis and medical image computing is beyond the scope of the review. Searches were performed on PubMed and IEEE Explore using combinations of keywords: ML, surgery, robotics, surgical and medical robotics, skill learning, skill analysis and learning to perceive.

RESULTS

Studies making use of ML methods in the context of surgery are increasingly being reported. In particular, there is an increasing interest in using ML for developing tools to understand and model surgical skill and competence or to extract surgical workflow. Many researchers begin to integrate this understanding into the control of recent surgical robots and devices.

CONCLUSION

ML is an expanding field. It is popular as it allows efficient processing of vast amounts of data for interpreting and real-time decision making. Already widely used in imaging and diagnosis, it is believed that ML will also play an important role in surgery and interventional treatments. In particular, ML could become a game changer into the conception of cognitive surgical robots. Such robots endowed with cognitive skills would assist the surgical team also on a cognitive level, such as possibly lowering the mental load of the team. For example, ML could help extracting surgical skill, learned through demonstration by human experts, and could transfer this to robotic skills. Such intelligent surgical assistance would significantly surpass the state of the art in surgical robotics. Current devices possess no intelligence whatsoever and are merely advanced and expensive instruments.

Vision-based real-time position control of a semi-automated system for robot-assisted joint fracture surgery

PURPOSE

Joint fracture surgery quality can be improved by robotic system with high-accuracy and high-repeatability fracture fragment manipulation. A new real-time vision-based system for fragment manipulation during robot-assisted fracture surgery was developed and tested.

METHODS

The control strategy was accomplished by merging fast open-loop control with vision-based control. This two-phase process is designed to eliminate the open-loop positioning errors by closing the control loop using visual feedback provided by an optical tracking system. Evaluation of the control system accuracy was performed using robot positioning trials, and fracture reduction accuracy was tested in trials on ex vivo porcine model.

RESULTS

The system resulted in high fracture reduction reliability with a reduction accuracy of 0.09 mm (translations) and of [Formula: see text] (rotations), maximum observed errors in the order of 0.12 mm (translations) and of [Formula: see text] (rotations), and a reduction repeatability of 0.02 mm and [Formula: see text].

CONCLUSIONS

The proposed vision-based system was shown to be effective and suitable for real joint fracture surgical procedures, contributing a potential improvement of their quality.

Computer-simulated biopsy marking system for endoscopic surveillance of gastric lesions: a pilot study

Endoscopic tattoo with India ink injection for surveillance of premalignant gastric lesions is technically cumbersome and may not be durable. The aim of the study is to evaluate the accuracy of a novel, computer-simulated biopsy marking system (CSBMS) developed for the endoscopic marking of gastric lesions. Twenty-five patients with history of gastric intestinal metaplasia received both CSBMS-guided marking and India ink injection in five points in the stomach at index endoscopy. A second endoscopy was performed at three months. Primary outcome was accuracy of CSBMS (distance between CSBMS probe-guided site and tattoo site measured by CSBMS). The mean accuracy of CSBMS at angularis was 5.3 ± 2.2 mm, antral lesser curvature 5.7 ± 1.4 mm, antral greater curvature 6.1 ± 1.1 mm, antral anterior wall 6.9 ± 1.6 mm, and antral posterior wall 6.9 ± 1.6 mm. CSBMS (2.3 ± 0.9 versus 12.5 ± 4.6 seconds; P = 0.02) required less procedure time compared to endoscopic tattooing. No adverse events were encountered. CSBMS accurately identified previously marked gastric sites by endoscopic tattooing within 1 cm on follow-up endoscopy.

A touch panel surgical navigation system with automatic depth perception

PURPOSE

A touch panel navigation system may be used to enhance endoscopic surgery, especially for cauterization. We developed and tested the in vitro performance of a new touch panel navigation (TPN) system.

METHODS

This TPN system uses finger motion trajectories on a touch panel to control an argon plasma coagulation (APC) attached to a robot arm. Thermal papers with printed figures were soaked in saline for repeated recording and analysis of cauterized trajectory. A novice and an expert surgeon traced squares and circles displayed on the touch panel and cauterized them using the APC. Sixteen novices and eight experts cauterized squares and circles using both conventional endoscopic and TPN procedures. Six novices cauterized arcs using the endoscopic and TPN procedures 20 times a day for 5 consecutive days.

RESULTS

For square shapes, the offset was 5.5 mm with differences between the novice and the expert at 2 of 16 points. For circles, the offset was 5.0 mm and did not differ at any point. Task completion time for the TPN procedure was significantly longer than that for the endoscopic procedure for both squares and circles. For squares, the distance from the target for the TPN procedure was significantly smaller than that for the endoscopic procedure. For circles, the distance did not differ. There was no difference in task completion time and distance between the novices and the experts. Task completion time and distance improved significantly for the endoscopic procedure but not for the TPN procedure.

CONCLUSIONS

A new TPN system enabled the surgeons to accomplish continuous 3D positioning of the surgical device with automatic depth perception using finger tracing on a 2D monitor. This technology is promising for application in surgical procedures that require precise control of cauterization.

Appearance learning for 3D tracking of robotic surgical tools

In this paper, we present an appearance learning approach which is used to detect and track surgical robotic tools in laparoscopic sequences. By training a robust visual feature descriptor on low-level landmark features, we build a framework for fusing robot kinematics and 3D visual observations to track surgical tools over long periods of time across various types of environment. We demonstrate 3D tracking on multiple types of tool (with different overall appearances) as well as multiple tools simultaneously. We present experimental results using the da Vinci® surgical robot using a combination of both ex-vivo and in-vivo environments.

A minimally invasive surgery robotic assistant for HALS-SILS techniques

This paper is focused in the design and implementation of a robotic surgical motion controller. The proposed control scheme addresses the issues related to the application of a robot assistant in novel surgical scenario, which combines hand assisted laparoscopic surgery (HALS) with the single incision laparoscopic surgery (SILS) techniques. It is designed for collaborating with the surgeon in a natural way, by performing autonomous movements, in order to assist the surgeon during a surgical maneuver. In this way, it is implemented a hierarchical architecture which includes an upper auto-guide velocity planner connected to a low-level force feedback controller. The first one, based on a behavior approach, computes a collision free trajectory of the surgical instrument tip, held by the robot, for reaching a goal location inside of the abdominal cavity. On the other hand, the force feedback controller uses this trajectory for performing the instrument displacement by taking into account the holonomic movement constraints introduced by the fulcrum point. The aim of this controller is positioning the surgical instrument by minimizing the forces exerted over the abdominal wall due to the fulcrum location uncertainty. The overall system has been integrated in the control architecture of the surgical assistant CISOBOT, designed and developed at the University of Malaga. The whole architecture performance has been tested by means of in vitro trials.

Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging - techniques and applications

BACKGROUND

Intra-operative imaging is widely used to provide visual feedback to a clinician when he/she performs a procedure. In visual servoing, surgical instruments and parts of tissue/body are tracked by processing the acquired images. This information is then used within a control loop to manoeuvre a robotic manipulator during a procedure.

METHODS

A comprehensive search of electronic databases was completed for the period 2000-2013 to provide a survey of the visual servoing applications in medical robotics. The focus is on medical applications where image-based tracking is used for closed-loop control of a robotic system.

RESULTS

Detailed classification and comparative study of various contributions in visual servoing using endoscopic or direct visual images are presented and summarized in tables and diagrams.

CONCLUSION

The main challenges in using visual servoing for medical robotic applications are identified and potential future directions are suggested. 'Supervised automation of medical robotics' is found to be a major trend in this field.

Toward detection and localization of instruments in minimally invasive surgery

Methods for detecting and localizing surgical instruments in laparoscopic images are an important element of advanced robotic and computer-assisted interventions. Robotic joint encoders and sensors integrated or mounted on the instrument can provide information about the tool's position, but this often has inaccuracy when transferred to the surgeon's point of view. Vision sensors are currently a promising approach for determining the position of instruments in the coordinate frame of the surgical camera. In this study, we propose a vision algorithm for localizing the instrument's pose in 3-D leaving only rotation in the axis of the tool's shaft as an ambiguity. We propose a probabilistic supervised classification method to detect pixels in laparoscopic images that belong to surgical tools. We then use the classifier output to initialize an energy minimization algorithm for estimating the pose of a prior 3-D model of the instrument within a level set framework. We show that the proposed method is robust against noise using simulated data and we perform quantitative validation of the algorithm compared to ground truth obtained using an optical tracker. Finally, we demonstrate the practical application of the technique on in vivo data from minimally invasive surgery with traditional laparoscopic and robotic instruments.

EVA: laparoscopic instrument tracking based on Endoscopic Video Analysis for psychomotor skills assessment

INTRODUCTION

The EVA (Endoscopic Video Analysis) tracking system is a new system for extracting motions of laparoscopic instruments based on nonobtrusive video tracking. The feasibility of using EVA in laparoscopic settings has been tested in a box trainer setup.

METHODS

EVA makes use of an algorithm that employs information of the laparoscopic instrument's shaft edges in the image, the instrument's insertion point, and the camera's optical center to track the three-dimensional position of the instrument tip. A validation study of EVA comprised a comparison of the measurements achieved with EVA and the TrEndo tracking system. To this end, 42 participants (16 novices, 22 residents, and 4 experts) were asked to perform a peg transfer task in a box trainer. Ten motion-based metrics were used to assess their performance.

RESULTS

Construct validation of the EVA has been obtained for seven motion-based metrics. Concurrent validation revealed that there is a strong correlation between the results obtained by EVA and the TrEndo for metrics, such as path length (ρ = 0.97), average speed (ρ = 0.94), or economy of volume (ρ = 0.85), proving the viability of EVA.

CONCLUSIONS

EVA has been successfully validated in a box trainer setup, showing the potential of endoscopic video analysis to assess laparoscopic psychomotor skills. The results encourage further implementation of video tracking in training setups and image-guided surgery.

Surgical vision

The emergence of Minimal Access Surgery (MAS) as a paradigm in modern healthcare treatment has created new challenges and opportunities for automated image understanding and computer vision. In MAS, images recovered from inside the body using specialized devices are used to visualize and operate on the surgical site but they can also be used to computationally infer in vivo 3D tissue surface shape, soft-tissue morphology, and surgical instrument motion. This information is important for facilitating in vivo biophotonic imaging modalities where the interaction between light and tissue is used to infer the structural and functional properties of the tissue. This article provides a review of the literature for computer vision and image understanding techniques applied to MAS images. The focus of this article is to elucidate a perspective on how computer vision techniques can be used to support and enhance the capabilities of biophotonic imaging modalities during surgery. Note that while MAS encompasses a variety of surgical specializations this review does not involve procedures performed in the interventional suite. The review has been carried out based on searches in the PubMed and IEEE databases using the article's keywords.