Minimally invasive holographic surface scanning for soft-tissue image registration

Patient-specific simulations and navigation systems for partial nephrectomy

Partial nephrectomy (PN) is the standard treatment for T1 renal cell carcinoma. PN is affected more by surgical variations and requires greater surgical experience than radical nephrectomy. Patient-specific simulations and navigation systems may help to reduce the surgical experience required for PN. Recent advances in three-dimensional (3D) virtual reality (VR) imaging and 3D printing technology have allowed accurate patient-specific simulations and navigation systems. We reviewed previous studies about patient-specific simulations and navigation systems for PN. Recently, image reconstruction technology has developed, and commercial software that converts two-dimensional images into 3D images has become available. Many urologists are now able to view 3DVR images when preparing for PN. Surgical simulations based on 3DVR images can change surgical plans and improve surgical outcomes, and are useful during patient consultations. Patient-specific simulators that are capable of simulating surgical procedures, the gold-standard form of patient-specific simulations, have also been reported. Besides VR, 3D printing is also useful for understanding patient-specific information. Some studies have reported simulation and navigation systems for PN based on solid 3D models. Patient-specific simulations are a form of preoperative preparation, whereas patient-specific navigation is used intraoperatively. Navigation-assisted PN procedures using 3DVR images have become increasingly common, especially in robotic surgery. Some studies found that these systems produced improvements in surgical outcomes. Once its accuracy has been confirmed, it is hoped that this technology will spread further and become more generalized.

A Method for Spatially Registered Microprofilometry Combining Intensity-Height Datasets from Interferometric Sensors

A recognized problem in profilometry applied to artworks is the spatial referencing of the surface topography at micrometer scale due to the lack of references in the height data with respect to the "visually readable" surface. We demonstrate a novel workflow for spatially referenced microprofilometry based on conoscopic holography sensors for scanning in situ heterogeneous artworks. The method combines the raw intensity signal collected by the single-point sensor and the (interferometric) height dataset, which are mutually registered. This dual dataset provides a surface topography registered to the artwork features up to the precision that is given by the acquisition scanning system (mainly, scan step and laser spot). The advantages are: (1) the raw signal map provides additional information about materials texture, e.g., color changes or artist marks, for spatial registration and data fusion tasks; (2) and microtexture information can be reliably processed for precision diagnostic tasks, e.g., surface metrology in specific sub-domains and multi-temporal monitoring. Proof of concept is given with exemplary applications: book heritage, 3D artifacts, surface treatments. The potential of the method is clear for both quantitative surface metrology and qualitative inspection of the morphology, and it is expected to open future applications for microprofilometry in heritage science.

An incremental registration method for endoscopic sinus and skull base surgery navigation: From phantom study to clinical trials

PURPOSE

Surface-based image-to-patient registration in current surgical navigation is mainly achieved by a 3D scanner, which has several limitations in clinical practice such as uncontrollable scanning range, complicated operation, and even high failure rate. An accurate, robust, and easy-to-perform image-to-patient registration method is urgently required.

METHODS

An incremental point cloud registration method was proposed for surface-based image-to-patient registration. The point cloud in image space was extracted from the computed tomography (CT) image, and a template matching method was applied to remove the redundant points. The corresponding point cloud in patient space was incrementally collected by an optically tracked pointer, while the nearest point distance (NPD) constraint was applied to ensure the uniformity of the collected points. A coarse-to-fine registration method under the constraints of coverage ratio (CR) and outliers ratio (OR) was then proposed to obtain the optimal rigid transformation from image to patient space. The proposed method was integrated in the recently developed endoscopic navigation system, and phantom study and clinical trials were conducted to evaluate the performance of the proposed method.

RESULTS

The results of the phantom study revealed that the proposed constraints greatly improved the accuracy and robustness of registration. The comparative experimental results revealed that the proposed registration method significantly outperform the scanner-based method, and achieved comparable accuracy to the fiducial-based method. In the clinical trials, the average registration duration was 1.24 ± 0.43 min, the target registration error (TRE) of 294 marker points (59 patients) was 1.25 ± 0.40 mm, and the lower 97.5% confidence limit of the success rate of positioning marker points exceeds the expected value (97.56% vs. 95.00%), revealed that the accuracy of the proposed method significantly met the clinical requirements (TRE ⩽ 2 mm, p < 0.05).

CONCLUSIONS

The proposed method has both the advantages of high accuracy and convenience, which were absent in the scanner-based method and the fiducial-based method. Our findings will help improve the quality of endoscopic sinus and skull base surgery.

Patient-specific, touch-based registration during robotic, image-guided partial nephrectomy

Image-guidance during partial nephrectomy enables navigation within the operative field alongside a 3-dimensional roadmap of renal anatomy generated from patient-specific imaging. Once a process is performed by the human mind, the technology will allow standardization of the task for the benefit of all patients undergoing robot-assisted partial nephrectomy. Any surgeon will be able to visualize the kidney and key subsurface landmarks in real-time within a 3-dimensional simulation, with the goals of improving operative efficiency, decreasing surgical complications, and improving oncologic outcomes. For similar purposes, image-guidance has already been adopted as a standard of care in other surgical fields; we are now at the brink of this in urology. This review summarizes touch-based approaches to image-guidance during partial nephrectomy, as the technology begins to enter in vivo human evaluation. The processes of segmentation, localization, registration, and re-registration are all described with seamless integration into the da Vinci surgical system; this will facilitate clinical adoption sooner.

Calibrating 3D Scanner in the Coordinate System of Optical Tracker for Image-To-Patient Registration

Three-dimensional scanners have been widely applied in image-guided surgery (IGS) given its potential to solve the image-to-patient registration problem. How to perform a reliable calibration between a 3D scanner and an external tracker is especially important for these applications. This study proposes a novel method for calibrating the extrinsic parameters of a 3D scanner in the coordinate system of an optical tracker. We bound an optical marker to a 3D scanner and designed a specified 3D benchmark for calibration. We then proposed a two-step calibration method based on the pointset registration technique and nonlinear optimization algorithm to obtain the extrinsic matrix of the 3D scanner. We applied repeat scan registration error (RSRE) as the cost function in the optimization process. Subsequently, we evaluated the performance of the proposed method on a recaptured verification dataset through RSRE and Chamfer distance (CD). In comparison with the calibration method based on 2D checkerboard, the proposed method achieved a lower RSRE (1.73 mm vs. 2.10, 1.94, and 1.83 mm) and CD (2.83 mm vs. 3.98, 3.46, and 3.17 mm). We also constructed a surgical navigation system to further explore the application of the tracked 3D scanner in image-to-patient registration. We conducted a phantom study to verify the accuracy of the proposed method and analyze the relationship between the calibration accuracy and the target registration error (TRE). The proposed scanner-based image-to-patient registration method was also compared with the fiducial-based method, and TRE and operation time (OT) were used to evaluate the registration results. The proposed registration method achieved an improved registration efficiency (50.72 ± 6.04 vs. 212.97 ± 15.91 s in the head phantom study). Although the TRE of the proposed registration method met the clinical requirements, its accuracy was lower than that of the fiducial-based registration method (1.79 ± 0.17 mm vs. 0.92 ± 0.16 mm in the head phantom study). We summarized and analyzed the limitations of the scanner-based image-to-patient registration method and discussed its possible development.

Characterization and correction of intraoperative soft tissue deformation in image-guided laparoscopic liver surgery

Laparoscopic liver surgery is challenging to perform due to a compromised ability of the surgeon to localize subsurface anatomy in the constrained environment. While image guidance has the potential to address this barrier, intraoperative factors, such as insufflation and variable degrees of organ mobilization from supporting ligaments, may generate substantial deformation. The severity of laparoscopic deformation in humans has not been characterized, and current laparoscopic correction methods do not account for the mechanics of how intraoperative deformation is applied to the liver. We first measure the degree of laparoscopic deformation at two insufflation pressures over the course of laparoscopic-to-open conversion in 25 patients. With this clinical data alongside a mock laparoscopic phantom setup, we report a biomechanical correction approach that leverages anatomically load-bearing support surfaces from ligament attachments to iteratively reconstruct and account for intraoperative deformations. Laparoscopic deformations were significantly larger than deformations associated with open surgery, and our correction approach yielded subsurface target error of [Formula: see text] and surface error of [Formula: see text] using only sparse surface data with realistic surgical extent. Laparoscopic surface data extents were examined and found to impact registration accuracy. Finally, we demonstrate viability of the correction method with clinical data.

A method for going from 2D laparoscope to 3D acquisition of surface landmarks by a novel computer vision approach

PURPOSE

This paper presents a method to use the Smart Trocars-our new surgical instrument recognition system-or any accurate localization system of surgical instrument for acquiring intraoperative surface data. Complex laparoscopic surgeries need a proper guidance system which requires registering the preoperative data from a CT or MRI scan to the intraoperative patient state. The Smart Trocar can be used to localize the instruments when it comes to contact with the soft tissue surface.

METHOD

Two successive views through the laparoscope at different angles with the 3D localization of a fixed tool at one single location using the Smart Trocars can point out visible features during the surgery and acquire their location in 3D to provide a depth map in the region of interest. In other words, our method transforms a standard laparoscope system into a system with three-dimensional registration capability.

RESULT

This method was initially tested on a simulation for uncertainty assessment and then on a rigid model for verification with an accuracy within 2 mm distance. In addition, an in vivo experiment on pig model was also conducted to investigate how the method might be used during a physiologic respiratory cycle.

CONCLUSION

This method can be applied in a large number of surgical applications as a guidance system on its own or in conjunction with other navigation techniques. Our work encourages further testing with realistic surgical applications in the near future.

Improving Registration Robustness for Image-Guided Liver Surgery in a Novel Human-to-Phantom Data Framework

In open image-guided liver surgery (IGLS), a sparse representation of the intraoperative organ surface can be acquired to drive image-to-physical registration. We hypothesize that uncharacterized error induced by variation in the collection patterns of organ surface data limits the accuracy and robustness of an IGLS registration. Clinical validation of such registration methods is challenged due to the difficulty in obtaining data representative of the true state of organ deformation. We propose a novel human-to-phantom validation framework that transforms surface collection patterns from in vivo IGLS procedures (n = 13) onto a well-characterized hepatic deformation phantom for the purpose of validating surface-driven, volumetric nonrigid registration methods. An important feature of the approach is that it centers on combining workflow-realistic data acquisition and surgical deformations that are appropriate in behavior and magnitude. Using the approach, we investigate volumetric target registration error (TRE) with both current rigid IGLS and our improved nonrigid registration methods. Additionally, we introduce a spatial data resampling approach to mitigate the workflow-sensitive sampling problem. Using our human-to-phantom approach, TRE after routine rigid registration was 10.9 ± 0.6 mm with a signed closest point distance associated with residual surface fit in the range of ±10 mm, highly representative of open liver resections. After applying our novel resampling strategy and improved deformation correction method, TRE was reduced by 51%, i.e., a TRE of 5.3 ± 0.5 mm. This paper reported herein realizes a novel tractable approach for the validation of image-to-physical registration methods and demonstrates promising results for our correction method.

Contact-less stylus for surgical navigation: registration without digitization

PURPOSE

We present a laser-based, contact-less, stylus for the purpose of fiducial registration and digitization in the context of surgical navigation.

METHODS

We augmented a laser pointer with a spatial measurement device and used the laser beam as a means to locate a fiducial in 3D space. We developed a method for calibrating the orientation of the laser beam with respect to its attached tracking target. Digitization of a fiducial was formulated as a line intersection problem, and registration was formulated as a point-to-line registration problem.

RESULTS

We achieved an RMS fiducial localization error of 0.63 mm for 151 measurements of 12 fiducial markers. Mean TRE values of less than 1.5 mm over the entire surface of a lumbar vertebra were achievable using 4 fiducial markers. We found that contact-based rigid registration performed carefully under near-ideal conditions outperforms contact-less registration in terms of TRE.

CONCLUSION

An inexpensive contact-less stylus can be used to obtain accurate fiducial registration, which can be performed without explicit fiducial digitization.

ConoSurf: Open-source 3D scanning system based on a conoscopic holography device for acquiring surgical surfaces

Background

A difficulty in computer‐assisted interventions is acquiring the patient's anatomy intraoperatively. Standard modalities have several limitations: low image quality (ultrasound), radiation exposure (computed tomography) or high costs (magnetic resonance imaging). An alternative approach uses a tracked pointer; however, the pointer causes tissue deformation and requires sterilizing. Recent proposals, utilizing a tracked conoscopic holography device, have shown promising results without the previously mentioned drawbacks.

Methods

We have developed an open‐source software system that enables real‐time surface scanning using a conoscopic holography device and a wide variety of tracking systems, integrated into pre‐existing and well‐supported software solutions.

Results

The mean target registration error of point measurements was 1.46 mm. For a quick guidance scan, surface reconstruction improved the surface registration error compared with point‐set registration.

Conclusions

We have presented a system enabling real‐time surface scanning using a tracked conoscopic holography device. Results show that it can be useful for acquiring the patient's anatomy during surgery.

A novel method for texture-mapping conoscopic surfaces for minimally invasive image-guided kidney surgery

PURPOSE

Organ-level registration is critical to image-guided therapy in soft tissue. This is especially important in organs such as the kidney which can freely move. We have developed a method for registration that combines three-dimensional locations from a holographic conoscope with an endoscopically obtained textured surface. By combining these data sources clear decisions as to the tissue from which the points arise can be made.

METHODS

By localizing the conoscope's laser dot in the endoscopic space, we register the textured surface to the cloud of conoscopic points. This allows the cloud of points to be filtered for only those arising from the kidney surface. Once a valid cloud is obtained we can use standard surface registration techniques to perform the image-space to physical-space registration. Since our methods use two distinct data sources we test for spatial accuracy and characterize temporal effects in phantoms, ex vivo porcine and human kidneys. In addition we use an industrial robot to provide controlled motion and positioning for characterizing temporal effects.

RESULTS

Our initial surface acquisitions are hand-held. This means that we take approximately 55 s to acquire a surface. At that rate we see no temporal effects due to acquisition synchronization or probe speed. Our surface registrations were able to find applied targets with submillimeter target registration errors.

CONCLUSION

The results showed that the textured surfaces could be reconstructed with submillimetric mean registration errors. While this paper focuses on kidney applications, this method could be applied to any anatomical structures where a line of sight can be created via open or minimally invasive surgical techniques.

A clinically applicable laser-based image-guided system for laparoscopic liver procedures

PURPOSE

Laser range scanners (LRS) allow performing a surface scan without physical contact with the organ, yielding higher registration accuracy for image-guided surgery (IGS) systems. However, the use of LRS-based registration in laparoscopic liver surgery is still limited because current solutions are composed of expensive and bulky equipment which can hardly be integrated in a surgical scenario.

METHODS

In this work, we present a novel LRS-based IGS system for laparoscopic liver procedures. A triangulation process is formulated to compute the 3D coordinates of laser points by using the existing IGS system tracking devices. This allows the use of a compact and cost-effective LRS and therefore facilitates the integration into the laparoscopic setup. The 3D laser points are then reconstructed into a surface to register to the preoperative liver model using a multi-level registration process.

RESULTS

Experimental results show that the proposed system provides submillimeter scanning precision and accuracy comparable to those reported in the literature. Further quantitative analysis shows that the proposed system is able to achieve a patient-to-image registration accuracy, described as target registration error, of [Formula: see text].

CONCLUSIONS

We believe that the presented approach will lead to a faster integration of LRS-based registration techniques in the surgical environment. Further studies will focus on optimizing scanning time and on the respiratory motion compensation.

Evaluation of conoscopic holography for estimating tumor resection cavities in model-based image-guided neurosurgery

Surgical navigation relies on accurately mapping the intraoperative state of the patient to models derived from preoperative images. In image-guided neurosurgery, soft tissue deformations are common and have been shown to compromise the accuracy of guidance systems. In lieu of whole-brain intraoperative imaging, some advocate the use of intraoperatively acquired sparse data from laser-range scans, ultrasound imaging, or stereo reconstruction coupled with a computational model to drive subsurface deformations. Some authors have reported on compensating for brain sag, swelling, retraction, and the application of pharmaceuticals such as mannitol with these models. To date, strategies for modeling tissue resection have been limited. In this paper, we report our experiences with a novel digitization approach, called a conoprobe, to document tissue resection cavities and assess the impact of resection on model-based guidance systems. Specifically, the conoprobe was used to digitize the interior of the resection cavity during eight brain tumor resection surgeries and then compared against model prediction results of tumor locations. We should note that no effort was made to incorporate resection into the model but rather the objective was to determine if measurement was possible to study the impact on modeling tissue resection. In addition, the digitized resection cavity was compared with early postoperative MRI scans to determine whether these scans can further inform tissue resection. The results demonstrate benefit in model correction despite not having resection explicitly modeled. However, results also indicate the challenge that resection provides for model-correction approaches. With respect to the digitization technology, it is clear that the conoprobe provides important real-time data regarding resection and adds another dimension to our noncontact instrumentation framework for soft-tissue deformation compensation in guidance systems.

Influence of surface position along the working range of conoscopic holography sensors on dimensional verification of AISI 316 wire EDM machined surfaces

Conoscopic holography (CH) is a non-contact interferometric technique used for surface digitization which presents several advantages over other optical techniques such as laser triangulation. Among others, the ability for the reconstruction of high-sloped surfaces stands out, and so does its lower dependence on surface optical properties. Nevertheless, similarly to other optical systems, adjustment of CH sensors requires an adequate selection of configuration parameters for ensuring a high quality surface digitizing. This should be done on a surface located as close as possible to the stand-off distance by tuning frequency (F) and power (P) until the quality indicators Signal-to-Noise Ratio (SNR) and signal envelope (Total) meet proper values. However, not all the points of an actual surface are located at the stand-off distance, but they could be located throughout the whole working range (WR). Thus, the quality of a digitized surface may not be uniform. The present work analyses how the quality of a reconstructed surface is affected by its relative position within the WR under different combinations of the parameters F and P. Experiments have been conducted on AISI 316 wire EDM machined flat surfaces. The number of high-quality points digitized as well as distance measurements between different surfaces throughout the WR allowed for comparing the metrological behaviour of the CH sensor with respect to a touch probe (TP) on a CMM.

A Mechanics-Based Nonrigid Registration Method for Liver Surgery Using Sparse Intraoperative Data

In open abdominal image-guided liver surgery, sparse measurements of the organ surface can be taken intraoperatively via a laser-range scanning device or a tracked stylus with relatively little impact on surgical workflow. We propose a novel nonrigid registration method which uses sparse surface data to reconstruct a mapping between the preoperative CT volume and the intraoperative patient space. The mapping is generated using a tissue mechanics model subject to boundary conditions consistent with surgical supportive packing during liver resection therapy. Our approach iteratively chooses parameters which define these boundary conditions such that the deformed tissue model best fits the intraoperative surface data. Using two liver phantoms, we gathered a total of five deformation datasets with conditions comparable to open surgery. The proposed nonrigid method achieved a mean target registration error (TRE) of 3.3 mm for targets dispersed throughout the phantom volume, using a limited region of surface data to drive the nonrigid registration algorithm, while rigid registration resulted in a mean TRE of 9.5 mm. In addition, we studied the effect of surface data extent, the inclusion of subsurface data, the trade-offs of using a nonlinear tissue model, robustness to rigid misalignments, and the feasibility in five clinical datasets.

A study on the theoretical and practical accuracy of conoscopic holography-based surface measurements: toward image registration in minimally invasive surgery

BACKGROUND

Registered medical images can assist with surgical navigation and enable image-guided therapy delivery. In soft tissues, surface-based registration is often used and can be facilitated by laser surface scanning. Tracked conoscopic holography (which provides distance measurements) has been recently proposed as a minimally invasive way to obtain surface scans. Moving this technique from concept to clinical use requires a rigorous accuracy evaluation, which is the purpose of our paper.

METHODS

We adapt recent non-homogeneous and anisotropic point-based registration results to provide a theoretical framework for predicting the accuracy of tracked distance measurement systems. Experiments are conducted a complex objects of defined geometry, an anthropomorphic kidney phantom and a human cadaver kidney.

RESULTS

Experiments agree with model predictions, producing point RMS errors consistently < 1 mm, surface-based registration with mean closest point error < 1 mm in the phantom and a RMS target registration error of 0.8 mm in the human cadaver kidney.

CONCLUSIONS

Tracked conoscopic holography is clinically viable; it enables minimally invasive surface scan accuracy comparable to current clinical methods that require open surgery.

3D-holoscopic imaging: a new dimension to enhance imaging in minimally invasive therapy in urologic oncology

BACKGROUND AND PURPOSE

Existing imaging modalities of urologic pathology are limited by three-dimensional (3D) representation on a two-dimensional screen. We present 3D-holoscopic imaging as a novel method of representing Digital Imaging and Communications in Medicine data images taken from CT and MRI to produce 3D-holographic representations of anatomy without special eyewear in natural light. 3D-holoscopic technology produces images that are true optical models. This technology is based on physical principles with duplication of light fields. The 3D content is captured in real time with the content viewed by multiple viewers independently of their position, without 3D eyewear.

METHODS

We display 3D-holoscopic anatomy relevant to minimally invasive urologic surgery without the need for 3D eyewear.

RESULTS

The results have demonstrated that medical 3D-holoscopic content can be displayed on commercially available multiview auto-stereoscopic display.

CONCLUSION

The next step is validation studies comparing 3D-Holoscopic imaging with conventional imaging.

Comparison study of intraoperative surface acquisition methods for surgical navigation

Soft-tissue image-guided interventions often require the digitization of organ surfaces for providing correspondence from medical images to the physical patient in the operating room. In this paper, the effect of several inexpensive surface acquisition techniques on target registration error and surface registration error (SRE) for soft tissue is investigated. A systematic approach is provided to compare image-to-physical registrations using three different methods of organ spatial digitization: 1) a tracked laser-range scanner (LRS), 2) a tracked pointer, and 3) a tracked conoscopic holography sensor (called a conoprobe). For each digitization method, surfaces of phantoms and biological tissues were acquired and registered to CT image volume counterparts. A comparison among these alignments demonstrated that registration errors were statistically smaller with the conoprobe than the tracked pointer and LRS (p<0.01). In all acquisitions, the conoprobe outperformed the LRS and tracked pointer: for example, the arithmetic means of the SRE over all data acquisitions with a porcine liver were 1.73 ± 0.77 mm, 3.25 ± 0.78 mm, and 4.44 ± 1.19 mm for the conoprobe, LRS, and tracked pointer, respectively. In a cadaveric kidney specimen, the arithmetic means of the SRE over all trials of the conoprobe and tracked pointer were 1.50 ± 0.50 mm and 3.51 ± 0.82 mm, respectively. Our results suggest that tissue displacements due to contact force and attempts to maintain contact with tissue, compromise registrations that are dependent on data acquired from a tracked surgical instrument and we provide an alternative method (tracked conoscopic holography) of digitizing surfaces for clinical usage. The tracked conoscopic holography device outperforms LRS acquisitions with respect to registration accuracy.

Image-guided robotic surgery: update on research and potential applications in urologic surgery

PURPOSE OF REVIEW

New methods of imaging and image-guidance technology have the potential to provide surgeons with spatially accurate three-dimensional information about the location and anatomical relationships of critical subsurface structures and instrument position updated and displayed during the performance of surgery. Robotic platforms and technology in various forms continues to revolutionize surgery and will soon incorporate image guidance.

RECENT RESEARCH

Image-guided surgery (IGS) for abdominal and urologic interventions presents complex engineering and surgical challenges along with potential benefits to surgeons and patients. Key concepts such as registration, localization, accuracy, and targeting error are necessary for surgeons to understand and utilize the potential of IGS. Standard robotic surgeries, such as partial nephrectomy and radical prostatectomy may soon incorporate IGS.

SUMMARY

Research continues to explore the potential for combining image guidance and robotics to augment and improve a variety of surgical interventions.