Image-to-Graph Convolutional Network for 2D/3D Deformable Model Registration of Low-Contrast Organs

Organ shape reconstruction based on a single-projection image during treatment has wide clinical scope, e.g., in image-guided radiotherapy and surgical guidance. We propose an image-to-graph convolutional network that achieves deformable registration of a three-dimensional (3D) organ mesh for a low-contrast two-dimensional (2D) projection image. This framework enables simultaneous training of two types of transformation: from the 2D projection image to a displacement map, and from the sampled per-vertex feature to a 3D displacement that satisfies the geometrical constraint of the mesh structure. Assuming application to radiation therapy, the 2D/3D deformable registration performance is verified for multiple abdominal organs that have not been targeted to date, i.e., the liver, stomach, duodenum, and kidney, and for pancreatic cancer. The experimental results show shape prediction considering relationships among multiple organs can be used to predict respiratory motion and deformation from digitally reconstructed radiographs with clinically acceptable accuracy.

Deformation analysis of surface and bronchial structures in intraoperative pneumothorax using deformable mesh registration

The positions of nodules can change because of intraoperative lung deflation, and the modeling of pneumothorax-associated deformation remains a challenging issue for intraoperative tumor localization. In this study, we introduce spatial and geometric analysis methods for inflated/deflated lungs and discuss heterogeneity in pneumothorax-associated lung deformation. Contrast-enhanced CT images simulating intraoperative conditions were acquired from live Beagle dogs. The images contain the overall shape of the lungs, including all lobes and internal bronchial structures, and were analyzed to provide a statistical deformation model that could be used as prior knowledge to predict pneumothorax. To address the difficulties of mapping pneumothorax CT images with topological changes and CT intensity shifts, we designed deformable mesh registration techniques for mixed data structures including the lobe surfaces and the bronchial centerlines. Three global-to-local registration steps were performed under the constraint that the deformation was spatially continuous and smooth, while matching visible bronchial tree structures as much as possible. The developed framework achieved stable registration with a Hausdorff distance of less than 1 mm and a target registration error of less than 5 mm, and visualized deformation fields that demonstrate per-lobe contractions and rotations with high variability between subjects. The deformation analysis results show that the strain of lung parenchyma was 35% higher than that of bronchi, and that deformation in the deflated lung is heterogeneous.

A new geometric combination of cutting and bleeding modules for surgical simulation systems

BACKGROUND AND OBJECTIVES

Cutting and bleeding are often independent of each other in the traditional virtual surgery system because of the differences in the calculation of physical models and the lack of internal structure. In order to improve the fidelity of virtual surgery scene and the training value for surgeons, a new geometric combination of cutting and bleeding modules is introduced.

METHODS

In this paper, we introduce a cutting model based on volume rendering and meshless method. The multidimensional parameters derived from the gray values are presented to participate in the calculation of both physical and geometric models, which distinguishes between different adjacent soft tissues. The bleeding simulation with improved physical properties and rendering algorithms of geometric model is proposed to meet several different bleeding states. After cutting procedures, the tearing parts can be judged through the vision and the tactile sensation. The initial velocity and rendering algorithm of bleeding particles are determined by the multidimensional parameters of the cutting position, which realizes the geometric combination of cutting and bleeding modules.

RESULTS AND CONCLUSIONS

Simulation results show that tearing different tissue structures will produce corresponding bleeding states. When the skin and flesh are torn, the blood is slowly generated at the incision, and then diffuses to the surface of soft tissue. When the important blood vessels are ruptured, the blood gushes from the laceration. Compared with the conventional virtual surgery system, both visual effect and interactivity of the cutting and bleeding modules are improved in the proposed geometric combination.

Statistical shape model-based planning organ-at-risk volume: application to pancreatic cancer patients

PURPOSE

To introduce the concept of statistical shape model (SSM)-based planning organ-at-risk volume (sPRV) for pancreatic cancer patients.

METHODS

A total of 120 pancreatic cancer patients were enrolled in this study. After correcting inter-patient variations in the centroid position of the planning target volume (PTV), four different SSMs were constructed by registering a deformable template model to an individual model for the stomach and duodenum. The sPRV, which focused on the following different components of the inter-patient variations, was then created: Scenario A: shape, rotational angle, volume, and centroid position; Scenario B: shape, rotational angle, and volume; Scenario C: shape and rotational angle; and Scenario D: shape. The conventional PRV (cPRV) was created by adding an isotropic margin R (3-15 mm) to the mean shape model. The corresponding sPRV was created from the SSM until the volume difference between the cPRV and sPRV was less than 1%. Thereafter, we computed the overlapping volume between the PTV and cPRV (OL_c) or sPRV (OL_s) in each patient. OL_s being larger than OL_c implies that the local shape variations in the corresponding OAR close to the PTV were large. Therefore, OL_s/OL_c was calculated in each patient for each R-value, and the median value of OL_s/OL_c was regarded as a surrogate for plan quality for each R-value.

RESULTS

For R = 3 and 5 mm, OL_s/OL_c exceeded 1 for the stomach and duodenum in all scenarios, with a maximum OL_s/OL_c of 1.21. This indicates that smaller isotropic margins did not sufficiently account for the local shape changes close to the PTV.

CONCLUSIONS

Our results indicated that, in contrast to conventional PRV, SSM-based PRVs, which account for local shape changes, would result in better dose sparing for the stomach and duodenum in pancreatic cancer patients.

Model-based registration for pneumothorax deformation analysis using intraoperative cone-beam CT images

Because the lung deforms during surgery because of pneumothorax, it is important to be able to track the location of a tumor. Deformation of the whole lung can be estimated using intraoperative cone-beam CT (CBCT) images. In this study, we used deformable mesh registration methods for paired CBCT images in the inflated and deflated states, and analyzed their deformation. We proposed a deformable mesh registration framework for deformations of partial organ shapes involving large deformation and rotation. Experimental results showed that the proposed methods reduced errors in point-to-point correspondence. As a result of registration using surgical clips placed on the lung surface during imaging, it was confirmed that an average error of 3.9 mm occurred in eight cases. The result of analysis showed that both tissue rotation and contraction had large effects on displacement.

Cutting procedures with improved visual effects and haptic interaction for surgical simulation systems

BACKGROUND AND OBJECTIVES

Surface rendering and physical models with constant parameters are often employed for cutting procedures in conventional surgical simulators. As a consequence, the internal structures of soft tissues cannot be rendered properly and haptic interaction is unrealistic. In order to improve both the visual and force feedback, a new volumetric geometric model is introduced.

METHODS

In this paper, we introduce a new volumetric geometric model, for which multidimensional parameters are derived from the gray values to map the color and transparency of the 3D soft tissues. In the meantime, the biomechanical properties of soft tissues are described by a meshless physical model and the model parameters are closely correlated to the multidimensional parameters of the developed volumetric geometric model. As a beneficial result, the force feedback changes according to the physical properties of different soft tissue structures, which reflects better the real-life scenarios during the course of cutting procedures.

RESULTS AND CONCLUSIONS

Simulation results show that both the surface and internal structures of soft tissues can be rendered properly and the boundaries between different tissue structures are visually distinct in incision. The curves of feedback force change according to the different structures of soft tissue, improving haptic interaction. Compared with the conventional cutting model, both visual effect and haptic interaction are improved in the proposed volumetric geometric model.

Performance evaluation of a newly developed three-dimensional model-based global-to-local registration in prostate cancer

We evaluated the performance of a newly developed three-dimensional (3D) model-based global-to-local registration of multiple organs, by comparing it with a 3D model-based global registration in the prostate region. This study included 220 prostate cancer patients who underwent intensity-modulated radiotherapy or volumetric-modulated arc therapy. Our registration proceeded sequentially, i.e. global registration including affine and piece-wise affine transformation followed by local registration. As a local registration, Laplacian-based and finite element method-based registration was implemented in Algorithm A and B, respectively. Algorithm C was for global registration alone. The template models for the prostate, seminal vesicles, rectum and bladder were constructed from the first 20 patients, and then three different registrations were performed on these organs for the remaining 200 patients, to assess registration accuracy. The 75th percentile Hausdorff distance was <1 mm in Algorithm A; it was >1 mm in Algorithm B, except for the prostate; and 3.9 mm for the prostate and >7.8 mm for other organs in Algorithm C. The median computation time to complete registration was <101, 30 and 16 s in Algorithms A, B and C, respectively. Analysis of variance revealed significant differences among Algorithms A-C in the Hausdorff distance and computation time. In addition, no significant difference was observed in the difference of Hausdorff distance between Algorithm A and B with Tukey's multiple comparison test. The 3D model-based global-to-local registration, especially that implementing Laplacian-based registration, completed surface registration rapidly and provided sufficient registration accuracy in the prostate region.

A new volumetric geometric model for cutting procedures in surgical simulation

BACKGROUND AND OBJECTIVES

Cutting procedures are the most common operations in surgical simulation. In order to provide realistic visual feedback with the details of the internal structures of soft tissue to the operator, a novel volumetric geometric model is presented for cutting procedures in surgical simulation.

METHODS

A novel volumetric geometric model, which is based on volume rendering and the Bézier curve, is presented for cutting procedures. The Bézier curve is used to optimize the physical model of cutting simulation, making the edge of incision smooth without increasing the computational load of the physical model. Volume rendering is used to render the cutting process, which improves significantly the realism of simulation since both surface textures and the details of the internal structures of soft tissues are rendered.

RESULTS AND CONCLUSIONS

The simulation results show that the edges of the incision optimized by using the proposed geometric model are smooth and the details of internal structures of soft tissue can be rendered. In comparison with other volumetric models, the computational efficiency is much improved. Compared with conventional cutting simulation methods, the proposed volumetric geometric model improves the effects of visual feedback since both surface and internal structures are rendered according to the optimized physical model.

Surface deformation analysis of collapsed lungs using model-based shape matching

Purpose

To facilitate intraoperative localization of lung nodules, this study used model-based shape matching techniques to analyze the inter-subject three-dimensional surface deformation induced by pneumothorax. Methods: Contrast- enhanced computed tomography (CT) images of the left lungs of 11 live beagle dogs were acquired at two bronchial pressures (14 and 2 cm\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\,\hbox {H}_2\hbox {O}$$\end{document}H2O). To address shape matching problems for largely deformed lung images with pixel intensity shift, a complete Laplacian-based shape matching solution that optimizes the differential displacement field was introduced.

Results

Experiments were performed to confirm the methods’ registration accuracy using CT images of lungs. Shape similarity and target displacement errors in the registered models were improved compared with those from existing shape matching methods. Spatial displacement of the whole lung’s surface was visualized with an average error of within 5 mm.

Conclusion

The proposed methods address problems with the matching of surfaces with large curvatures and deformations and achieved smaller registration errors than existing shape matching methods, even at the tip and ridge regions. The findings and inter-subject statistical representation are directly available for further research on pneumothorax deformation modeling.

Electronic supplementary material

The online version of this article (10.1007/s11548-019-02013-0) contains supplementary material, which is available to authorized users.

Elastic modulus estimation based on local displacement observation of elastic body

A method is proposed that provides estimates of the spatial variation of elastic moduli using local displacements of the elastic body. A central issue of elastography imaging has been the limited area of measurement. With the proposed method, stiffness parameter estimations are considered as minimization problems using finite-element models. The sparseness of the gradient of tissue elasticity is also exploited to improve estimation accuracy. Simulation experiments show that based on a 5% area of observation of a simple plate model with non-uniform elasticity the spatial variation of Young's modulus is reconstructed to within 5% accuracy. This result suggests that the proposed framework significantly extends the area of estimation overcoming the limitations of conventional elastography techniques.

A Novel Haptic Interactive Approach to Simulation of Surgery Cutting Based on Mesh and Meshless Models

In the present work, the majority of implemented virtual surgery simulation systems have been based on either a mesh or meshless strategy with regard to soft tissue modelling. To take full advantage of the mesh and meshless models, a novel coupled soft tissue cutting model is proposed. Specifically, the reconstructed virtual soft tissue consists of two essential components. One is associated with surface mesh that is convenient for surface rendering and the other with internal meshless point elements that is used to calculate the force feedback during cutting. To combine two components in a seamless way, virtual points are introduced. During the simulation of cutting, the Bezier curve is used to characterize smooth and vivid incision on the surface mesh. At the same time, the deformation of internal soft tissue caused by cutting operation can be treated as displacements of the internal point elements. Furthermore, we discussed and proved the stability and convergence of the proposed approach theoretically. The real biomechanical tests verified the validity of the introduced model. And the simulation experiments show that the proposed approach offers high computational efficiency and good visual effect, enabling cutting of soft tissue with high stability.

Continuous lung region segmentation from endoscopic images for intra-operative navigation

Although preoperative Computed tomography images are widely used in intraoperative navigation, they can not provide precise information for organs such as the lungs, which deform severely during surgery because of deflation. By segmenting lung regions using intraoperative endoscopic images, a more accurate navigation can be obtained because endoscopic images directly provide real-time organ descriptions. However, satisfactory segmentation is rarely achieved with the algorithms in the literature due to the high deformability of the lungs and similarity between the background and object. This article addresses these problems by describing a novel approach for lung region segmentation based on endoscopic images. The proposed method leverages both GrabCut and optical flow for continuous segmentation. It also introduces a novel technique for quick user interaction, in which users are required to quickly provide a rough curve that shows the possible area of the boundary, and then a much more precise segmentation is deduced based on the rough curve. The effectiveness of the proposed approach was demonstrated by comparing it with conventional algorithms. The results show that the average F-measure of the proposed method is more than 97%. The position, size, and boundary of the lungs obtained by the proposed method can provide useful intraoperative navigation for lung resection surgeries.

Deformable resection process map for intraoperative cutting guides

In this paper, we will introduce the concept of deformable resection process mapping as a time-varying geometric guide for soft tissue tumor resection procedures. The deformable resection process map (RPM) estimates the local appearance of vascular structures after cuts as a novel guide. The RPM can be directly generated from patient-specific medical images using volumetric resampling techniques. Since user input of some cutting points is the only requirement for generating the RPM, the developed software will be directly available for clinical use to preview surgical procedures and intraoperative workflow management without time-consuming setups or additional workloads. We used the CT images of the patients with hepatic cancer for the experiments. The performance of our method in the shape representation for a curved cut surface was compared with that of conventional shape modeling methods.

Study of a Ray Casting Technique for the Visualization of Deformable Volumes

Deformable models are widely used in many disciplines such as engineering and medicine. Real objects are usually scanned to create models in such applications. In many cases the shape of the object is extracted from volumetric data acquired during the scanning phase. At the same time, this volume can be used to define the model's appearance. In order to achieve a visualization that unifies the shape (physical model) and appearance (scanned volume) specially adapted volume rendering techniques are required. One of the most common volumetric visualization techniques is ray casting, which also enables the use of different corrections or improvements such as adaptive sampling or stochastic jittering. This paper presents an extensive study about a ray casting method for tetrahedral meshes with an underlying structured volume. This allows a direct visualization of the deformed model without losing the information contained in the volume. The aim of this study is to analyse and compare the different methods for ray traversal and illumination correction, resulting in a comprehensive relation of the different methods, their computational cost and visual performance.