A high-resolution model for soft tissue deformation based on point primitives

A deformation model of pulsating brain tissue for neurosurgery simulation

BACKGROUND AND OBJECTIVES

For neurological simulation, an accurate deformation model of brain tissue is of key importance for faithful visual feedback. Existing models, however, do not take into account intracranial pulsation, which degrades significantly the realism of visual feedback.

METHODS

In this paper, a finite element model incorporating intracranial pressure is proposed for simulating brain tissue deformation with pulsation. An implicit Euler method is developed to calculate the deformation of brain tissue. A circuit model of intracranial pressure dynamics is established based on cerebral blood and cerebrospinal fluid circulations. The intracranial pulsation of pressure is introduced into the deformation model, so that the simulated brain tissues pulsate with a rhythm in accord with the changes of intracranial pressure, which resembles real-life neurosurgery.

RESULTS AND CONCLUSIONS

The experimental implementation of the proposed deformation model and the calculation method shows that it provides realistic simulation of brain tissue pulsation and real-time performance is achieved on an ordinary computer for certain procedures of neurosurgery.

A new geometric combination of cutting and bleeding modules for surgical simulation systems

BACKGROUND AND OBJECTIVES

Cutting and bleeding are often independent of each other in the traditional virtual surgery system because of the differences in the calculation of physical models and the lack of internal structure. In order to improve the fidelity of virtual surgery scene and the training value for surgeons, a new geometric combination of cutting and bleeding modules is introduced.

METHODS

In this paper, we introduce a cutting model based on volume rendering and meshless method. The multidimensional parameters derived from the gray values are presented to participate in the calculation of both physical and geometric models, which distinguishes between different adjacent soft tissues. The bleeding simulation with improved physical properties and rendering algorithms of geometric model is proposed to meet several different bleeding states. After cutting procedures, the tearing parts can be judged through the vision and the tactile sensation. The initial velocity and rendering algorithm of bleeding particles are determined by the multidimensional parameters of the cutting position, which realizes the geometric combination of cutting and bleeding modules.

RESULTS AND CONCLUSIONS

Simulation results show that tearing different tissue structures will produce corresponding bleeding states. When the skin and flesh are torn, the blood is slowly generated at the incision, and then diffuses to the surface of soft tissue. When the important blood vessels are ruptured, the blood gushes from the laceration. Compared with the conventional virtual surgery system, both visual effect and interactivity of the cutting and bleeding modules are improved in the proposed geometric combination.

Fast and accurate nonlinear hyper-elastic deformation with a posteriori numerical verification of the convergence of solution: Application to the simulation of liver deformation

In this paper, we propose a new method to reduce the computational complexity of calculating the tangential stiffness matrix in a nonlinear finite element formulation. Our approach consists in partially updating the tangential stiffness matrix during a classic Newton-Raphson iterative process. The complexity of such an update process has the order of the number of mesh vertices to the power of two. With our approach, this complexity is reduced to the power of two of only the number of updated vertices. We numerically study the convergence of the solution with our modified algorithm. We describe the deformation through a strain energy density function which is defined with respect to the Lagrangian strain. We derive the conditions of convergence for a given tangential stiffness matrix and a given set of updated vertices. We use nonlinear geometric deformation and the nonlinear Mooney-Rivilin model with both tetrahedron and hexahedron element meshing. We provide extensive results using a cube with small and large number of elements. We provide results on nonlinearly deformed liver with multiple deformation ranges of updated vertices. We compare the proposed method to state-of-the-art work and we prove its efficiency at three levels: accuracy, speed of convergence and small radius of convergence.

Extended Kalman Filter Nonlinear Finite Element Method for Nonlinear Soft Tissue Deformation

BACKGROUND AND OBJECTIVE

Soft tissue modelling is crucial to surgery simulation. This paper introduces an innovative approach to realistic simulation of nonlinear deformation behaviours of biological soft tissues in real time.

METHODS

This approach combines the traditional nonlinear finite-element method (NFEM) and nonlinear Kalman filtering to address both physical fidelity and real-time performance for soft tissue modelling. It defines tissue mechanical deformation as a nonlinear filtering process for dynamic estimation of nonlinear deformation behaviours of biological tissues. Tissue mechanical deformation is discretized in space using NFEM in accordance with nonlinear elastic theory and in time using the central difference scheme to establish the nonlinear state-space models for dynamic filtering.

RESULTS

An extended Kalman filter is established to dynamically estimate nonlinear mechanical deformation of biological tissues. Interactive deformation of biological soft tissues with haptic feedback is accomplished as well for surgery simulation.

CONCLUSIONS

The proposed approach conquers the NFEM limitation of step computation but without trading off the modelling accuracy. It not only has a similar level of accuracy as NFEM, but also meets the real-time requirement for soft tissue modelling.

Deformation modeling based on mechanical properties of liver tissue for virtuanormal vectors of trianglesl surgical simulation

PURPOSE

In this paper, a method for rapidly constructing a virtual surgical simulation system is proposed. A deformation model based on the mechanical properties of the liver and a rapid collision detection between the surgical micro-instruments and the liver tissue are included in this method. The purpose of this work is to improve the accuracy and real time of particle model deformation interaction in virtual surgery system.

METHODS

Firstly, a finite element model is established based on the constitutive model parameters of liver tissue. According to the simulation results, a mathematical model of node displacement is established. Secondly, the virtual liver is established based on the fast model reconstruction method, and the virtual manipulator is controlled by Geomagic Touch manipulator. Based on the hybrid bounding box, a rapid collision detection process between the instrument and liver is realized and the proposed deformation method is used to simulate the deformation of liver tissue.

RESULTS

The simulation and experiment results show that the proposed deformation model can achieve high deformation interaction accuracy. The collision detection algorithm based on the hybrid bounding boxes can realize the collision between the liver and the instrument, and the established virtual surgical simulation system can simulate the liver tissue deformation in the case of small loading displacement.

CONCLUSIONS

The effectiveness of the collision detection algorithm and deformation model was verified by an established virtual surgery simulation system. The proposed rapid construction method of virtual surgical simulation is feasible.

A vortex method of 3D smoke simulation for virtual surgery

BACKGROUND AND OBJECTIVE

A realistic virtual surgery simulation needs to simulate smoke as electrical cutting causes thermal tissue damage. The vortex particle method of simulating smoke can realistically present the vortex details and motion trajectory of the smoke, but there is high computational cost.

METHODS

To address this problem, we propose the 3D Vortex Particles in Cube Algorithm (3D-VPICA). 3D-VPICA can realistically show the visual effect of smoke and reduce the computational cost. In addition, in order to enhance the reality of the smoke, we propose the Auxiliary Particles Algorithm (APA) method to deal with the collision problem of smoke.

RESULTS

The 3D-VPICA can calculate the velocity of the vortex particles speedily with the help of cube grids and with the complexity decreasing from O(N²) to O(N) + O(Mlog ₂M). The APA can ensure that boundary conditions are satisfied when the smoke collides with irregular surfaces. Experimental results show that 3D-VPICA is faster than traditional methods of smoke simulation and that APA is successful in simulating smoke colliding with moving objects with irregular surfaces.

CONCLUSIONS

The proposed 3D smoke simulation method was applied to a virtual surgery system using a high frequency electric knife. The cutting and coagulate operations were fluent and the smoke flowed with fidelity.

Modeling of connective tissue damage for blunt dissection of brain tumor in neurosurgery simulation

We introduce a new model for connective tissue damage in blunt dissection, which is a very important process in neurosurgery simulation. Specifically, the tool-tissue interaction between the instrument and connective tissue is incorporated into the model of connective tissue damage. This damage develops with the evolution criterion due to the effect of the external load. The tetrahedral mesh in the soft tissue model is removed for the representation of rupture as the damage accumulates to the threshold value. Analysis and experiments show that the connective tissue damage model provides stable, visually realistic results for the simulation of the connective tissue rupture process. The stiffness of the connective tissue decreases as the damage accumulates. The proposed model for connective tissue damage was incorporated into the development of a neurosurgery simulator, in which blunt dissection of a brain tumor was simulated.

Cutting procedures with improved visual effects and haptic interaction for surgical simulation systems

BACKGROUND AND OBJECTIVES

Surface rendering and physical models with constant parameters are often employed for cutting procedures in conventional surgical simulators. As a consequence, the internal structures of soft tissues cannot be rendered properly and haptic interaction is unrealistic. In order to improve both the visual and force feedback, a new volumetric geometric model is introduced.

METHODS

In this paper, we introduce a new volumetric geometric model, for which multidimensional parameters are derived from the gray values to map the color and transparency of the 3D soft tissues. In the meantime, the biomechanical properties of soft tissues are described by a meshless physical model and the model parameters are closely correlated to the multidimensional parameters of the developed volumetric geometric model. As a beneficial result, the force feedback changes according to the physical properties of different soft tissue structures, which reflects better the real-life scenarios during the course of cutting procedures.

RESULTS AND CONCLUSIONS

Simulation results show that both the surface and internal structures of soft tissues can be rendered properly and the boundaries between different tissue structures are visually distinct in incision. The curves of feedback force change according to the different structures of soft tissue, improving haptic interaction. Compared with the conventional cutting model, both visual effect and haptic interaction are improved in the proposed volumetric geometric model.

A Review on Mixed Reality: Current Trends, Challenges and Prospects

Currently, new technologies have enabled the design of smart applications that are used as decision-making tools in the problems of daily life. The key issue in designing such an application is the increasing level of user interaction. Mixed reality (MR) is an emerging technology that deals with maximum user interaction in the real world compared to other similar technologies. Developing an MR application is complicated, and depends on the different components that have been addressed in previous literature. In addition to the extraction of such components, a comprehensive study that presents a generic framework comprising all components required to develop MR applications needs to be performed. This review studies intensive research to obtain a comprehensive framework for MR applications. The suggested framework comprises five layers: the first layer considers system components; the second and third layers focus on architectural issues for component integration; the fourth layer is the application layer that executes the architecture; and the fifth layer is the user interface layer that enables user interaction. The merits of this study are as follows: this review can act as a proper resource for MR basic concepts, and it introduces MR development steps and analytical models, a simulation toolkit, system types, and architecture types, in addition to practical issues for stakeholders such as considering MR different domains.

A new volumetric geometric model for cutting procedures in surgical simulation

BACKGROUND AND OBJECTIVES

Cutting procedures are the most common operations in surgical simulation. In order to provide realistic visual feedback with the details of the internal structures of soft tissue to the operator, a novel volumetric geometric model is presented for cutting procedures in surgical simulation.

METHODS

A novel volumetric geometric model, which is based on volume rendering and the Bézier curve, is presented for cutting procedures. The Bézier curve is used to optimize the physical model of cutting simulation, making the edge of incision smooth without increasing the computational load of the physical model. Volume rendering is used to render the cutting process, which improves significantly the realism of simulation since both surface textures and the details of the internal structures of soft tissues are rendered.

RESULTS AND CONCLUSIONS

The simulation results show that the edges of the incision optimized by using the proposed geometric model are smooth and the details of internal structures of soft tissue can be rendered. In comparison with other volumetric models, the computational efficiency is much improved. Compared with conventional cutting simulation methods, the proposed volumetric geometric model improves the effects of visual feedback since both surface and internal structures are rendered according to the optimized physical model.

A new model of soft tissue with constraints for interactive surgical simulation

BACKGROUND AND OBJECTIVES

An accurate and real-time model of soft tissue is critical for surgical simulation for which a user interacts haptically and visually with simulated patients. This paper focuses on the real-time deformation model of brain tissue for the interactive surgical simulation, such as neurosurgical simulation.

METHODS

A new Finite Element Method (FEM) based model with constraints is proposed for the brain tissue in neurosurgical simulation. A new energy function of constraints characterizing the interaction between the virtual instrument and the soft tissue is incorporated into the optimization problem derived from the implicit integration scheme. Distance and permanent deformation constraints are introduced to describe the interaction in the convexity meningioma dissection and hemostasis. The proposed model is particularly suitable for GPU-based computing, making it possible to achieve real-time performance.

RESULTS AND CONCLUSIONS

Simulation results show that the simulated soft tissue exhibits the behaviors of adhesion and permanent deformation under the constraints. Experiments show that the proposed model is able to converge to the exact solution of the implicit Euler method after 96 iterations. The proposed model was implemented in the development of a neurosurgical simulator, in which surgical procedures such as dissection of convexity meningioma and hemostasis were simulated.

SLAM-based dense surface reconstruction in monocular Minimally Invasive Surgery and its application to Augmented Reality

BACKGROUND AND OBJECTIVE

While Minimally Invasive Surgery (MIS) offers considerable benefits to patients, it also imposes big challenges on a surgeon's performance due to well-known issues and restrictions associated with the field of view (FOV), hand-eye misalignment and disorientation, as well as the lack of stereoscopic depth perception in monocular endoscopy. Augmented Reality (AR) technology can help to overcome these limitations by augmenting the real scene with annotations, labels, tumour measurements or even a 3D reconstruction of anatomy structures at the target surgical locations. However, previous research attempts of using AR technology in monocular MIS surgical scenes have been mainly focused on the information overlay without addressing correct spatial calibrations, which could lead to incorrect localization of annotations and labels, and inaccurate depth cues and tumour measurements. In this paper, we present a novel intra-operative dense surface reconstruction framework that is capable of providing geometry information from only monocular MIS videos for geometry-aware AR applications such as site measurements and depth cues. We address a number of compelling issues in augmenting a scene for a monocular MIS environment, such as drifting and inaccurate planar mapping.

METHODS

A state-of-the-art Simultaneous Localization And Mapping (SLAM) algorithm used in robotics has been extended to deal with monocular MIS surgical scenes for reliable endoscopic camera tracking and salient point mapping. A robust global 3D surface reconstruction framework has been developed for building a dense surface using only unorganized sparse point clouds extracted from the SLAM. The 3D surface reconstruction framework employs the Moving Least Squares (MLS) smoothing algorithm and the Poisson surface reconstruction framework for real time processing of the point clouds data set. Finally, the 3D geometric information of the surgical scene allows better understanding and accurate placement AR augmentations based on a robust 3D calibration.

RESULTS

We demonstrate the clinical relevance of our proposed system through two examples: (a) measurement of the surface; (b) depth cues in monocular endoscopy. The performance and accuracy evaluations of the proposed framework consist of two steps. First, we have created a computer-generated endoscopy simulation video to quantify the accuracy of the camera tracking by comparing the results of the video camera tracking with the recorded ground-truth camera trajectories. The accuracy of the surface reconstruction is assessed by evaluating the Root Mean Square Distance (RMSD) of surface vertices of the reconstructed mesh with that of the ground truth 3D models. An error of 1.24 mm for the camera trajectories has been obtained and the RMSD for surface reconstruction is 2.54 mm, which compare favourably with previous approaches. Second, in vivo laparoscopic videos are used to examine the quality of accurate AR based annotation and measurement, and the creation of depth cues. These results show the potential promise of our geometry-aware AR technology to be used in MIS surgical scenes.

CONCLUSIONS

The results show that the new framework is robust and accurate in dealing with challenging situations such as the rapid endoscopy camera movements in monocular MIS scenes. Both camera tracking and surface reconstruction based on a sparse point cloud are effective and operated in real-time. This demonstrates the potential of our algorithm for accurate AR localization and depth augmentation with geometric cues and correct surface measurements in MIS with monocular endoscopes.

Mathematical Modeling and Virtual Reality Simulation of Surgical Tool Interactions With Soft Tissue: A Review and Prospective

This is an informed assessment of the state of the art and an extensive inventory of modeling approaches and methods for soft tissue/medical cutting tool interaction and of the associated medical processes and phenomena. Modeling and simulation through numerical, theoretical, computational, experimental, and other methods was discussed in comprehensive review sections each of which is concluded with a plausible prospective discussion biased toward the development of so-called virtual reality (VR) simulator environments. The finalized prospective section reflects on the future demands in the area of soft tissue cutting modeling and simulation mostly from a conceptual angle with emphasis on VR development requirements including real-time VR simulator response, cost-effective “close-to-reality” VR implementations, and other demands. The review sections that serve as the basis for the suggested prospective needs are categorized based on: (1) Major VR simulator applications including virtual surgery education, training, operation planning, intraoperative simulation, image-guided surgery, etc. and VR simulator types, e.g., generic, patient-specific and surgery-specific and (2) Available numerical, theoretical, and computational methods in terms of robustness, time effectiveness, computational cost, error control, and accuracy of modeling of certain types of virtual surgical interventions and their experimental validation, geared toward ethically driven artificial “phantom” tissue-based approaches. Digital data processing methods used in modeling of various feedback modalities in VR environments are also discussed.