Real-time finite-element simulation of linear viscoelastic tissue behavior based on experimental data

Perception of Soft Objects in Virtual Environments Under Conflicting Visual and Haptic Cues

In virtual/augmented/mixed reality (VR/AR/MR) applications, rendering soft virtual objects using a hand-held haptic device is challenging due to the anatomical restrictions of the hand and the ungrounded nature of the design, which affect the selection of actuators and sensors and hence limit the resolution and range of forces displayed by the device. We developed a cable-driven haptic device for rendering the net forces involved in grasping and squeezing 3D virtual compliant (soft) objects being held between the index finger and thumb only. Using the proposed device, we investigate the perception of soft objects in virtual environments. We show that the range of object stiffness that can be effectively conveyed to a user in virtual environments (VEs) can be significantly expanded by controlling the relationship between the visual and haptic cues. We propose that a single variable, named Apparent Stiffness Difference, can predict the pattern of human stiffness perception under manipulated conflict, which can be used for rendering a range of soft objects in VEs larger than what is achievable by a haptic device alone due to its physical limits.

Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Robot-assisted minimally invasive surgery has been broadly employed in complicated operations. However, the multiple surgical instruments may occupy a large amount of visual space in complex operations performed in narrow spaces, which affects the surgeon's judgment on the shape and position of the lesion as well as the course of its adjacent vessels/lacunae. In this paper, a surgical scene reconstruction method is proposed, which involves the tracking and removal of surgical instruments and the dynamic prediction of the obscured region. For tracking and segmentation of instruments, the image sequences are preprocessed by a modified U-Net architecture composed of a pre-trained ResNet101 encoder and a redesigned decoder. Also, the segmentation boundaries of the instrument shafts are extended using image filtering and a real-time index mask algorithm to achieve precise localization of the obscured elements. For predicting the deformation of soft tissues, a soft tissue deformation prediction algorithm is proposed based on dense optical flow gravitational field and entropy increase, which can achieve local dynamic visualization of the surgical scene by integrating image morphological operations. Finally, the preliminary experiments and the pre-clinical evaluation were presented to demonstrate the performance of the proposed method. The results show that the proposed method can provide the surgeon with a clean and comprehensive surgical scene, reconstruct the course of important vessels/lacunae, and avoid inadvertent injuries.

Cell-substrate mechanics guide collective cell migration through intercellular adhesion: a dynamic finite element cellular model

During the process of tissue formation and regeneration, cells migrate collectively while remaining connected through intercellular adhesions. However, the roles of cell-substrate and cell-cell mechanical interactions in regulating collective cell migration are still unclear. In this study, we employ a newly developed finite element cellular model to study collective cell migration by exploring the effects of mechanical feedback between cell and substrate and mechanical signal transmission between adjacent cells. Our viscoelastic model of cells consists many triangular elements and is of high resolution. Cadherin adhesion between cells is modeled explicitly as linear springs at subcellular level. In addition, we incorporate a mechano-chemical feedback loop between cell-substrate mechanics and Rac-mediated cell protrusion. Our model can reproduce a number of experimentally observed patterns of collective cell migration during wound healing, including cell migration persistence, separation distance between cell pairs and migration direction. Moreover, we demonstrate that cell protrusion determined by the cell-substrate mechanics plays an important role in guiding persistent and oriented collective cell migration. Furthermore, this guidance cue can be maintained and transmitted to submarginal cells of long distance through intercellular adhesions. Our study illustrates that our finite element cellular model can be employed to study broad problems of complex tissue in dynamic changes at subcellular level.

A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

Simulating deformations of soft tissues is a complex engineering task, and it is even more difficult when facing the constraint between computation speed and system accuracy. However, literature lacks of a holistic review of all necessary aspects (computational approaches, interaction devices, system architectures, and clinical validations) for developing an effective system of soft-tissue simulations. This paper summarizes and analyses recent achievements of resolving these issues to estimate general trends and weakness for future developments. A systematic review process was conducted using the PRISMA protocol with three reliable scientific search engines (ScienceDirect, PubMed, and IEEE). Fifty-five relevant papers were finally selected and included into the review process, and a quality assessment procedure was also performed on them. The computational approaches were categorized into mesh, meshfree, and hybrid approaches. The interaction devices concerned about combination between virtual surgical instruments and force-feedback devices, 3D scanners, biomechanical sensors, human interface devices, 3D viewers, and 2D/3D optical cameras. System architectures were analysed based on the concepts of system execution schemes and system frameworks. In particular, system execution schemes included distribution-based, multithread-based, and multimodel-based executions. System frameworks are grouped into the input and output interaction frameworks, the graphic interaction frameworks, the modelling frameworks, and the hybrid frameworks. Clinical validation procedures are ordered as three levels: geometrical validation, model behavior validation, and user acceptability/safety validation. The present review paper provides useful information to characterize how real-time medical simulation systems with soft-tissue deformations have been developed. By clearly analysing advantages and drawbacks in each system development aspect, this review can be used as a reference guideline for developing systems of soft-tissue simulations.

Computational model of the mechanoelectrophysiological coupling in axons with application to neuromodulation

For more than half a century, the action potential (AP) has been considered a purely electrical phenomenon. However, experimental observations of membrane deformations occurring during APs have revealed that this process also involves mechanical features. This discovery has recently fuelled a controversy on the real nature of APs: whether they are mechanical or electrical. In order to examine some of the modern hypotheses regarding APs, we propose here a coupled mechanoelectrophysiological membrane finite-element model for neuronal axons. The axon is modeled as an axisymmetric thin-wall cylindrical tube. The electrophysiology of the membrane is modeled using the classic Hodgkin-Huxley (H-H) equations for the Nodes of Ranvier or unmyelinated axons and the cable theory for the internodal regions, whereas the axonal mechanics is modeled by means of viscoelasticity theory. Membrane potential changes induce a strain gradient field via reverse flexoelectricity, whereas mechanical pulses result in an electrical self-polarization field following the direct flexoelectric effect, in turn influencing the membrane potential. Moreover, membrane deformation also alters the values of membrane capacitance and resistance in the H-H equation. These three effects serve as the fundamental coupling mechanisms between the APs and mechanical pulses in the model. A series of numerical studies was systematically conducted to investigate the consequences of interaction between the APs and mechanical waves on both myelinated and unmyelinated axons. Simulation results illustrate that the AP is always accompanied by an in-phase propagating membrane displacement of ≈1nm, whereas mechanical pulses with enough magnitude can also trigger APs. The model demonstrates that mechanical vibrations, such as the ones arising from ultrasound stimulations, can either annihilate or enhance axonal electrophysiology depending on their respective directionality and frequency. It also shows that frequency of pulse repetition can also enhance signal propagation independently of the amplitude of the signal. This result not only reconciles the mechanical and electrical natures of the APs but also provides an explanation for the experimentally observed mechanoelectrophysiological phenomena in axons, especially in the context of ultrasound neuromodulation.

Estimation of Object Elasticity by Capturing Fingernail Images During Haptic Palpation

In this study, we present a system that performs natural-touch-based elasticity estimation for an object by using a depth camera. To estimate elasticity, which is defined as an object's Young's modulus, a strain-stress curve is obtained from fingernail images during haptic palpation. From a color image, the proposed system detects a fingernail and extracts 10 feature values related to the contact force; then, it estimates the force using a multiple regression model. Deformation of the object was estimated from the finger's three-dimensional position obtained from both color and depth images. Then, a strain-stress curve was determined using the force and deformation data. Evaluation experiments were designed to obtain the strain-stress curves of five objects from 10 participants; then, the estimation performance was investigated. The results show that the reliable range of sensing was within Young's modulus values of 0.12-5.6 MPa and the precision of the measurement was 55 percent of the estimated elasticity.

Comparison between isotropic linear-elastic law and isotropic hyperelastic law in the finite element modeling of the brachial plexus

Augmented reality could help the identification of nerve structures in brachial plexus surgery. The goal of this study was to determine which law of mechanical behavior was more adapted by comparing the results of Hooke's isotropic linear elastic law to those of Ogden's isotropic hyperelastic law, applied to a biomechanical model of the brachial plexus. A model of finite elements was created using the ABAQUS^® from a 3D model of the brachial plexus acquired by segmentation and meshing of MRI images at 0°, 45° and 135° of shoulder abduction of a healthy subject. The offset between the reconstructed model and the deformed model was evaluated quantitatively by the Hausdorff distance and qualitatively by the identification of 3 anatomical landmarks. In every case the Hausdorff distance was shorter with Ogden's law compared to Hooke's law. On a qualitative aspect, the model deformed by Ogden's law followed the concavity of the reconstructed model whereas the model deformed by Hooke's law remained convex. In conclusion, the results of this study demonstrate that the behavior of Ogden's isotropic hyperelastic mechanical model was more adapted to the modeling of the deformations of the brachial plexus.

On the Transparency of Client/Server-Based Haptic Interaction with Deformable Objects

This paper studies the transparency of client/server-based haptic interaction with simulated deformable objects. In the considered remote interaction scenario, the server simulates the computationally expensive finite-element-based object deformation at a low temporal update rate and transmits the result to the clients. There, the received deformation data is applied to the polygon mesh, which is used to locally render force feedback with a penalty-based force rendering algorithm at the required high rate. Based on a one-dimensional deformable object example, we analyze the transparency of this multi-rate architecture for a two-user interaction. Communication delay leads to increased force magnitudes and an increased impedance displayed to the clients that actively interact with the object. We propose a method that adjusts the stiffness used in the local force rendering at the clients to compensate for this effect. The conducted objective and subjective evaluations show that the proposed method successfully compensates for the effect of communication delay in the tested delay range of up to 100 ms.

Real-Time Visual Tracking of Deformable Objects in Robot-Assisted Surgery

One of the most challenging problems in robot-assisted surgical systems is to provide surgical realism at interactive simulation rates. The proposed visual tracking system can track and register object deformations in real time using a physically based formulation, despite the occlusions produced by the robotic system itself. The results obtained provide an accurate visual representation of the deformed solid and will thus enable new assistance approaches to help surgeons during surgical procedures.

Ramp-hold relaxation solutions for the KVFD model applied to soft viscoelastic media

The standard step-hold load-relaxation profile can yield variable estimates of mechanical properties due to the difficulty in achieving a step strain experimentally. A ramp-hold profile overcomes this limitation if appropriate model functions can be derived. Utilizing Boltzmann hereditary integral operators for two indentation geometries, analytical ramp solutions for load-relaxation were developed based on the Kelvin-Voigt fractional derivative (KVFD) model. The results identify three model parameters for characterizing viscoelastic behavior from a single model curve fit to the data: the elastic modulus E₀, fractional-order parameter α, and relaxation time constant τ. The quantitative nature of the analysis was validated through measurements on gelatin emulsion samples exhibiting viscoelastic behavior. KVFD-model-based solutions provide mathematically simple and experimentally flexible descriptions of load-relaxation behavior for a range of viscoelastic properties and experimental conditions; e.g. one closed-form solution can fit the ramp and the hold phases of the relaxation time series. Experiments show that the solution for a spherical indenter and plate compressor each fit well to the corresponding experimental relaxation curves with a coefficient of determination R² > 0.98. Parameters obtained from the spherical-tip indentation and plate-compression geometries agree within one standard deviation, confirming that the ramp solution based KVFD model yields consistent measurements for characterizing viscoelastic materials.

Quasi-non-linear deformation modeling of a human liver based on artificial and experimental data

BACKGROUND

Researchers working on error-prevention theories have shown that the use of replica models within simulation systems has improved operating skills, resulting in better patient outcomes.

METHODS

This study aims to provide material test data specifically for a human liver to validate the accuracy of viscoelastic soft tissue models. This allows the validation of virtual surgery simulators by comparison with physical test data obtained from material tests on a viscoelastic silicone gel pad.

RESULTS

The results proved that stress behavior and relaxation curves of Aquaflex® experiment and FEM simulation are close if average liver response and respective material parameters and model are used.

CONCLUSIONS

The precise representation of manipulated tissues used in virtual surgery trainers involves the accurate characterization of mechanical properties of the tissue. Consequently, successful implementations of these mechanical properties in a mathematical model of the deforming organ are of major importance. Copyright © 2015 John Wiley & Sons, Ltd.

Distributed haptic interactions with physically based 3D deformable models over lossy networks

Researchers have faced great challenges when simulating complicated 3D volumetric deformable models in haptics-enabled collaborative/cooperative virtual environments (HCVEs) due to the expensive simulation cost, heavy communication load, and unstable network conditions. When general network services are applied to HCVEs, network problems such as packet loss, delay, and jitter can cause severe visual distortion, haptic instability, and system inconsistency. In this paper, we propose a novel approach to support haptic interactions with physically based 3D deformable models in a distributed virtual environment. Our objective is to achieve real-time sharing of deformable and force simulations over general networks. Combining linear modal analysis and corotational methods, we can effectively simulate physical behaviors of 3D objects, even for large rotational deformations. We analyze different factors that influence HCVEs' performance and focus on exploring solutions for streaming over lossy networks. In our system, 3D deformation can be described by a fairly small amount of data (several KB) using accelerations in the spectral domain, so that we can achieve low communication load and effective streaming. We develop a loss compensation and prediction algorithm to correct the errors/distortions caused by network problem, and a force prediction method to simulate force at users' side to ensure the haptic stability, and the visual and haptic consistency. Our system works well under both the client-server and the peer-to-peer distribution structures, and can be easily extended to other topologies. In addition to theoretical analysis, we have tested the proposed system and algorithms under various network conditions. The experimental results are remarkably good, confirming the effectiveness, robustness, and validity of our approach.

Difference of perceiving object softness during palpation through single-node and multi-node contacts

Virtual Reality (VR) simulators can offer alternatives for training procedures in the medical field. Most current VR simulators consider single-node contact for interacting with an object to convey displacement and force on a discrete mesh. However, a single-node contact does not closely simulate palpation, which requires a surface made of a multi-node contact to touch a soft object. Thus, we hypothesize that the softness of a deformable object (such as a virtual breast phantom) palpated through a single-node contact would be perceived differently from that of the same phantom palpated through a multi-node contact with various force arrays. We conducted a study to investigate this hypothesis. Using a co-located VR setup that aligns visual and haptic stimuli onto a spatial location, we tested 15 human participants under conditions of both visual and haptic stimuli available and only visual (or haptic) stimulus available. In a trial, each participant palpated and discriminated two virtual breast phantoms of same softness through different contacts with varying force arrays. The results of this study revealed that virtual breast phantoms palpated through a single-node contact were constantly perceived harder than their counterparts palpated through a multi-node contact with varying force arrays, when visual stimuli were available. These results imply a constraint for developing a VR system of training palpation.

Constraint-Based Haptic Rendering of Multirate Compliant Mechanisms

The paper is dedicated to haptic rendering of complex physics-based environment in the context of surgical simulation. A new unified formalism for modeling the mechanical interactions between medical devices and anatomical structures and for computing accurately the haptic force feedback is presented. The approach deals with the mechanical interactions using appropriate force and/or motion transmission models named compliant mechanisms. These mechanisms are formulated as a constraint-based problem that is solved in two separate threads running at different frequencies. The first thread processes the whole simulation including the soft-tissue deformations, whereas the second one only deals with computer haptics. This method builds a bridge between the so-called virtual mechanisms (that were proposed for haptic rendering of rigid bodies) and intermediate representations (used for rendering of complex simulations). With this approach, it is possible to describe the specific behavior of various medical devices while relying on a unified method for solving the mechanical interactions between deformable objects and haptic rendering. The technique is demonstrated in interactive simulation of flexible needle insertion through soft anatomical structures with force feedback.

Dual-channel Haptic Synthesis of Viscoelastic Tissue Properties using Programmable Eddy Current Brakes

We describe a novel method for haptic synthesis of viscoelastic responses which employs a dual-channel haptic interface. It has motors that generate torque independently of velocity and brakes that generate viscous torque independently of position. In this way, twice as many states are directly accessible, which reduces reliance on observation and feedback. Torque-generating actuators, e.g. DC motors, are well known. For the viscous actuators, we use eddy current brakes as programmable, linear, non-contact, physical dampers. By decomposing a mechanical impedance to be realized into viscous and elastic components, we can dedicate each actuator to that component it is ideally suited to synthesize, dampers for the viscous component, and motors for the elastic component. The decomposition is in general not unique so it is possible to select the option that takes the best advantage of the hardware. Experimental results show that this technique can render a variety of viscoelastic models without the artifacts that can occur when synthesizing viscous components on conventional haptic interfaces. The synthesized mechanical impedances have guaranteed passivity, and can have arbitrarily high or low viscous and elastic components.

Sensing, Acquisition, and Interactive Playback of Data-based Models for Elastic Deformable Objects

In this paper we describe the design and implementation of an automated system to build models of elastic deformable objects and to render these models in an interactive virtual environment. By automating model creation, a greater number of models and models that are more realistic can be included in these interactive simulations. Model geometry is acquired by a novel range sensor. Techniques for contact detection, slip detection on deformable surfaces, and enhanced open-loop force control enable haptic model acquisition through active probing with a haptic device. For model rendering, the force field is approximated from these measured samples.

A Comparison of Some Model Order Reduction Methods for Fast Simulation of Soft Tissue Response using the Point Collocation-based Method of Finite Spheres (PCMFS)

In this paper we develop the Point Collocation-based Method of Finite Spheres (PCMFS) to simulate the viscoelastic response of soft biological tissues and evaluate the effectiveness of model order reduction methods such as modal truncation, Hankel optimal model and truncated balanced realization techniques for PCMFS. The PCMFS was developed in [1] as a physics-based technique for real time simulation of surgical procedures. It is a meshfree numerical method in which discretization is performed using a set of nodal points with approximation functions compactly supported on spherical subdomains centered at the nodes. The point collocation method is used as the weighted residual technique where the governing differential equations are directly applied at the nodal points. Since computational speed has a significant role in simulation of surgical procedures, model order reduction methods have been compared for relative gains in efficiency and computational accuracy. Of these methods, truncated balanced realization results in the highest accuracy while modal truncation results in the highest efficiency.

A robotic indenter for minimally invasive measurement and characterization of soft tissue response

The lack of experimental data in current literature on material properties of soft tissues in living condition has been a significant obstacle in the development of realistic soft tissue models for virtual reality based surgical simulators used in medical training. A robotic indenter was developed for minimally invasive measurement of soft tissue properties in abdominal region during a laparoscopic surgery. Using the robotic indenter, force versus displacement and force versus time responses of pig liver under static and dynamic loading conditions were successfully measured to characterize its material properties in three consecutive steps. First, the effective elastic modulus of pig liver was estimated as 10-15 kPa from the force versus displacement data of static indentations based on the small deformation assumption. Then, the stress relaxation function, relating the variation of stress with respect to time, was determined from the force versus time response data via curve fitting. Finally, an inverse finite element solution was developed using ANSYS finite element package to estimate the optimum values of viscoelastic and nonlinear hyperelastic material properties of pig liver through iterations. The initial estimates of the material properties for the iterations were extracted from the experimental data for faster convergence of the solutions.

VR-based simulators for training in minimally invasive surgery

Simulation-based training using VR techniques is a promising alternative to traditional training in minimally invasive surgery (MIS). Simulators let the trainee touch, feel, and manipulate virtual tissues and organs through the same surgical tool handles used in actual MIS while viewing images of tool-tissue interactions on a monitor as in real laparoscopic procedures.