A Systematic Review of Real-Time Medical Simulations with Soft-Tissue Deformation: Computational Approaches, Interaction Devices, System Architectures, and Clinical Validations

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Immersive technologies have thrived on a strong foundation of software and hardware, injecting vitality into medical training. This surge has witnessed numerous endeavors incorporating immersive technologies into surgery simulation for surgical skills training, with a growing number of researchers delving into this domain. Relevant experiences and patterns need to be summarized urgently to enable researchers to establish a comprehensive understanding of this field, thus promoting its continuous growth. This study provides a forward-looking perspective by reviewing the latest development of immersive interactive technologies for surgery simulation. The investigation commences from a technological standpoint, delving into the core aspects of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, namely, haptic rendering and tracking. Subsequently, we summarize recent work based on the categorization of minimally invasive surgery (MIS) and open surgery simulations. Finally, the study showcases the impressive performance and expansive potential of immersive technologies in surgical simulation while also discussing the current limitations. We find that the design of interaction and the choice of immersive technology in virtual surgery development should be closely related to the corresponding interactive operations in the real surgical speciality. This alignment facilitates targeted technological adaptations in the direction of greater applicability and fidelity of simulation.

Clean visual field reconstruction in robot-assisted laparoscopic surgery based on dynamic prediction

Robot-assisted minimally invasive surgery has been broadly employed in complicated operations. However, the multiple surgical instruments may occupy a large amount of visual space in complex operations performed in narrow spaces, which affects the surgeon's judgment on the shape and position of the lesion as well as the course of its adjacent vessels/lacunae. In this paper, a surgical scene reconstruction method is proposed, which involves the tracking and removal of surgical instruments and the dynamic prediction of the obscured region. For tracking and segmentation of instruments, the image sequences are preprocessed by a modified U-Net architecture composed of a pre-trained ResNet101 encoder and a redesigned decoder. Also, the segmentation boundaries of the instrument shafts are extended using image filtering and a real-time index mask algorithm to achieve precise localization of the obscured elements. For predicting the deformation of soft tissues, a soft tissue deformation prediction algorithm is proposed based on dense optical flow gravitational field and entropy increase, which can achieve local dynamic visualization of the surgical scene by integrating image morphological operations. Finally, the preliminary experiments and the pre-clinical evaluation were presented to demonstrate the performance of the proposed method. The results show that the proposed method can provide the surgeon with a clean and comprehensive surgical scene, reconstruct the course of important vessels/lacunae, and avoid inadvertent injuries.

HyperMSM: A new MSM variant for efficient simulation of dynamic soft-tissue deformations

BACKGROUND AND OBJECTIVE

Fast, accurate, and stable simulation of soft tissue deformation is a challenging task. Mass-Spring Model (MSM) is one of the popular methods used for this purpose for its simple implementation and potential to provide fast dynamic simulations. However, accurately simulating a non-linear material within the mass-spring framework is still challenging. The objective of the present study is to develop and evaluate a new efficient hyperelastic Mass-Spring Model formulation to simulate the Neo-Hookean deformable material, called HyperMSM.

METHODS

Our novel HyperMSM formulation is applicable for both tetrahedral and hexahedral mesh configurations and is compatible with the original projective dynamics solver. In particular, the proposed MSM variant includes springs with variable rest-lengths and a volume conservation constraint. Two applications (transtibial residual limb and the skeletal muscle) were conducted.

RESULTS

Compared to finite element simulations, obtained results show RMSE ranges of [2.8%-5.2%] and [0.46%-5.4%] for stress-strain and volumetric responses respectively for strains ranging from -50% to +100%. The displacement error range in our transtibial residual limb simulation is around [0.01mm-0.7 mm]. The RMSE range of relative nodal displacements for the skeletal psoas muscle model is [0.4%-1.7%].

CONCLUSIONS

Our novel HyperMSM formulation allows hyperelastic behavior of soft tissues to be described accurately and efficiently within the mass-spring framework. As perspectives, our formulation will be enhanced with electric behavior toward a multi-physical soft tissue mass-spring modeling framework. Then, the coupling with an augmented reality environment will be performed.

Soft Tissue Hybrid Model for Real-Time Simulations

In this article, a recent formulation for real-time simulation is developed combining the strain energy density of the Spring Mass Model (SMM) with the equivalent representation of the Strain Energy Density Function (SEDF). The resulting Equivalent Energy Spring Model (EESM) is expected to provide information in real-time about the mechanical response of soft tissue when subjected to uniaxial deformations. The proposed model represents a variation of the SMM and can be used to predict the mechanical behavior of biological tissues not only during loading but also during unloading deformation states. To assess the accuracy achieved by the EESM, experimental data was collected from liver porcine samples via uniaxial loading and unloading tensile tests. Validation of the model through numerical predictions achieved a refresh rate of 31 fps (31.49 ms of computation time for each frame), achieving a coefficient of determination R² from 93.23% to 99.94% when compared to experimental data. The proposed hybrid formulation to characterize soft tissue mechanical behavior is fast enough for real-time simulation and captures the soft material nonlinear virgin and stress-softened effects with high accuracy.

Kinect-driven Patient-specific Head, Skull, and Muscle Network Modelling for Facial Palsy Patients

BACKGROUND AND OBJECTIVE

Facial palsy negatively affects both professional and personal life qualities of involved patients. Classical facial rehabilitation strategies can recover facial mimics into their normal and symmetrical movements and appearances. However, there is a lack of objective, quantitative, and in-vivo facial texture and muscle activation bio-feedbacks for personalizing rehabilitation programs and diagnosing recovering progresses. Consequently, this study proposed a novel patient-specific modelling method for generating a full patient specific head model from a visual sensor and then computing the facial texture and muscle activation in real-time for further clinical decision making.

METHODS

The modeling workflow includes (1) Kinect-to-head, (2) head-to-skull, and (3) muscle network definition & generation processes. In the Kinect-to-head process, subject-specific data acquired from a new user in neutral mimic were used for generating his/her geometrical head model with facial texture. In particular, a template head model was deformed to optimally fit with high-definition facial points acquired by the Kinect sensor. Moreover, the facial texture was also merged from his/her facial images in left, right, and center points of view. In the head-to-skull process, a generic skull model was deformed so that its shape was statistically fitted with his/her geometrical head model. In the muscle network definition & generation process, a muscle network was defined from the head and skull models for computing muscle strains during facial movements. Muscle insertion points and muscle attachment points were defined as vertex positions on the head model and the skull model respectively based on the standard facial anatomy. Three healthy subjects and two facial palsy patients were selected for validating the proposed method. In neutral positions, magnetic resonance imaging (MRI)-based head and skull models were compared with Kinect-based head and skull models. In mimic positions, infrared depth-based head models in smiling and [u]-pronouncing mimics were compared with appropriate animated Kinect-driven head models. The Hausdorff distance metric was used for these comparisons. Moreover, computed muscle lengths and strains in the tested facial mimics were validated with reported values in literature.

RESULTS

With the current hardware configuration, the patient-specific head model with skull and muscle network could be fast generated within 17.16±0.37s and animated in real-time with the framerate of 40 fps. In neutral positions, the best mean error was 1.91 mm for the head models and 3.21 mm for the skull models. On facial regions, the best mean errors were 1.53 mm and 2.82 mm for head and skull models respectively. On muscle insertion/attachment point regions, the best mean errors were 1.09 mm and 2.16 mm for head and skull models respectively. In mimic positions, these errors were 2.02 mm in smiling mimics and 2.00 mm in [u]-pronouncing mimics for the head models on facial regions. All above error values were computed on a one-time validation procedure. Facial muscles exhibited muscle shortening and muscle elongating for smiling and pronunciation of sound [u] respectively. Extracted muscle features (i.e. muscle length and strain) are in agreement with experimental and literature data.

CONCLUSIONS

This study proposed a novel modeling method for fast generating and animating patient-specific biomechanical head model with facial texture and muscle activation bio-feedbacks. The Kinect-driven muscle strains could be applied for further real-time muscle-oriented facial paralysis grading and other facial analysis applications.