Using the PhysX engine for physics-based virtual surgery with force feedback

A Survey on Open-Source Simulation Platforms for Multi-Copter UAV Swarms

Simulation platforms are critical and indispensable tools for application developments of unmanned aerial vehicles (UAVs) because the UAVs are generally costly, have certain requirements for the test environment, and need professional licensed operators. Thus, developers prefer (or have) to test their applications on simulation platforms before implementing them on real machines. In the past decades, a considerable number of simulation platforms for robots have been developed, which brings convenience to developers, but also makes them hard to choose a proper one as they are not always familiar with all the features of platforms. To alleviate this dilemma, this paper provides a survey of open-source simulation platforms and employs the simulation of a multi-copter UAV swarm as an example. The survey covers seven widely used simulators, including Webots, Gazebo, CoppeliaSim, ARGoS, MRDS, MORSE, and USARSim. The paper outlines the requirements for multi-copter UAV swarms and shows how to select an appropriate platform. Additionally, the paper presents a case study of a UAV swarm based on Webots. This research will be beneficial to researchers, developers, educators, and engineers who seek suitable simulation platforms for application development, (not only multi-copter UAV swarms but also other types of robots), which further helps them to save expenses for testing, and speed up development progress.

Integration of Reinforcement Learning in a Virtual Robotic Surgical Simulation

Background. The revolutions in AI hold tremendous capacity to augment human achievements in surgery, but robust integration of deep learning algorithms with high-fidelity surgical simulation remains a challenge. We present a novel application of reinforcement learning (RL) for automating surgical maneuvers in a graphical simulation.Methods. In the Unity3D game engine, the Machine Learning-Agents package was integrated with the NVIDIA FleX particle simulator for developing autonomously behaving RL-trained scissors. Proximal Policy Optimization (PPO) was used to reward movements and desired behavior such as movement along desired trajectory and optimized cutting maneuvers along the deformable tissue-like object. Constant and proportional reward functions were tested, and TensorFlow analytics was used to informed hyperparameter tuning and evaluate performance.Results. RL-trained scissors reliably manipulated the rendered tissue that was simulated with soft-tissue properties. A desirable trajectory of the autonomously behaving scissors was achieved along 1 axis. Proportional rewards performed better compared to constant rewards. Cumulative reward and PPO metrics did not consistently improve across RL-trained scissors in the setting for movement across 2 axes (horizontal and depth).Conclusion. Game engines hold promising potential for the design and implementation of RL-based solutions to simulated surgical subtasks. Task completion was sufficiently achieved in one-dimensional movement in simulations with and without tissue-rendering. Further work is needed to optimize network architecture and parameter tuning for increasing complexity.

Construction of force haptic reappearance system based on Geomagic Touch haptic device

Force haptic reappearance technology is considered to be one of the top ten technologies that can change human life in the future. It has broad application prospects and market demand. Most of the existing medical robots, especially the remote diagnosis and treatment robots, lack haptic feedback, or the calculation of feedback force is insufficient. Haptic reappearance technology is an effective method to solve the problem of haptic presence and improve the practicability of medical robot. The ultimate goal of the force haptic reappearance system is to let the operator feel the haptic feedback when interacting with the soft tissue model in the virtual environment in real time. Haptic device is the necessary condition to realize force haptic reappearance, and it is an essential part of the system. Its important role is to introduce the external force imposed by the operator into the virtual environment, and let the operator feel the force in the virtual environment, which effectively guarantees the operator's sense of reality and immersion when interacting with the virtual environment. Therefore, starting with the key technology of force and haptic reappearance system, this paper studies the construction of force and haptic reappearance system. Soft tissue surface model is drawn by OpenGL, and hand model is drawn by 3Ds Max. The haptic reappearance and visual feedback of soft tissue model of hand palpation are realized. The quality of feedback is evaluated. The haptic reappearance is stable and realistic, and the visual feedback is smooth. This indicates that the system has a certain application value and is worth to promote to the public.

Virtual reality simulation of robotic transsphenoidal brain tumor resection: Evaluating dynamic motion scaling in a master-slave system

Background

Integrating simulators with robotic surgical procedures could assist in designing and testing of novel robotic control algorithms and further enhance patient‐specific pre‐operative planning and training for robotic surgeries.

Methods

A virtual reality simulator, developed to perform the transsphenoidal resection of pituitary gland tumours, tested the usability of robotic interfaces and control algorithms. It used position‐based dynamics to allow soft‐tissue deformation and resection with haptic feedback; dynamic motion scaling control was also incorporated into the simulator.

Results

Neurosurgeons and residents performed the surgery under constant and dynamic motion scaling conditions (CMS vs DMS). DMS increased dexterity and reduced the risk of damage to healthy brain tissue. Post‐experimental questionnaires indicated that the system was well‐evaluated by experts.

Conclusion

The simulator was intuitively and realistically operated. It increased the safety and accuracy of the procedure without affecting intervention time. Future research can investigate incorporating this simulation into a real micro‐surgical robotic system.

Prevalence of haptic feedback in robot-mediated surgery: a systematic review of literature

With the successful uptake and inclusion of robotic systems in minimally invasive surgery and with the increasing application of robotic surgery (RS) in numerous surgical specialities worldwide, there is now a need to develop and enhance the technology further. One such improvement is the implementation and amalgamation of haptic feedback technology into RS which will permit the operating surgeon on the console to receive haptic information on the type of tissue being operated on. The main advantage of using this is to allow the operating surgeon to feel and control the amount of force applied to different tissues during surgery thus minimising the risk of tissue damage due to both the direct and indirect effects of excessive tissue force or tension being applied during RS. We performed a two-rater systematic review to identify the latest developments and potential avenues of improving technology in the application and implementation of haptic feedback technology to the operating surgeon on the console during RS. This review provides a summary of technological enhancements in RS, considering different stages of work, from proof of concept to cadaver tissue testing, surgery in animals, and finally real implementation in surgical practice. We identify that at the time of this review, while there is a unanimous agreement regarding need for haptic and tactile feedback, there are no solutions or products available that address this need. There is a scope and need for new developments in haptic augmentation for robot-mediated surgery with the aim of improving patient care and robotic surgical technology further.

Efficient Surgical Cutting with Position-Based Dynamics

Simulations of cuts on deformable bodies have been an active research subject for more than two decades. However, previous works based on finite element methods and mass spring meshes cannot scale to complex surgical scenarios. This article presents a novel method that uses position-based dynamics (PBD) for mesh-free cutting simulation. The proposed solutions include a method to efficiently render force feedback while cutting, an efficient heat diffusion model to simulate electrocautery, and a novel adaptive skinning scheme based on oriented particles.https://extras.computer.org/extra/mcg2017030024s1.mp4.

Real-time GPU-based ultrasound simulation using deformable mesh models

This paper presents a real-time capable graphics processing unit (GPU)-based ultrasound simulator suitable for medical education. The main focus of the simulator is to synthesize realistic looking ultrasound images in real-time including artifacts, which are essential for the interpretation of this data. The simulation is based on a convolution-enhanced ray-tracing approach and uses a deformable mesh model. Deformations of the mesh model are calculated using the PhysX engine. Our method advances the state of the art for real-time capable ultrasound simulators by following the path of the ultrasound pulse, which enables better simulation of ultrasound-specific artifacts. An evaluation of our proposed method in comparison with recent generative slicing-based strategies as well as real ultrasound images is performed. Hereby, a gelatin ultrasound phantom containing syringes filled with different media is scanned with a real transducer. The obtained images are then compared to images which are simulated using a slicing-based technique and our proposed method. The particular benefit of our method is the accurate simulation of ultrasound-specific artifacts, like range distortion, refraction and acoustic shadowing. Several test scenarios are evaluated regarding simulation time, to show the performance and the bottleneck of our method. While being computationally more intensive than slicing techniques, our simulator is able to produce high-quality images in real-time, tracing over 5000 rays through mesh models with more than 2 000 000 triangles of which up to 200 000 may be deformed each frame.

Surgical model-view-controller simulation software framework for local and collaborative applications

PURPOSE

Surgical simulations require haptic interactions and collaboration in a shared virtual environment. A software framework for decoupled surgical simulation based on a multi-controller and multi-viewer model-view-controller (MVC) pattern was developed and tested.

METHODS

A software framework for multimodal virtual environments was designed, supporting both visual interactions and haptic feedback while providing developers with an integration tool for heterogeneous architectures maintaining high performance, simplicity of implementation, and straightforward extension. The framework uses decoupled simulation with updates of over 1,000 Hz for haptics and accommodates networked simulation with delays of over 1,000 ms without performance penalty.

RESULTS

The simulation software framework was implemented and was used to support the design of virtual reality-based surgery simulation systems. The framework supports the high level of complexity of such applications and the fast response required for interaction with haptics. The efficacy of the framework was tested by implementation of a minimally invasive surgery simulator.

CONCLUSION

A decoupled simulation approach can be implemented as a framework to handle simultaneous processes of the system at the various frame rates each process requires. The framework was successfully used to develop collaborative virtual environments (VEs) involving geographically distributed users connected through a network, with the results comparable to VEs for local users.

GPU-based efficient realistic techniques for bleeding and smoke generation in surgical simulators

BACKGROUND

In actual surgery, smoke and bleeding due to cauterization processes provide important visual cues to the surgeon, which have been proposed as factors in surgical skill assessment. While several virtual reality (VR)-based surgical simulators have incorporated the effects of bleeding and smoke generation, they are not realistic due to the requirement of real-time performance. To be interactive, visual update must be performed at at least 30 Hz and haptic (touch) information must be refreshed at 1 kHz. Simulation of smoke and bleeding is, therefore, either ignored or simulated using highly simplified techniques, since other computationally intensive processes compete for the available Central Processing Unit (CPU) resources.

METHODS

In this study we developed a novel low-cost method to generate realistic bleeding and smoke in VR-based surgical simulators, which outsources the computations to the graphical processing unit (GPU), thus freeing up the CPU for other time-critical tasks. This method is independent of the complexity of the organ models in the virtual environment. User studies were performed using 20 subjects to determine the visual quality of the simulations compared to real surgical videos.

RESULTS

The smoke and bleeding simulation were implemented as part of a laparoscopic adjustable gastric banding (LAGB) simulator. For the bleeding simulation, the original implementation using the shader did not incur noticeable overhead. However, for smoke generation, an input/output (I/O) bottleneck was observed and two different methods were developed to overcome this limitation. Based on our benchmark results, a buffered approach performed better than a pipelined approach and could support up to 15 video streams in real time. Human subject studies showed that the visual realism of the simulations were as good as in real surgery (median rating of 4 on a 5-point Likert scale).

CONCLUSIONS

Based on the performance results and subject study, both bleeding and smoke simulations were concluded to be efficient, highly realistic and well suited to VR-based surgical simulators.

Validation of a novel laparoscopic adjustable gastric band simulator

BACKGROUND

Morbid obesity accounts for more than 90,000 deaths per year in the United States. Laparoscopic adjustable gastric banding (LAGB) is the second most common weight loss procedure performed in the US and the most common in Europe and Australia. Simulation in surgical training is a rapidly advancing field that has been adopted by many to prepare surgeons for surgical techniques and procedures. The aim of our study was to determine face, construct, and content validity for a novel virtual reality laparoscopic adjustable gastric band simulator.

METHODS

Twenty-eight subjects were categorized into two groups (expert and novice), determined by their skill level in laparoscopic surgery. Experts consisted of subjects who had at least 4 years of laparoscopic training and operative experience. Novices consisted of subjects with medical training but with less than 4 years of laparoscopic training. The subjects used the virtual reality laparoscopic adjustable band surgery simulator. They were automatically scored according to various tasks. The subjects then completed a questionnaire to evaluate face and content validity.

RESULTS

On a 5-point Likert scale (1 = lowest score, 5 = highest score), the mean score for visual realism was 4.00 ± 0.67 and the mean score for realism of the interface and tool movements was 4.07 ± 0.77 (face validity). There were significant differences in the performances of the two subject groups (expert and novice) based on total scores (p < 0.001) (construct validity). Mean score for utility of the simulator, as addressed by the expert group, was 4.50 ± 0.71 (content validity).

CONCLUSION

We created a virtual reality laparoscopic adjustable gastric band simulator. Our initial results demonstrate excellent face, construct, and content validity findings. To our knowledge, this is the first virtual reality simulator with haptic feedback for training residents and surgeons in the laparoscopic adjustable gastric banding procedure.

Telehealth innovations in health education and training

Telehealth applications are increasingly important in many areas of health education and training. In addition, they will play a vital role in biomedical research and research training by facilitating remote collaborations and providing access to expensive/remote instrumentation. In order to fulfill their true potential to leverage education, training, and research activities, innovations in telehealth applications should be fostered across a range of technology fronts, including online, on-demand computational models for simulation; simplified interfaces for software and hardware; software frameworks for simulations; portable telepresence systems; artificial intelligence applications to be applied when simulated human patients are not options; and the development of more simulator applications. This article presents the results of discussion on potential areas of future development, barries to overcome, and suggestions to translate the promise of telehealth applications into a transformed environment of training, education, and research in the health sciences.