A mixed-reality surgical trainer with comprehensive sensing for fetal laser minimally invasive surgery

A multimodal virtual vision platform as a next-generation vision system for a surgical robot

Robot-assisted surgery platforms are utilized globally thanks to their stereoscopic vision systems and enhanced functional assistance. However, the necessity of ergonomic improvement for their use by surgeons has been increased. In surgical robots, issues with chronic fatigue exist owing to the fixed posture of the conventional stereo viewer (SV) vision system. A head-mounted display was adopted to alleviate the inconvenience, and a virtual vision platform (VVP) is proposed in this study. The VVP can provide various critical data, including medical images, vital signs, and patient records, in three-dimensional virtual reality space so that users can access medical information simultaneously. An availability of the VVP was investigated based on various user evaluations by surgeons and novices, who executed the given tasks and answered questionnaires. The performances of the SV and VVP were not significantly different; however, the craniovertebral angle of the VVP was 16.35° higher on average than that of the SV. Survey results regarding the VVP were positive; participants indicated that the optimal number of displays was six, preferring the 2 × 3 array. Reflecting the tendencies, the VVP can be a neoconceptual candidate to be customized for medical use, which opens a new prospect in a next-generation surgical robot.

Robot assisted Fetoscopic Laser Coagulation: Improvements in navigation, re-location and coagulation

Fetoscopic Laser Coagulation (FLC) for Twin to Twin Transfusion Syndrome is a challenging intervention due to the working conditions: low quality images acquired from a 3 mm fetoscope inside a turbid liquid environment, local view of the placental surface, unstable surgical field and delicate tissue layers. FLC is based on locating, coagulating and reviewing anastomoses over the placenta's surface. The procedure demands the surgeons to generate a mental map of the placenta with the distribution of the anastomoses, maintaining, at the same time, precision in coagulation and protecting the placenta and amniotic sac from potential damages. This paper describes a teleoperated platform with a cognitive-based control that provides assistance to improve patient safety and surgery performance during fetoscope navigation, target re-location and coagulation processes. A comparative study between manual and teleoperated operation, executed in dry laboratory conditions, analyzes basic fetoscopic skills: fetoscope navigation and laser coagulation. Two exercises are proposed: first, fetoscope guidance and precise coagulation. Second, a resolved placenta (all anastomoses are indicated) to evaluate navigation, re-location and coagulation. The results are analyzed in terms of economy of movement, execution time, coagulation accuracy, amount of coagulated placental surface and risk of placenta puncture. In addition, new metrics, based on navigation and coagulation maps evaluate robotic performance. The results validate the developed platform, showing noticeable improvements in all the metrics.

Development and validation of a flexible fetoscope for fetoscopic laser coagulation

PURPOSE

Fetoscopic laser coagulation for twin-to-twin transfusion syndrome is challenging for anterior placenta due to the rigidity of current tools. The capacity to keep entry port forces minimal is critical for this procedure, as is optimal coagulation distance and orientation. This work introduces technological tools to this end.

METHODS

A novel fetoscope is presented with a rigid shaft and a flexible steerable segment at the distal end. The steerable segment can bend up to 90[Formula: see text] even when loaded with a laser fiber. An artificial pneumatic muscle makes such acute bending possible while allowing for a low-weight and disposable device.

RESULTS

The flexible fetoscope was validated in a custom-made phantom model to measure visual range and coagulation efficacy. The flexible fetoscope shows promising results when compared to a clinical rigid curved fetoscope to reach anterior targets. The new fetoscope was then evaluated in vivo (pregnant ewe) where it successfully coagulated placental vasculature.

CONCLUSION

The flexible fetoscope improved the ability to achieve optimal coagulation angle and distance on anteriorly located targets. The fetoscope also showed the potential to lead fetoscopic laser coagulation and other fetal surgical procedures toward safer and more effective interventions.

Joint scene and object tracking for cost-Effective augmented reality guided patient positioning in radiation therapy

BACKGROUND AND OBJECTIVE

The research is done in the field of Augmented Reality (AR) for patient positioning in radiation therapy is scarce. We propose an efficient and cost-effective algorithm for tracking the scene and the patient to interactively assist the patient's positioning process by providing visual feedback to the operator. Up to our knowledge, this is the first framework that can be employed for mobile interactive AR to guide patient positioning.

METHODS

We propose a pointcloud processing method that, combined with a fiducial marker-mapper algorithm and the generalized ICP algorithm, tracks the patient and the camera precisely and efficiently only using the CPU unit. The 3D reference model and body marker map alignment is calculated employing an efficient body reconstruction algorithm.

RESULTS

Our quantitative evaluation shows that the proposed method achieves a translational and rotational error of 4.17 mm/0.82^∘ at 9 fps. Furthermore, the qualitative results demonstrate the usefulness of our algorithm in patient positioning on different human subjects.

CONCLUSION

Since our algorithm achieves a relatively high frame rate and accuracy employing a regular laptop (without a dedicated GPU), it is a very cost-effective AR-based patient positioning method. It also opens the way for other researchers by introducing a framework that could be improved upon for better mobile interactive AR patient positioning solutions in the future.

On the Use of Virtual Reality for Medical Imaging Visualization

Advanced visualization of medical imaging has been a motive for research due to its value for disease analysis, surgical planning, and academical training. More recently, attention has been turning toward mixed reality as a means to deliver more interactive and realistic medical experiences. However, there are still many limitations to the use of virtual reality for specific scenarios. Our intent is to study the current usage of this technology and assess the potential of related development tools for clinical contexts. This paper focuses on virtual reality as an alternative to today's majority of slice-based medical analysis workstations, bringing more immersive three-dimensional experiences that could help in cross-slice analysis. We determine the key features a virtual reality software should support and present today's software tools and frameworks for researchers that intend to work on immersive medical imaging visualization. Such solutions are assessed to understand their ability to address existing challenges of the field. It was understood that most development frameworks rely on well-established toolkits specialized for healthcare and standard data formats such as DICOM. Also, game engines prove to be adequate means of combining software modules for improved results. Virtual reality seems to remain a promising technology for medical analysis but has not yet achieved its true potential. Our results suggest that prerequisites such as real-time performance and minimum latency pose the greatest limitations for clinical adoption and need to be addressed. There is also a need for further research comparing mixed realities and currently used technologies.

Advances in tissue state recognition in spinal surgery: a review

Spinal disease is an important cause of cervical discomfort, low back pain, radiating pain in the limbs, and neurogenic intermittent claudication, and its incidence is increasing annually. From the etiological viewpoint, these symptoms are directly caused by the compression of the spinal cord, nerve roots, and blood vessels and are most effectively treated with surgery. Spinal surgeries are primarily performed using two different techniques: spinal canal decompression and internal fixation. In the past, tactile sensation was the primary method used by surgeons to understand the state of the tissue within the operating area. However, this method has several disadvantages because of its subjectivity. Therefore, it has become the focus of spinal surgery research so as to strengthen the objectivity of tissue state recognition, improve the accuracy of safe area location, and avoid surgical injury to tissues. Aside from traditional imaging methods, surgical sensing techniques based on force, bioelectrical impedance, and other methods have been gradually developed and tested in the clinical setting. This article reviews the progress of different tissue state recognition methods in spinal surgery and summarizes their advantages and disadvantages.

A novel "video-game" simulator for training fetoscopic laser coagulation of anastomoses in twin-to-twin transfusion syndrome

We have developed a high-fidelity interactive "video-game" simulator in order to teach fetoscopic laser ablation of placental anastomoses for twin-twin transfusion syndrome This simulator may be used by teachers in order to provide metrics-based simulator education to multiple trainees, in both hands-on and distanced learning settings WHAT IS ALREADY KNOWN ABOUT THIS TOPIC?: The use of simulation improves training of the fetoscopic laser techniques utilized in the treatment of twin-twin transfusion syndrome A number of mannequins have been developed to aid this education WHAT DOES THIS STUDY ADD?: Two new simulators are described for twin-twin transfusion syndrome training-silicone and digital The digital simulator is a novel digital video game virtual format This new format has enhanced interactivity and has the potential to enable distance learning.

A Systematic Review and Meta-analysis of the Use of High-Fidelity Simulation in Obstetric Ultrasound

SUMMARY STATEMENT

There is little global consensus on how to train, assess, and evaluate skills in obstetric ultrasound. The outcomes of curricula, where present, are often based on the number of clinical cases completed, rather than objective outcomes. The central question in this review is whether simulation enhances training and prepares trainees for clinical practice. A systematic review was conducted of the currently available literature in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. Studies considering the use of simulators in training or assessment of sonographers were eligible for inclusion. We conclude that simulation is best used for acquisition of technical skills and image optimization. Best outcomes are observed when simulation augments traditional learning, with a strong focus on specific, objective, and measurable skills. Integrating simulation into training curricula could allow trainees to contribute to clinical service while learning. How skills learned in a simulated environment translate to the clinic is poorly addressed by the literature.

Deep learning-based monocular placental pose estimation: towards collaborative robotics in fetoscopy

PURPOSE

Twin-to-twin transfusion syndrome (TTTS) is a placental defect occurring in monochorionic twin pregnancies. It is associated with high risks of fetal loss and perinatal death. Fetoscopic elective laser ablation (ELA) of placental anastomoses has been established as the most effective therapy for TTTS. Current tools and techniques face limitations in case of more complex ELA cases. Visualization of the entire placental surface and vascular equator; maintaining an adequate distance and a close to perpendicular angle between laser fiber and placental surface are central for the effectiveness of laser ablation and procedural success. Robot-assisted technology could address these challenges, offer enhanced dexterity and ultimately improve the safety and effectiveness of the therapeutic procedures.

METHODS

This work proposes a 'minimal' robotic TTTS approach whereby rather than deploying a massive and expensive robotic system, a compact instrument is 'robotised' and endowed with 'robotic' skills so that operators can quickly and efficiently use it. The work reports on automatic placental pose estimation in fetoscopic images. This estimator forms a key building block of a proposed shared-control approach for semi-autonomous fetoscopy. A convolutional neural network (CNN) is trained to predict the relative orientation of the placental surface from a single monocular fetoscope camera image. To overcome the absence of real-life ground-truth placenta pose data, similar to other works in literature (Handa et al. in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016; Gaidon et al. in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016; Vercauteren et al. in: Proceedings of the IEEE, 2019) the network is trained with data generated in a simulated environment and an in-silico phantom model. A limited set of coarsely manually labeled samples from real interventions are added to the training dataset to improve domain adaptation.

RESULTS

The trained network shows promising results on unseen samples from synthetic, phantom and in vivo patient data. The performance of the network for collaborative control purposes was evaluated in a virtual reality simulator in which the virtual flexible distal tip was autonomously controlled by the neural network.

CONCLUSION

Improved alignment was established compared to manual operation for this setting, demonstrating the feasibility to incorporate a CNN-based estimator in a real-time shared control scheme for fetoscopic applications.

Deep learning-based fetoscopic mosaicking for field-of-view expansion

PURPOSE

Fetoscopic laser photocoagulation is a minimally invasive surgical procedure used to treat twin-to-twin transfusion syndrome (TTTS), which involves localization and ablation of abnormal vascular connections on the placenta to regulate the blood flow in both fetuses. This procedure is particularly challenging due to the limited field of view, poor visibility, occasional bleeding, and poor image quality. Fetoscopic mosaicking can help in creating an image with the expanded field of view which could facilitate the clinicians during the TTTS procedure.

METHODS

We propose a deep learning-based mosaicking framework for diverse fetoscopic videos captured from different settings such as simulation, phantoms, ex vivo, and in vivo environments. The proposed mosaicking framework extends an existing deep image homography model to handle video data by introducing the controlled data generation and consistent homography estimation modules. Training is performed on a small subset of fetoscopic images which are independent of the testing videos.

RESULTS

We perform both quantitative and qualitative evaluations on 5 diverse fetoscopic videos (2400 frames) that captured different environments. To demonstrate the robustness of the proposed framework, a comparison is performed with the existing feature-based and deep image homography methods.

CONCLUSION

The proposed mosaicking framework outperformed existing methods and generated meaningful mosaic, while reducing the accumulated drift, even in the presence of visual challenges such as specular highlights, reflection, texture paucity, and low video resolution.

A recent review and a taxonomy for hard and soft tissue visualization-based mixed reality

BACKGROUND

Mixed reality (MR) visualization is gaining popularity in image-guided surgery (IGS) systems, especially for hard and soft tissue surgeries. However, a few MR systems are implemented in real time. Some factors are limiting MR technology and creating a difficulty in setting up and evaluating the MR system in real environments. Some of these factors include: the end users are not considered, the limitations in the operating room, and the medical images are not fully unified into the operating interventions.

METHODOLOGY

The purpose of this article is to use Data, Visualization processing, and View (DVV) taxonomy to evaluate the current MR systems. DVV includes all the components required to be considered and validated for the MR used in hard and soft tissue surgeries. This taxonomy helps the developers and end users like researchers and surgeons to enhance MR system for the surgical field.

RESULTS

We evaluated, validated, and verified the taxonomy based on system comparison, completeness, and acceptance criteria. Around 24 state-of-the-art solutions that are picked relate to MR visualization, which is then used to demonstrate and validate this taxonomy. The results showed that most of the findings are evaluated and others are validated.

CONCLUSION

The DVV taxonomy acts as a great resource for MR visualization in IGS. State-of-the-art solutions are classified, evaluated, validated, and verified to elaborate the process of MR visualization during surgery. The DVV taxonomy provides the benefits to the end users and future improvements in MR.

Computer-Based 3D Simulations to Formulate Preoperative Planning of Bridge Crane Technique for Thoracic Ossification of the Ligamentum Flavum

Background

The bridge crane technique is a novel surgical technique for the treatment of thoracic ossification of the ligamentum flavum (TOLF), but its preoperative planning has not been studied well, which limits the safety and efficacy of surgery to some extent. The purpose of this study was to investigate the method of application and effect of computer-aided preoperative planning (CAPP) on the bridge crane technique for TOLF.

Material/Methods

This retrospective multi-center included 40 patients with TOLF who underwent the bridge crane technique from 2016 to 2018. According to the utilization of CAPP, patients were divided into Group A (with CAPP, n=21) and Group B (without CAPP, n=19). Comparisons of clinical and radiological outcomes were carried out between the 2 groups.

Results

The patients in Group A had higher post-mJOA scores and IR of neurological function than those in Group B (p<0.05). Group A had shorter surgery time, fewer fluoroscopic images, and lower incidence of complications than Group B. In Group A, there was a high consistency of all the anatomical parameters between preoperative simulation and postoperative CT (p>0.05). In Group B, there were significant differences in 3 anatomical parameters between postoperative simulation and postoperative CT (p<0.05). In Group B, the patients with no complications had higher post-SVOR and lower SVRR and height of posterior suspension of LOC in postoperative CT than those in postoperative simulation (p<0.05).

Conclusions

CAPP can enable surgeons to control the decompression effect accurately and reduce the risk of related complications, which improves the safety and efficacy of surgery.

Handheld Active Add-On Control Unit for a Cable-Driven Flexible Endoscope

The instruments currently used by surgeons for in utero treatment of the twin-to-twin transfusion syndrome (TTTS) are rigid or semi-rigid. Their poor dexterity makes this surgical intervention risky and the surgeon's work very complex. This paper proposes the design, assembly and quantitative evaluation of an add-on system intended to be placed on a commercialized cable-driven flexible endoscope. The add-on system is lightweight and easily exchangeable thanks to the McKibben muscle actuators embedded in its system. The combination of the flexible endoscope and the new add-on unit results in an easy controllable flexible instrument with great potential use in TTTS treatment, and especially for regions that are hard to reach with conventional instruments. The fetoscope has a precision of 7.4% over its entire bending range and allows to decrease the maximum planar force on the body wall of 6.15% compared to the original endoscope. The add-on control system also allows a more stable and precise actuation of the endoscope flexible tip.

A novel mixed reality in breast and constructive jaw surgical tele-presence

BACKGROUND AND AIM

Surgical telepresence has been implemented using Mixed reality (MR) but, MR is theory based and only used for investigating research. The Aim of this paper is to propose and implement a new solution by merging augmented video (generating in local site) and virtual expertise surgeon hand (remote site). This system is to improve the visualization of surgical area, overlay accuracy in the merged video without having any discoloured patterns on hand, smudging artefacts on surgeon hand boundary and occluded areas of surgical area.

METHODOLOGY

The Proposed system consists of an Enhanced Multi-Layer Mean Value Cloning (EMLMV) algorithm that improves the overlay accuracy, visualization accuracy and the processing time. This proposed algorithm includes trimap and alpha matting as a pre-processing stage of merging process, which helps to remove the smudging and discoloured artefacts surrounded by remote surgeon hand.

RESULTS

Results showing that the proposed system improved the accuracy by reducing the overlay error of merging image from 1.3 mm (Millimeter) to 0.9 mm. Furthermore, it improves the visibility of surgeon hand in the final merged image from 98.4% (visibility of pixels) to 99.1% (visibility of pixels). Similarly, the processing time in our proposed solution is reduced, which is computed as 10 s to produce 50 frames, whilst, the state of art solution computes 11 s for the same number of frames.

CONCLUSION

The proposed system focuses on the merging of augmented reality video (local site), and the virtual reality video (remote site) with the accurate visualization. we consider discoloured areas, smudging artefacts and occlusion as the main aspects to improve the accuracy of merged video in terms of overlay error and visualization error. So, the proposed system would produce the merged video with the removal of artefacts around the expert surgeon hand.

Mixed Reality-Based Preoperative Planning for Training of Percutaneous Transforaminal Endoscopic Discectomy: A Feasibility Study

OBJECTIVE

To explore the effect of preoperative planning using mixed reality (MR) on training of percutaneous transforaminal endoscopic discectomy (PTED).

METHODS

Before the training, we invited an experienced chief physician to plan the puncture path of PTED on the X-ray films of the lumbar spine model and the 3D Slicer platform, respectively, and used this as the standard to guide trainees. In the aggregate, 60 young residents were randomly divided into Group A (N = 30) and Group B (N = 30). Group A learned the 2-dimensional standard planning route, whereas Group B learned the standard route planning based on MR through the 3D Slicer platform. Then, trainees were asked to conduct PTED puncture on a lumbar spine model. Questionnaires were distributed to trainees before and after the training. During the training, puncture times, operating time (minutes), and fluoroscopy times were recorded.

RESULTS

After the training, it was obvious that more trainees showed their recognition of MR, believing that MR could help preoperative planning and training of PTED. Their high satisfaction with the training indicated the success of our training. Moreover, puncture times, operating time (minutes), and fluoroscopy times of Group B were significantly lower than those of Group A.

CONCLUSIONS

MR technology contributes to preoperative planning of PTED and is beneficial in the training of PTED. It significantly reduces puncture times and fluoroscopy times, providing a standardized method for the training of PTED.

From a Disposable Ureteroscope to an Active Lightweight Fetoscope-Characterization and Usability Evaluation

The twin-to-twin transfusion syndrome is a severe fetal anomaly appearing in up to 15% of identical twin pregnancies. This anomaly occurs when twins share blood vessels from a common placenta. The complication leads to an unbalanced blood transfusion between both fetuses. A current surgical treatment consists in coagulating the shared vessels using a fetoscope with an embedded laser. Such treatment is very delicate and constraining due to limited vision and size of the insertion area. The rigidity and lack of controllability of the current used instruments add an additional difficulty and limit the choice in insertion site. This letter proposes an improved flexible fetoscope, offering an enhanced laser controllability and higher versatility regarding the location of the insertion site. A better approach angle can therefore be realized. Also, tissue damage may be further reduced. This single-handed controllable active fetoscope is obtained after adaptation of a LithoVue (Boston Scientific, Natick, MA, USA), a commercially available passive flexible ureteroscope. The LithoVue is fitted with a unique lightweight add-on actuation module foreseen of an artificial muscle and a dedicated control system. Experiments in a mixed reality trainer suggested that the proposed fetoscope is compact, ergonomic, and intuitive in use, allowing an adequate control of the flexible end.