Mixed reality ultrasound guidance system: a case study in system development and a cautionary tale

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Immersive technologies have thrived on a strong foundation of software and hardware, injecting vitality into medical training. This surge has witnessed numerous endeavors incorporating immersive technologies into surgery simulation for surgical skills training, with a growing number of researchers delving into this domain. Relevant experiences and patterns need to be summarized urgently to enable researchers to establish a comprehensive understanding of this field, thus promoting its continuous growth. This study provides a forward-looking perspective by reviewing the latest development of immersive interactive technologies for surgery simulation. The investigation commences from a technological standpoint, delving into the core aspects of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, namely, haptic rendering and tracking. Subsequently, we summarize recent work based on the categorization of minimally invasive surgery (MIS) and open surgery simulations. Finally, the study showcases the impressive performance and expansive potential of immersive technologies in surgical simulation while also discussing the current limitations. We find that the design of interaction and the choice of immersive technology in virtual surgery development should be closely related to the corresponding interactive operations in the real surgical speciality. This alignment facilitates targeted technological adaptations in the direction of greater applicability and fidelity of simulation.

Development of a Head-Mounted Holographic Needle Guidance System for Enhanced Ultrasound-Guided Neuraxial Anesthesia: System Development and Observational Evaluation (Preprint)

Background

Objective

Methods

Results

Conclusions

Trial Registration

Mixed-Reality-Assisted Puncture of the Common Femoral Artery in a Phantom Model

Percutaneous femoral arterial access is daily practice in a variety of medical specialties and enables physicians worldwide to perform endovascular interventions. The reported incidence of percutaneous femoral arterial access complications is 3–18% and often results from suboptimal puncture location due to insufficient visualization of the target vessel. The purpose of this proof-of-concept study was to evaluate the feasibility and the positional error of a mixed-reality (MR)-assisted puncture of the common femoral artery in a phantom model using a commercially available navigation system. In total, 15 MR-assisted punctures were performed. Cone-beam computed tomography angiography (CTA) was used following each puncture to allow quantification of positional error of needle placements in the axial and sagittal planes. Technical success was achieved in 14/15 cases (93.3%) with a median axial positional error of 1.0 mm (IQR 1.3) and a median sagittal positional error of 1.1 mm (IQR 1.6). The median duration of the registration process and needle insertion was 2 min (IQR 1.0). MR-assisted puncture of the common femoral artery is feasible with acceptable positional errors in a phantom model. Future studies should aim to measure and reduce the positional error resulting from MR registration.

Towards a First-Person Perspective Mixed Reality Guidance System for Needle Interventions

While ultrasound (US) guidance has been used during central venous catheterization to reduce complications, including the puncturing of arteries, the rate of such problems remains non-negligible. To further reduce complication rates, mixed-reality systems have been proposed as part of the user interface for such procedures. We demonstrate the use of a surgical navigation system that renders a calibrated US image, and the needle and its trajectory, in a common frame of reference. We compare the effectiveness of this system, whereby images are rendered on a planar monitor and within a head-mounted display (HMD), to the standard-of-care US-only approach, via a phantom-based user study that recruited 31 expert clinicians and 20 medical students. These users performed needle-insertions into a phantom under the three modes of visualization. The success rates were significantly improved under HMD-guidance as compared to US-guidance, for both expert clinicians (94% vs. 70%) and medical students (70% vs. 25%). Users more consistently positioned their needle closer to the center of the vessel's lumen under HMD-guidance compared to US-guidance. The performance of the clinicians when interacting with this monitor system was comparable to using US-only guidance, with no significant difference being observed across any metrics. The results suggest that the use of an HMD to align the clinician's visual and motor fields promotes successful needle guidance, highlighting the importance of continued HMD-guidance research.

e-Learning in regional anesthesia

PURPOSE OF REVIEW

To explore the role and impact of e-learning technologies on regional anesthesia.

RECENT FINDINGS

21st century technologies, such as 'smart' medical appliances, personal computers, sophisticated apps, the ubiquitous Internet, and online 'e-learning' curricula, are having a powerful impact on anesthesia training: when we learn, what we learn, and how we learn. But is 'new' necessarily 'better'? The answer will result from the application of developments in IT technology through the current vision of architects of future anesthesia training programs. This narrative review aims to summarize the recent developments in anesthesia e-learning, and to forecast trends using regional anesthesia as an example.

SUMMARY

The review offers some recommendations to ensure that the blessings promised to human learning by this 'Brave New Cyberworld' do not become its nemesis.

Potential Efficacy of Multimodal Mixed Reality in Epilepsy Surgery

BACKGROUND

Mixed reality (MR) technology, which can fuse things in real and virtual space in real time, has been used mainly for simulation in neurosurgical training.

OBJECTIVE

To develop MR technology into multimodal MR for intraoperative guidance during epilepsy surgery.

METHODS

A 33-yr-old male patient suffered from intractable general tonic seizures due to left temporal meningoencephalocele. Preoperative scalp electroencephalograms localized interictal epileptic activity on the left temporal lobe. Iomazenil single photon emission tomography revealed temporal lobe lateralization. Magnetic resonance imaging (MRI) demonstrated left basal temporal meningoencephalocele extending into the pterygopalatine fossa through a bone defect at the base of the greater sphenoid wing. A 3-dimensional model was created for MR based on multimodal data including computed tomography, MRI tractography, and digital subtraction angiography, which enabled 3-dimensional visualization of abnormal subcortical fiber connections between the meningoencephalocele and the epileptic focus.

RESULTS

By using intraoperative multimodal MR, we were able to safely remove the meningoencephalocele and perform epileptic focus resection. The patient was seizure-free postoperatively, and no adverse effects were noted.

CONCLUSION

Intraoperative multimodal MR was a feasible and effective technique, and it can be applied for a wide range of epilepsy surgeries.

Application of Mixed Reality Using Optical See-Through Head-Mounted Displays in Transforaminal Percutaneous Endoscopic Lumbar Discectomy

Purpose

Mixed reality (MixR) technology merges the real and virtual worlds to produce new environments and visualizations; it is being tested for numerous minimally invasive surgical procedures. This study is aimed at evaluating the use of MixR technology using optical see-through head-mounted displays (OST-HMDs) during transforaminal percutaneous endoscopic discectomy (TPED).

Methods

Forty-four patients treated with MixR-assisted TPED through OST-HMDs were compared with matched patients treated with conventional TPED (n = 43). In the MixR-assisted TPED group, MixR technology was used to navigate the four procedures of marking, needle insertion, foraminoplasty, and positioning of the working sheath. The clinical outcomes were evaluated based on the numerical rating scale (NRS) scores and Oswestry Disability Index (ODI) on preoperative and postoperative day 1 and at the last follow-up examination. The procedural times, radiation exposure, and eye fatigue were also recorded. All patients were followed up for at least 6 months.

Results

The NRS scores and ODI were significantly improved in both groups at the last follow-up visit compared with the preoperative values (P < 0.05); these values were not statistically different between the groups. The operation time and radiation exposure during marking, needle insertion, and total procedure significantly decreased in the MixR-assisted TPED group compared to those in the conventional TPED group (P < 0.05). Unfortunately, the incidence of eye fatigue increased owing to the use of OST-HMDs in the MixR-assisted TPED group.

Conclusion

This study shows the utility of MixR technology for image guidance in conventional TPED. Radiation exposure is decreased, and this technology serves as a valuable tool during the TPED procedure; however, the assistance of conventional fluoroscopy is still required.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Mixed-reality hologram for diagnosis and surgical planning of double outlet of the right ventricle: a pilot study

AIM

To evaluate the mixed-reality (MR) hologram, a novel technology based on two-dimensional images, which simulates three-dimensional (3D) images and provides a dynamic and interactive alternative, for its usefulness in the diagnosis and surgical planning of double outlet of the right ventricle (DORV).

MATERIALS AND METHODS

Thirty-four patients who were suspected of DORV based on ultrasound findings underwent cardiac computed tomography angiography (CTA). The patients were assigned randomly to the MR holographic guidance (MRHG) group or the control group. For the patients in the MRHG group, the CTA images were converted into Standard Template Library (STL) files after segmentation, 3D reconstruction, colourisation, and transparentisation, and then exported for MR holographic visualisation. The CTA images of the patients in the control group were analysed using routine 3D reconstruction only. Diagnostic accuracy and surgical planning were compared between the two groups based on visualisation at surgery.

RESULTS

In the MRHG group, the 3D hologram observation was in concordance with the actual anatomical findings, and the DORV type was classified accurately in all patients. The diagnostic accuracy for the malformation was 95.5% in the MRHG group and 89.7% in the control group, but the difference was not significant (p=0.3). All the procedures were exactly the same as planned based on the 3D MR holographic model. The surgical planning time was shorter for the MRHG group (51.65 ± 11.11 min) than that for the control group (65.71 ± 18.07 min, p<0.05).

CONCLUSION

MR 3D holograms may provide a clear and deeper anatomical perception of DORV and improve surgical planning.

A recent review and a taxonomy for hard and soft tissue visualization-based mixed reality

BACKGROUND

Mixed reality (MR) visualization is gaining popularity in image-guided surgery (IGS) systems, especially for hard and soft tissue surgeries. However, a few MR systems are implemented in real time. Some factors are limiting MR technology and creating a difficulty in setting up and evaluating the MR system in real environments. Some of these factors include: the end users are not considered, the limitations in the operating room, and the medical images are not fully unified into the operating interventions.

METHODOLOGY

The purpose of this article is to use Data, Visualization processing, and View (DVV) taxonomy to evaluate the current MR systems. DVV includes all the components required to be considered and validated for the MR used in hard and soft tissue surgeries. This taxonomy helps the developers and end users like researchers and surgeons to enhance MR system for the surgical field.

RESULTS

We evaluated, validated, and verified the taxonomy based on system comparison, completeness, and acceptance criteria. Around 24 state-of-the-art solutions that are picked relate to MR visualization, which is then used to demonstrate and validate this taxonomy. The results showed that most of the findings are evaluated and others are validated.

CONCLUSION

The DVV taxonomy acts as a great resource for MR visualization in IGS. State-of-the-art solutions are classified, evaluated, validated, and verified to elaborate the process of MR visualization during surgery. The DVV taxonomy provides the benefits to the end users and future improvements in MR.

Development and Evaluation of an Augmented Reality Ultrasound Guidance System for Spinal Anesthesia: Preliminary Results

Applications of ultrasound guidance for epidural injections are hindered by poor needle and epidural space visualization. This work presents an augmented reality (AR) ultrasound guidance system that addresses challenges in both needle visualization during navigation and epidural space identification for needle positioning. In this system, (i) B-mode ultrasound and the needle are visualized in a 3-D AR environment for improved navigation, and (ii) A-mode ultrasound, obtained from a custom-made single-element transducer housed at the needle tip, is used to identify the epidural space for improved needle positioning. Performance of the system was evaluated against ultrasound-only guidance in a phantom study with novice operators and an expert anesthesiologist. The procedure success rate was higher with the AR system (100%) than ultrasound-only guidance (57%). The AR system has the potential to improve procedure outcomes in terms of success rate, time, needle path-length and usability.

The Effects of Positioning on the Volume/Location of the Internal Jugular Vein Using 2-Dimensional Tracked Ultrasound

OBJECTIVE

To investigate the effects of different positioning on the volume/location of the internal jugular vein (IJV) using 2-dimensional (2D) tracked ultrasound.

DESIGN

This was a prospective, observational study.

SETTING

Local research institute.

PARTICIPANTS

Healthy volunteers.

INTERVENTIONS

Twenty healthy volunteers were scanned in the following 6 positions: (1) supine with head neutral, rotated 15 and 30 degrees to the left and (2) 5-, 10-, and 15-degree Trendelenburg position with head neutral. In each position the volunteer's neck was scanned using a 2D ultrasound probe tracked with a magnetic tracker. These spatially tracked 2D images were collected and reconstructed into a 3D volume of the IJV and carotid artery. This 3D ultrasound volume then was segmented to obtain a 3D surface on which measurements and calculations were performed.

MEASUREMENTS AND MAIN RESULTS

The measurements included average cross-section area (CSA), CSA along the length of IJV, and average overlap rate. CSA (mm²) in the supine and 5-, 10-, and 15-degree Trendelenburg positions were as follows: 86.7 ± 44.8, 104.3 ± 54.5, 119.1 ± 58.6, and 133.7 ± 53.3 (p < 0.0001). CSA enlarged with the increase of Trendelenburg degree. Neither Trendelenburg position nor head rotation showed a correlation with overlap rate.

CONCLUSIONS

Trendelenburg position significantly increased the CSA of the IJV, thus facilitating IJV cannulation. This new 3D reconstruction method permits the creation of a 3D volume through a tracked 2D ultrasound scanning system with image acquisition and integration and may prove useful in providing the user with a "road map" of the vascular anatomy of a patient's neck or other anatomic structures.

Quantitative Analysis of Needle Navigation under Ultrasound Guidance in a Simulated Central Venous Line Procedure

Complications in ultrasound-guided central line insertions are associated with the expertise level of the operator. However, a lack of standards for teaching, training and evaluation of ultrasound guidance results in various levels of competency during training. To address such shortcomings, there has been a paradigm shift in medical education toward competency-based training, promoting the use of simulators and quantitative skills assessment. It is therefore necessary to develop reliable quantitative metrics to establish standards for the attainment and maintenance of competence. This work identifies such a metric for simulated central line procedures. The distance between the needle tip and ultrasound image plane was quantified as a metric of efficacy in ultrasound guidance implementation. In a simulated procedure, performed by experienced physicians, this distance was significantly greater in unsuccessful procedures (p = 0.04). The use of this metric has the potential to enhance the teaching, training and skills assessment of ultrasound-guided central line insertions.