An Augmented Reality System for MR Image–guided Needle Biopsy: Initial Results in a Swine Model

Smartphone Augmented Reality Outperforms Conventional CT Guidance for Composite Ablation Margins in Phantom Models

PURPOSE

To develop and evaluate a smartphone augmented reality (AR) system for a large 50-mm liver tumor ablation with treatment planning for composite overlapping ablation zones.

MATERIALS AND METHODS

A smartphone AR application was developed to display tumor, probe, projected probe paths, ablated zones, and real-time percentage of the ablated target tumor volume. Fiducial markers were attached to phantoms and an ablation probe hub for tracking. The system was evaluated with tissue-mimicking thermochromic phantoms and gel phantoms. Four interventional radiologists performed 2 trials each of 3 probe insertions per trial using AR guidance versus computed tomography (CT) guidance approaches in 2 gel phantoms. Insertion points and optimal probe paths were predetermined. On Gel Phantom 2, serial ablated zones were saved and continuously displayed after each probe placement/adjustment, enabling feedback and iterative planning. The percentages of tumor ablated for AR guidance versus CT guidance, and with versus without display of recorded ablated zones, were compared among interventional radiologists with pairwise t-tests.

RESULTS

The means of percentages of tumor ablated for CT freehand and AR guidance were 36% ± 7 and 47% ± 4 (P = .004), respectively. The mean composite percentages of tumor ablated for AR guidance were 43% ± 1 (without) and 50% ± 2 (with display of ablation zone) (P = .033). There was no strong correlation between AR-guided percentage of ablation and years of experience (r < 0.5), whereas there was a strong correlation between CT-guided percentage of ablation and years of experience (r > 0.9).

CONCLUSIONS

A smartphone AR guidance system for dynamic iterative large liver tumor ablation was accurate, performed better than conventional CT guidance, especially for less experienced interventional radiologists, and enhanced more standardized performance across experience levels for ablation of a 50-mm tumor.

The Development And Applications Of Augmented And Virtual Reality Technology In Spine Surgery Training: A Systematic Review

BACKGROUND

The COVID-19 pandemic has accelerated the growing global interest in the role of augmented and virtual reality in surgical training. While this technology grows at a rapid rate, its efficacy remains unclear. To that end, we offer a systematic review of the literature summarizing the role of virtual and augmented reality on spine surgery training.

METHODS

A systematic review of the literature was conducted on May 13th, 2022. PubMed, Web of Science, Medline, and Embase were reviewed for relevant studies. Studies from both orthopedic and neurosurgical spine programs were considered. There were no restrictions placed on the type of study, virtual/augmented reality modality, nor type of procedure. Qualitative data analysis was performed, and all studies were assigned a Medical Education Research Study Quality Instrument (MERSQI) score.

RESULTS

The initial review identified 6752 studies, of which 16 were deemed relevant and included in the final review, examining a total of nine unique augmented/virtual reality systems. These studies had a moderate methodological quality with a MERSQI score of 12.1 + 1.8; most studies were conducted at single-center institutions, and unclear response rates. Statistical pooling of the data was limited by the heterogeneity of the study designs.

CONCLUSION

This review examined the applications of augmented and virtual reality systems for training residents in various spine procedures. As this technology continues to advance, higher-quality, multi-center, and long-term studies are required to further the adaptation of VR/AR technologies in spine surgery training programs.

Magnetic resonance - guided treatment of low-flow vascular malformations and the technologies to potentiate adoption

Vascular malformations are congenital, non-neoplastic lesions that arise secondary to defects in angiogenesis. Vascular malformations are divided into high-flow (arteriovenous malformation) and low-flow (venous malformations and lymphatic malformations). Magnetic resonance imaging (MRI) is the standard for pre-and post-intervention assessments, while ultrasound (US), X-ray fluoroscopy and computed tomography (CT) are used for intra-procedural guidance. Sclerotherapy, an image-guided therapy that involves the injection of a sclerosant directly into the malformation, is typically the first-line therapy for treating low-flow vascular malformations. Sclerotherapy induces endothelial damage and necrosis/fibrosis with eventual involution of the malformation. Image-guided thermal therapies involve freezing or heating target tissue to induce cell death and necrosis. MRI is an alternative for intra-procedural guidance and monitoring during the treatment of vascular malformations. MR can provide dynamic, multiplanar imaging that delineates surrounding critical structures such as nerves and vasculature. Multiple studies have demonstrated that MR-guided treatment of vascular malformations is safe and effective. This review will detail (1) the use of MR for the classification and diagnosis of vascular malformations, (2) the current literature surrounding MR-guided treatment of vascular malformations, (3) a series of cases of MR-guided sclerotherapy and thermal ablation for the treatment of vascular malformations, and (4) a discussion of technologies that may potentiate interventional MRI adoption including high intensity focused ultrasound and guided laser ablation.

Learning how to perform ultrasound-guided interventions with and without augmented reality visualization: a randomized study

OBJECTIVES

Augmented reality (AR), which entails overlay of in situ images onto the anatomy, may be a promising technique for assisting image-guided interventions. The purpose of this study was to investigate and compare the learning experience and performance of untrained operators in puncture of soft tissue lesions, when using AR ultrasound (AR US) compared with standard US (sUS).

METHODS

Forty-four medical students (28 women, 16 men) who had completed a basic US course, but had no experience with AR US, were asked to perform US-guided biopsies with both sUS and AR US, with a randomized selection of the initial modality. The experimental setup aimed to simulate biopsies of superficial soft tissue lesions, such as for example breast masses in clinical practice, by use of a turkey breast containing olives. Time to puncture(s) and success (yes/no) of the biopsies was documented. All participants completed questionnaires about their coordinative skills and their experience during the training.

RESULTS

Despite having no experience with the AR technique, time to puncture did not differ significantly between AR US and sUS (median [range]: 17.0 s [6-60] and 14.5 s [5-41], p = 0.16), nor were there any gender-related differences (p = 0.22 and p = 0.50). AR US was considered by 79.5% of the operators to be the more enjoyable means of learning and performing US-guided biopsies. Further, a more favorable learning curve was achieved using AR US.

CONCLUSIONS

Students considered AR US to be the preferable and more enjoyable modality for learning how to obtain soft tissue biopsies; however, they did not perform the biopsies faster than when using sUS.

KEY POINTS

• Performance of standard and augmented reality US-guided biopsies was comparable • A more favorable learning curve was achieved using augmented reality US. • Augmented reality US was the preferred technique and was considered more enjoyable.

Phantom and Animal Study of a Robot-Assisted, CT-Guided Targeting System using Image-Only Navigation for Stereotactic Needle Insertion without Positional Sensors

PURPOSE

To evaluate the feasibility and accuracy of a robotic system to integrate and map computed tomography (CT) and robotic coordinates, followed by automatic trajectory execution by a robotic arm. The system was hypothesized to achieve a targeting error of <5 mm without significant influence from variations in angulation or depth.

MATERIALS AND METHODS

An experimental study was conducted using a robotic system (Automated Needle Targeting device for CT [ANT-C]) for needle insertions into a phantom model on both moving patient table and moving gantry CT scanners. Eight spherical markers were registered as targets for 90 insertions at different trajectories. After a single ANT-C registration, the closed-loop software targeted multiple markers via the insertion of robotically aligned 18-gauge needles. Accuracy (distance from the needle tip to the target) was assessed by postinsertion CT scans. Similar procedures were repeated to guide 10 needle insertions into a porcine lung. A regression analysis was performed to test the effect of needle angulation and insertion depth on the accuracy of insertion.

RESULTS

In the phantom model, all needle insertions (median trajectory depth, 64.8 mm; range, 46.1-153 mm) were successfully performed in single attempts. The overall accuracy was 1.36 mm ± 0.53, which did not differ between the 2 types of CT scanners (1.39 mm ± 0.54 [moving patient table CT] vs 1.33 mm ± 0.52 [moving gantry CT]; P = .54) and was not significantly affected by the needle angulation and insertion depth. The accuracy for the porcine model was 9.09 mm ± 4.21.

CONCLUSIONS

Robot-assisted needle insertion using the ANT-C robotic device was feasible and accurate for targeting multiple markers in a phantom model.

Noncontact measurement of puncture needle angle using augmented reality technology in computed tomography-guided biopsy: stereotactic coordinate design and accuracy evaluation

PURPOSE

This study aims to introduce a new handheld device application for noncontact and real-time measurements of the angle of a biopsy needle using augmented reality (AR) image tracking technology. Furthermore, this study discusses the methods used to optimize the related coordinate design for computed tomography (CT)-guided biopsy procedures.

METHODS

An in-house noncontact angle measurement application was developed using AR platform software. This application tracks the position and direction of a printed texture located on the handle of a biopsy needle. The needle direction was factorized into two directions: tilting or rolling. Tilting was defined following the tilting of the CT gantry so that rolling would match the angle measured in CT images. In this study, CT-guided tumor biopsies were performed using a conventional guiding method with a protractor. The true value of needle rolling was measured by CT imaging and was then compared to the rolling measurement provided by the application developed in the current study using a mobile phone.

RESULTS

This study enrolled 18 cases of tumor biopsy (five renal tumors, five lung tumors, four retroperitoneal tumors, one soft tissue tumor, one thyroid tumor, one mesentery tumor, and one bone tumor). The measurement accuracy was - 0.2°, which was the average difference between AR and CT, and the measurement precision was 2.0°, which was the standard deviation of the difference between AR and CT measurements. The coefficient of determination (R²) was 0.996.

CONCLUSION

The noncontact needle measurement software using AR technology is sufficiently reliable for use in clinical settings. A real-time display of the needle angle that also shows the direction of the CT gantry is expected to enable a simple biopsy needle navigation.

A System for Real-Time, Online Mixed-Reality Visualization of Cardiac Magnetic Resonance Images

Image-guided cardiovascular interventions are rapidly evolving procedures that necessitate imaging systems capable of rapid data acquisition and low-latency image reconstruction and visualization. Compared to alternative modalities, Magnetic Resonance Imaging (MRI) is attractive for guidance in complex interventional settings thanks to excellent soft tissue contrast and large fields-of-view without exposure to ionizing radiation. However, most clinically deployed MRI sequences and visualization pipelines exhibit poor latency characteristics, and spatial integration of complex anatomy and device orientation can be challenging on conventional 2D displays. This work demonstrates a proof-of-concept system linking real-time cardiac MR image acquisition, online low-latency reconstruction, and a stereoscopic display to support further development in real-time MR-guided intervention. Data are acquired using an undersampled, radial trajectory and reconstructed via parallelized through-time radial generalized autocalibrating partially parallel acquisition (GRAPPA) implemented on graphics processing units. Images are rendered for display in a stereoscopic mixed-reality head-mounted display. The system is successfully tested by imaging standard cardiac views in healthy volunteers. Datasets comprised of one slice (46 ms), two slices (92 ms), and three slices (138 ms) are collected, with the acquisition time of each listed in parentheses. Images are displayed with latencies of 42 ms/frame or less for all three conditions. Volumetric data are acquired at one volume per heartbeat with acquisition times of 467 ms and 588 ms when 8 and 12 partitions are acquired, respectively. Volumes are displayed with a latency of 286 ms or less. The faster-than-acquisition latencies for both planar and volumetric display enable real-time 3D visualization of the heart.

A Holographic Augmented Reality Guidance System for Patient Alignment: A Feasibility Study

Purpose

To evaluate the accuracy of an augmented reality holographic guidance system for potential use in patient alignment in radiotherapy applications.

Methods

A cubic phantom was scanned on a CT simulator and a 3D mesh was extracted using the Eclipse Scripting API. An application was created for the Microsoft HoloLens to allow users to see the scanned mesh as a hologram overlaid in the treatment vault. Six therapists were equipped with the HoloLens glasses and instructed to move the real phantom to align with the perceived spatial hologram using only couch controls. The initial couch coordinates were recorded and then recorded at each step as the therapist moved the phantom to each new location. The application varied the position of the virtual phantom to 10 preprogrammed locations within a 40-cm cubic volume in a combination of vertical, longitudinal, and lateral axis shifts. The absolute position difference between the holographic world and real-world phantom was recorded at each step. Also, the relative position from one position to the next was recorded.

Results

Fifty shifts were collected across the six therapists. The mean difference between the physical position and instructed holographic position was 0.58 ± 0.31 cm for relative shifts and 0.51 ± 0.33 cm for absolute position. The maximum difference between the holographic position and the actual post shift position was 1.53 cm for relative and 1.58 cm for absolute.

Conclusion

Holographic augmented reality guidance using the Microsoft HoloLens provides adequate accuracy for initial treatment alignment but lacks the fine alignment accuracy of X-ray imaging systems.

Needle artifact characteristics and insertion accuracy using a 1.2T open MRI scanner: A phantom study

PURPOSE

To evaluate the characteristics of needle artifacts and the accuracy of needle insertion using a 1.2 Tesla open magnetic resonance imaging (MRI) system in a phantom.

MATERIALS AND METHODS

First, the apparent width of the needle on the MRI and the needle tip position error of 16- and 18-gauge MRI-compatible introducer needles and a 17-gauge cryoneedle were examined with different needle angles (0°, 30°, 45°, 60°, and 90°) to the main magnetic field (B0), sequence types (balanced steady-state acquisition with rewound gradient echo [BASG] and T2-weighted fast spin echo [FSE] sequence), and frequency encoding directions. Second, the accuracy of needle insertion was evaluated after 10 MRI fluoroscopy-guided insertions in a phantom.

RESULTS

The apparent needle widths was larger when the angle of the needle axis relative to B0 was larger. The needles appeared larger on BASG than on T2-weighted FSE images, with the largest apparent widths of 16-, 17-, and 18-gauge needles of 14.3, 11.6, and 11.0mm, respectively. The apparent needle tip position was always more distal than the actual position on BASG images, with the largest longitudinal error of 4.0mm. Meanwhile, the 16- and 18-gauge needle tips appeared more proximal on T2-weighted FSE images with right-to-left frequency encoding direction. The mean accuracy of MRI fluoroscopy-guided needle insertion was 3.1mm.

CONCLUSION

These experiments clarify the characteristics of needle artifacts in a 1.2 Tesla open MRI. With this system, the MRI fluoroscopy-guided needle insertion demonstrated an acceptable accuracy for clinical use.

Comparison of Augmented Reality Display Techniques to Support Medical Needle Insertion

Augmented reality (AR) may be a useful technique to overcome issues of conventionally used navigation systems supporting medical needle insertions, like increased mental workload and complicated hand-eye coordination. Previous research primarily focused on the development of AR navigation systems designed for specific displaying devices, but differences between employed methods have not been investigated before. To this end, a user study involving a needle insertion task was conducted comparing different AR display techniques with a monitor-based approach as baseline condition for the visualization of navigation information. A video see-through stationary display, an optical see-through head-mounted display and a spatial AR projector-camera-system were investigated in this comparison. Results suggest advantages of using projected navigation information in terms of lower task completion time, lower angular deviation and affirmative subjective participant feedback. Techniques requiring the intermediate view on screens, i.e. the stationary display and the baseline condition, showed less favorable results. Thus, benefits of providing AR navigation information compared to a conventionally used method could be identified. Significant objective measures results, as well as an identification of advantages and disadvantages of individual display techniques contribute to the development and design of improved needle navigation systems.

Smartphone- versus smartglasses-based augmented reality (AR) for percutaneous needle interventions: system accuracy and feasibility study

PURPOSE

To compare the system accuracy and needle placement performance of smartphone- and smartglasses-based augmented reality (AR) for percutaneous needle interventions.

METHODS

An AR platform was developed to enable the superimposition of annotated anatomy and a planned needle trajectory onto a patient in real time. The system accuracy of the AR display on smartphone (iPhone7) and smartglasses (HoloLens1) devices was evaluated on a 3D-printed phantom. The target overlay error was measured as the distance between actual and virtual targets (n = 336) on the AR display, derived from preprocedural CT. The needle overlay angle was measured as the angular difference between actual and virtual needles (n = 12) on the AR display. Three operators each used the iPhone (n = 8), HoloLens (n = 8) and CT-guided freehand (n = 8) to guide needles into targets in a phantom. Needle placement error was measured with post-placement CT. Needle placement time was recorded from needle puncture to navigation completion.

RESULTS

The target overlay error of the iPhone was comparable to the HoloLens (1.75 ± 0.59 mm, 1.74 ± 0.86 mm, respectively, p = 0.9). The needle overlay angle of the iPhone and HoloLens was similar (0.28 ± 0.32°, 0.41 ± 0.23°, respectively, p = 0.26). The iPhone-guided needle placements showed reduced error compared to the HoloLens (2.58 ± 1.04 mm, 3.61 ± 2.25 mm, respectively, p = 0.05) and increased time (87 ± 17 s, 71 ± 27 s, respectively, p = 0.02). Both AR devices reduced placement error compared to CT-guided freehand (15.92 ± 8.06 mm, both p < 0.001).

CONCLUSION

An augmented reality platform employed on smartphone and smartglasses devices may provide accurate display and navigation guidance for percutaneous needle-based interventions.

Integration and evaluation of a gradient-based needle navigation system for percutaneous MR-guided interventions

The purpose of the present study was to integrate an interactive gradient-based needle navigation system and to evaluate the feasibility and accuracy of the system for real-time MR guided needle puncture in a multi-ring phantom and in vivo in a porcine model. The gradient-based navigation system was implemented in a 1.5T MRI. An interactive multi-slice real-time sequence was modified to provide the excitation gradients used by two sets of three orthogonal pick-up coils integrated into a needle holder. Position and orientation of the needle holder were determined and the trajectory was superimposed on pre-acquired MR images. A gel phantom with embedded ring targets was used to evaluate accuracy using 3D distance from needle tip to target. Six punctures were performed in animals to evaluate feasibility, time, overall error (target to needle tip) and system error (needle tip to the guidance needle trajectory) in vivo. In the phantom experiments, the overall error was 6.2±2.9 mm (mean±SD) and 4.4±1.3 mm, respectively. In the porcine model, the setup time ranged from 176 to 204 seconds, the average needle insertion time was 96.3±40.5 seconds (min: 42 seconds; max: 154 seconds). The overall error and the system error was 8.8±7.8 mm (min: 0.8 mm; max: 20.0 mm) and 3.3±1.4 mm (min: 1.8 mm; max: 5.2 mm), respectively.

Augmented and Mixed Reality: Technologies for Enhancing the Future of IR

Augmented and mixed reality are emerging interactive and display technologies. These technologies are able to merge virtual objects, in either 2 or 3 dimensions, with the real world. Image guidance is the cornerstone of interventional radiology. With augmented or mixed reality, medical imaging can be more readily accessible or displayed in actual 3-dimensional space during procedures to enhance guidance, at times when this information is most needed. In this review, the current state of these technologies is addressed followed by a fundamental overview of their inner workings and challenges with 3-dimensional visualization. Finally, current and potential future applications in interventional radiology are highlighted.

Comparison of Projective Augmented Reality Concepts to Support Medical Needle Insertion

Augmented reality (AR) is a promising tool to improve instrument navigation in needle-based interventions. Limited research has been conducted regarding suitable navigation visualizations. In this work, three navigation concepts based on existing approaches were compared in a user study using a projective AR setup. Each concept was implemented with three different scales for accuracy-to-color mapping and two methods of navigation indicator scaling. Participants were asked to perform simulated needle insertion tasks with each of the resulting 18 prototypes. Insertion angle and insertion depth accuracies were measured and analyzed, as well as task completion time and participants' subjectively perceived task difficulty. Results show a clear ranking of visualization concepts across variables. Less consistent results were obtained for the color and indicator scaling factors. Results suggest that logarithmic indicator scaling achieved better accuracy, but participants perceived it to be more difficult than linear scaling. With specific results for angle and depth accuracy, our study contributes to the future composition of improved navigation support and systems for precise needle insertion or similar applications.

Augmented Reality-Guided Lumbar Facet Joint Injections

OBJECTIVES

The aim of this study was to assess feasibility and accuracy of augmented reality-guided lumbar facet joint injections.

MATERIALS AND METHODS

A spine phantom completely embedded in hardened opaque agar with 3 ring markers was built. A 3-dimensional model of the phantom was uploaded to an augmented reality headset (Microsoft HoloLens). Two radiologists independently performed 20 augmented reality-guided and 20 computed tomography (CT)-guided facet joint injections each: for each augmented reality-guided injection, the hologram was manually aligned with the phantom container using the ring markers. The radiologists targeted the virtual facet joint and tried to place the needle tip in the holographic joint space. Computed tomography was performed after each needle placement to document final needle tip position. Time needed from grabbing the needle to final needle placement was measured for each simulated injection. An independent radiologist rated images of all needle placements in a randomized order blinded to modality (augmented reality vs CT) and performer as perfect, acceptable, incorrect, or unsafe. Accuracy and time to place needles were compared between augmented reality-guided and CT-guided facet joint injections.

RESULTS

In total, 39/40 (97.5%) of augmented reality-guided needle placements were either perfect or acceptable compared with 40/40 (100%) CT-guided needle placements (P = 0.5). One augmented reality-guided injection missed the facet joint space by 2 mm. No unsafe needle placements occurred. Time to final needle placement was substantially faster with augmented reality guidance (mean 14 ± 6 seconds vs 39 ± 15 seconds, P < 0.001 for both readers).

CONCLUSIONS

Augmented reality-guided facet joint injections are feasible and accurate without potentially harmful needle placement in an experimental setting.

Percutaneous MR-guided interventions using an optical Moiré Phase tracking system: Initial results

The aim of this study was the development and evaluation of a real-time guidance support using optical Moiré Phase Tracking (MPT) for magnetic resonance (MR) guided percutaneous interventions. A gradient echo sequence, capable of real-time position updates by the MPT system, was modified to enable needle guidance based on four rigidly attached MPT markers at the back of a needle. Two perpendicular imaging planes were automatically aligned along the calibrated needle and centered at its tip. For user guidance, additional information about the needle trajectory and the tip to target distance were added as image overlay. Both, images and guiding information were displayed on the in-room monitor to facilitate MR guided interventions. The guidance support was evaluated by four experienced interventional radiologists and four novices targeting rubber O-rings embedded in a custom-made phantom on a 3T wide-bore MRI system (80 punctures). The skin to target time, user error, system error and total error were analyzed. The mean skin to target time was 146s±68s with no statistically significant difference between experts and novices. A low mean user error (0.91mm±0.43mm), system error (0.53mm±0.27mm) and total error (0.99mm±0.47mm) was reached in all directions. No statistically significant difference in user error, system error and total error could be found between experts and novices. The presented tracking and image guidance system combined with the user interface offers continuous and interactive control of the imaging plane while puncturing in the magnet enabling accurate real-time feedback for both, experienced and non-experienced users.

Concepts for augmented reality visualisation to support needle guidance inside the MRI

During MRI-guided interventions, navigation support is often separated from the operating field on displays, which impedes the interpretation of positions and orientations of instruments inside the patient's body as well as hand–eye coordination. To overcome these issues projector-based augmented reality can be used to support needle guidance inside the MRI bore directly in the operating field. The authors present two visualisation concepts for needle navigation aids which were compared in an accuracy and usability study with eight participants, four of whom were experienced radiologists. The results show that both concepts are equally accurate (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$2.0 \pm 0.6$\end{document}2.0±0.6 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{upgreek} \usepackage{mathrsfs} \setlength{\oddsidemargin}{-69pt} \begin{document} }{}$1.7 \pm 0.5\, {\rm mm}$\end{document}1.7±0.5mm), useful and easy to use, with clear visual feedback about the state and success of the needle puncture. For easier clinical applicability, a dynamic projection on moving surfaces and organ movement tracking are needed. For now, tests with patients with respiratory arrest are feasible.

A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014

Augmented Reality (AR) interfaces have been studied extensively over the last few decades, with a growing number of user-based experiments. In this paper, we systematically review 10 years of the most influential AR user studies, from 2005 to 2014. A total of 291 papers with 369 individual user studies have been reviewed and classified based on their application areas. The primary contribution of the review is to present the broad landscape of user-based AR research, and to provide a high-level view of how that landscape has changed. We summarize the high-level contributions from each category of papers, and present examples of the most influential user studies. We also identify areas where there have been few user studies, and opportunities for future research. Among other things, we find that there is a growing trend toward handheld AR user studies, and that most studies are conducted in laboratory settings and do not involve pilot testing. This research will be useful for AR researchers who want to follow best practices in designing their own AR user studies.

Augmented reality in neurosurgery

Neurosurgery is a medical specialty that relies heavily on imaging. The use of computed tomography and magnetic resonance images during preoperative planning and intraoperative surgical navigation is vital to the success of the surgery and positive patient outcome. Augmented reality application in neurosurgery has the potential to revolutionize and change the way neurosurgeons plan and perform surgical procedures in the future. Augmented reality technology is currently commercially available for neurosurgery for simulation and training. However, the use of augmented reality in the clinical setting is still in its infancy. Researchers are now testing augmented reality system prototypes to determine and address the barriers and limitations of the technology before it can be widely accepted and used in the clinical setting.

Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology

Context

Augmented reality (AR) devices such as the Microsoft HoloLens have not been well used in the medical field.

Objective

To test the HoloLens for clinical and nonclinical applications in pathology.

Design

A Microsoft HoloLens was tested for virtual annotation during autopsy, viewing 3D gross and microscopic pathology specimens, navigating whole slide images, telepathology, as well as real-time pathology-radiology correlation.

Results

Pathology residents performing an autopsy wearing the HoloLens were remotely instructed with real-time diagrams, annotations, and voice instruction. 3D-scanned gross pathology specimens could be viewed as holograms and easily manipulated. Telepathology was supported during gross examination and at the time of intraoperative consultation, allowing users to remotely access a pathologist for guidance and to virtually annotate areas of interest on specimens in real-time. The HoloLens permitted radiographs to be coregistered on gross specimens and thereby enhanced locating important pathologic findings. The HoloLens also allowed easy viewing and navigation of whole slide images, using an AR workstation, including multiple coregistered tissue sections facilitating volumetric pathology evaluation.

Conclusions

The HoloLens is a novel AR tool with multiple clinical and nonclinical applications in pathology. The device was comfortable to wear, easy to use, provided sufficient computing power, and supported high-resolution imaging. It was useful for autopsy, gross and microscopic examination, and ideally suited for digital pathology. Unique applications include remote supervision and annotation, 3D image viewing and manipulation, telepathology in a mixed-reality environment, and real-time pathology-radiology correlation.

1.5 T augmented reality navigated interventional MRI: paravertebral sympathetic plexus injections

PURPOSE

The high contrast resolution and absent ionizing radiation of interventional magnetic resonance imaging (MRI) can be advantageous for paravertebral sympathetic nerve plexus injections. We assessed the feasibility and technical performance of MRI-guided paravertebral sympathetic injections utilizing augmented reality navigation and 1.5 T MRI scanner.

METHODS

A total of 23 bilateral injections of the thoracic (8/23, 35%), lumbar (8/23, 35%), and hypogastric (7/23, 30%) paravertebral sympathetic plexus were prospectively planned in twelve human cadavers using a 1.5 Tesla (T) MRI scanner and augmented reality navigation system. MRI-conditional needles were used. Gadolinium-DTPA-enhanced saline was injected. Outcome variables included the number of control magnetic resonance images, target error of the needle tip, punctures of critical nontarget structures, distribution of the injected fluid, and procedure length.

RESULTS

Augmented-reality navigated MRI guidance at 1.5 T provided detailed anatomical visualization for successful targeting of the paravertebral space, needle placement, and perineural paravertebral injections in 46 of 46 targets (100%). A mean of 2 images (range, 1-5 images) were required to control needle placement. Changes of the needle trajectory occurred in 9 of 46 targets (20%) and changes of needle advancement occurred in 6 of 46 targets (13%), which were statistically not related to spinal regions (P = 0.728 and P = 0.86, respectively) and cadaver sizes (P = 0.893 and P = 0.859, respectively). The mean error of the needle tip was 3.9±1.7 mm. There were no punctures of critical nontarget structures. The mean procedure length was 33±12 min.

CONCLUSION

1.5 T augmented reality-navigated interventional MRI can provide accurate imaging guidance for perineural injections of the thoracic, lumbar, and hypogastric sympathetic plexus.

Cryosurgical Ablation of Renal Cell Carcinoma

BACKGROUND

Small renal masses are being commonly diagnosed incidentally in older patients. A partial nephrectomy is the first-line nephron sparing treatment option for these lesions. However, probe ablative therapy such as cryoablation is emerging as an alternative option for select patients requiring nephron sparing surgery.

METHODS

The current literature regarding the management of small renal lesions with cryoablation was retrospectively reviewed. We selected six of the largest published series of renal cryoablation with a total of 320 patients. The diagnosis, staging, treatment options, mechanism, efficacy and morbidity associated with renal cryoablation were evaluated.

RESULTS

Renal cryoablation for localized small renal masses is well tolerated and associated with a low complication rate. The range of mean tumor size in our literature review series (320 patients) was 2.3 to 2.6 cm. After a range of mean follow-up of 5.9 to 72 months, including a series with a minimum of 5 years of follow-up, the cancer specific survival was 97% to 100% and overall patient survival was 82% to 90.2%.

CONCLUSIONS

Renal cryoablation, based on available clinical reports, appears to be a curative option for patients with small localized renal cell carcinomas (RCCs) who are unwilling or unable to undergo a partial nephrectomy. With encouraging intermediate oncological follow-up available, longer-term follow-up is needed to validate the use of cryoablation as a primary treatment option.

In vivo comparison of two navigation systems for abdominal percutaneous needle intervention

PURPOSE

To compare the accuracy of a Kinect-Optical navigation system with an electromagnetic (EM) navigation system for percutaneous liver needle intervention.

MATERIALS AND METHODS

Five beagles with nine artificial tumors were used for validation. The Veran IG4 EM navigation system and a custom-made Kinect-Optical navigation system were used. Needle insertions into each tumor were conducted with these two guidance methods. The target positioning error (TPE) and the time cost of the puncture procedures were evaluated.

RESULTS

A total of 18 needle insertions were performed to evaluate the navigation accuracy of the two guidance approaches. The targeting error was 6.78 ± 3.22 mm and 8.72 ± 3.5 mm for the Kinect-Optical navigation system and the EM navigation system, respectively. There is no statistically significant difference in the TPE between the Kinect-Optical navigation system and the EM navigation system (p = 0.229). The processing time with the Kinect-Optical system (10 min) is similar to that of the Veran IG4 system (12 min).

CONCLUSIONS

The accuracy of the Kinect-Optical navigation system is comparable to that of the EM navigation system.

Magnetic Resonance Image-Guided Focal Prostate Ablation

Prostate cancer is the most common cancer (other than skin cancer) in American men, with one in seven men being diagnosed with this disease during his lifetime. The estimated number of new prostate cancer cases in 2016 is 180,890. For the first time, imaging has become the center of the search for contained, intraglandular, small-volume, and unifocal disease, and an increasing number of academic institutions as well as private practices are implementing programs for prostate multiplanar magnetic resonance imaging (MRI) as parts of their routine offerings. This article reviews the role of MRI-guided focal prostate ablation, as well as opportunities for further growth in this minimally invasive therapy of prostate cancer.

Targeting Accuracy, Procedure Times and User Experience of 240 Experimental MRI Biopsies Guided by a Clinical Add-On Navigation System

OBJECTIVES

MRI is of great clinical utility for the guidance of special diagnostic and therapeutic interventions. The majority of such procedures are performed iteratively ("in-and-out") in standard, closed-bore MRI systems with control imaging inside the bore and needle adjustments outside the bore. The fundamental limitations of such an approach have led to the development of various assistance techniques, from simple guidance tools to advanced navigation systems. The purpose of this work was to thoroughly assess the targeting accuracy, workflow and usability of a clinical add-on navigation solution on 240 simulated biopsies by different medical operators.

METHODS

Navigation relied on a virtual 3D MRI scene with real-time overlay of the optically tracked biopsy needle. Smart reference markers on a freely adjustable arm ensured proper registration. Twenty-four operators - attending (AR) and resident radiologists (RR) as well as medical students (MS) - performed well-controlled biopsies of 10 embedded model targets (mean diameter: 8.5 mm, insertion depths: 17-76 mm). Targeting accuracy, procedure times and 13 Likert scores on system performance were determined (strong agreement: 5.0).

RESULTS

Differences in diagnostic success rates (AR: 93%, RR: 88%, MS: 81%) were not significant. In contrast, between-group differences in biopsy times (AR: 4:15, RR: 4:40, MS: 5:06 min:sec) differed significantly (p<0.01). Mean overall rating was 4.2. The average operator would use the system again (4.8) and stated that the outcome justifies the extra effort (4.4). Lowest agreement was reported for the robustness against external perturbations (2.8).

CONCLUSIONS

The described combination of optical tracking technology with an automatic MRI registration appears to be sufficiently accurate for instrument guidance in a standard (closed-bore) MRI environment. High targeting accuracy and usability was demonstrated on a relatively large number of procedures and operators. Between groups with different expertise there were significant differences in experimental procedure times but not in the number of successful biopsies.

MRI-guided percutaneous transhepatic cholangiodrainage: feasibility study in a porcine model

BACKGROUND AND STUDY AIMS

MRI-guided procedures combine high-quality imaging with lack of radiation. Percutaneous transhepatic cholangiodrainage under real-time MRI guidance (MRI-PTCD) seems promising, allowing targeted puncture and avoiding multiple blind passes and use of contrast, which are associated with standard PTCD's heaviest complications.

PATIENTS AND METHODS

Aim of this study was to investigate the feasibility of MRI-PTCD in three outbred piglets. Obstructive cholestasis was induced by common bile duct ligation. Two days later, MRI-PTCD was performed (open MRI, 1.0 Tesla) with prototype MRI-compatible accessories. Visualization was achieved with a balanced steady-state free precession real-time sequence (bSSFP: 0.75 frames/s, TR/TE [ms]: 7.2/3.6; flip angle: 45°; 200 × 200 matrix size; resolution: 1.3 × 1.3 mm(2), slice thickness: 7 mm). Cannulation of the bile ducts was followed by placement of Yamakawa drainages.

RESULTS

Twelve punctures were performed (four per animal, 10/12 successful); in 2/10 the bile ducts could not be cannulated. Animal survival was 100% and no significant complications occurred.

CONCLUSIONS

Initial data show that MRI-PTCD can be successfully performed. This may lead to establishment of a new optimized PTCD technique compared to the standard approach under fluoroscopy.

Real-time 3D image reconstruction guidance in liver resection surgery

BACKGROUND

Minimally invasive surgery represents one of the main evolutions of surgical techniques. However, minimally invasive surgery adds difficulty that can be reduced through computer technology.

METHODS

From a patient's medical image [US, computed tomography (CT) or MRI], we have developed an Augmented Reality (AR) system that increases the surgeon's intraoperative vision by providing a virtual transparency of the patient. AR is based on two major processes: 3D modeling and visualization of anatomical or pathological structures appearing in the medical image, and the registration of this visualization onto the real patient. We have thus developed a new online service, named Visible Patient, providing efficient 3D modeling of patients. We have then developed several 3D visualization and surgical planning software tools to combine direct volume rendering and surface rendering. Finally, we have developed two registration techniques, one interactive and one automatic providing intraoperative augmented reality view.

RESULTS

From January 2009 to June 2013, 769 clinical cases have been modeled by the Visible Patient service. Moreover, three clinical validations have been realized demonstrating the accuracy of 3D models and their great benefit, potentially increasing surgical eligibility in liver surgery (20% of cases). From these 3D models, more than 50 interactive AR-assisted surgical procedures have been realized illustrating the potential clinical benefit of such assistance to gain safety, but also current limits that automatic augmented reality will overcome.

CONCLUSIONS

Virtual patient modeling should be mandatory for certain interventions that have now to be defined, such as liver surgery. Augmented reality is clearly the next step of the new surgical instrumentation but remains currently limited due to the complexity of organ deformations during surgery. Intraoperative medical imaging used in new generation of automated augmented reality should solve this issue thanks to the development of Hybrid OR.

Visualization of needle access pathway and a five-DoF evaluation

It is a common practice nowadays to plan needle access pathways to the volumetric organs before performing surgeries. An enormous amount of needle access planning systems has been proposed in recent years. Recent works mainly focus on the system usability or target accessibility. Visualization of the planned access pathways has drawn little attention and its effect on insertion quality is left unattended. We aim to address this problem by introducing an all-round evaluation framework that links up with human motions and computer graphics. Our evaluation framework provides an objective and quantitative analysis of the illustrativeness of the needle access pathway visualization techniques to an extent of five degrees of freedom. Our experimental results show that the visualization method adopted greatly influences insertion accuracy. Based on this finding, we propose a new visualization technique that intuitively conveys placement and orientation information. We also show that our method better conveys pathway orientation and thus enables a higher quality of insertion.

Short rigid scope and stereo-scope designed specifically for open abdominal navigation surgery: clinical application for hepatobiliary and pancreatic surgery

BACKGROUND

We have reported the utility of an image display system using augmented reality (AR) technology in hepatobiliary surgery under laparotomy. Among several procedures, we herein report a system using a novel short rigid scope and stereo-scope, both designed specifically for open abdominal navigation surgery, and their clinical application for hepatobiliary and pancreatic surgery.

METHODS

The 3D reconstructed images were obtained from preoperative computed tomography data. In our specialized operating room, after paired-point matching registration, the reconstructed images are overlaid onto the operative field images captured by the short rigid scopes. The scopes, which are compact and sterilizable, can be used in the operative field. The stereo-scope provides depth information. Eight patients underwent operations using this system, including hepatectomy in two, distal pancreatectomy in three, and pancreaticoduodenectomy in three patients. The stereo-scope was used in five patients.

RESULTS

All eight operations were performed safely using the novel short rigid scopes, and stereo images were acquired in all five patients for whom the stereo-scope was used. The scopes were user friendly, and the intraoperative time requirement for our system was reduced compared with the conventional method.

CONCLUSIONS

The novel short rigid scope and stereo-scope seem to be suitable for clinical use in open abdominal navigation surgery. In hepatobiliary and pancreatic surgery, our novel system may improve the safety, accuracy and efficiency of operations.

High-resolution magnetic resonance-guided posterior femoral cutaneous nerve blocks

OBJECTIVE

To assess the feasibility, technical success, and effectiveness of high-resolution magnetic resonance (MR)-guided posterior femoral cutaneous nerve (PFCN) blocks.

MATERIALS AND METHODS

A retrospective analysis of 12 posterior femoral cutaneous nerve blocks in 8 patients [6 (75%) female, 2 (25%) male; mean age, 47 years; range, 42-84 years] with chronic perineal pain suggesting PFCN neuropathy was performed. Procedures were performed with a clinical wide-bore 1.5-T MR imaging system. High-resolution MR imaging was utilized for visualization and targeting of the PFCN. Commercially available, MR-compatible 20-G needles were used for drug delivery. Variables assessed were technical success (defined as injectant surrounding the targeted PFCN on post-intervention MR images) effectiveness, (defined as post-interventional regional anesthesia of the target area innervation downstream from the posterior femoral cutaneous nerve block), rate of complications, and length of procedure time.

RESULTS

MR-guided PFCN injections were technically successful in 12/12 cases (100%) with uniform perineural distribution of the injectant. All blocks were effective and resulted in post-interventional regional anesthesia of the expected areas (12/12, 100%). No complications occurred during the procedure or during follow-up. The average total procedure time was 45 min (30-70) min.

CONCLUSIONS

Our initial results demonstrate that this technique of selective MR-guided PFCN blocks is feasible and suggest high technical success and effectiveness. Larger studies are needed to confirm our initial results.

Percutaneous punctures with MR imaging guidance: comparison between MR imaging-enhanced fluoroscopic guidance and real-time MR Imaging guidance

PURPOSE

To evaluate and compare the technical accuracy and feasibility of magnetic resonance (MR) imaging-enhanced fluoroscopic guidance and real-time MR imaging guidance for percutaneous puncture procedures in phantoms and animals.

MATERIALS AND METHODS

The experimental protocol was approved by the institutional animal care and use committee. Punctures were performed in phantoms, aiming for markers (20 each for MR imaging-enhanced fluoroscopic guidance and real-time MR imaging guidance), and pigs, aiming for anatomic landmarks (10 for MR imaging-enhanced fluoroscopic guidance and five for MR imaging guidance). To guide the punctures, T1-weighted three-dimensional (3D) MR images of the phantom or pig were acquired. Additional axial and coronal T2-weighted images were used to visualize the anatomy in the animals. For MR imaging-enhanced fluoroscopic guidance, phantoms and pigs were transferred to the fluoroscopic system after initial MR imaging and C-arm computed tomography (CT) was performed. C-arm CT and MR imaging data sets were coregistered. Prototype navigation software was used to plan a puncture path with use of MR images and to superimpose it on fluoroscopic images. For real-time MR imaging, an interventional MR imaging prototype for interactive real-time section position navigation was used. Punctures were performed within the magnet bore. After completion, 3D MR imaging was performed to evaluate the accuracy of insertions. Puncture durations were compared by using the log-rank test. The Mann-Whitney U test was applied to compare the spatial errors.

RESULTS

In phantoms, the mean total error was 8.6 mm ± 2.8 with MR imaging-enhanced fluoroscopic guidance and 4.0 mm ± 1.2 with real-time MR imaging guidance (P < .001). The mean puncture time was 2 minutes 10 seconds ± 44 seconds with MR imaging-enhanced fluoroscopic guidance and 37 seconds ± 14 with real-time MR imaging guidance (P < .001). In the animal study, a tolerable distance (<1 cm) between target and needle tip was observed for both MR imaging-enhanced fluoroscopic guidance and real-time MR imaging guidance. The mean total error was 7.7 mm ± 2.4 with MR imaging-enhanced fluoroscopic guidance and 7.9 mm ± 4.9 with real-time MR imaging guidance (P = .77). The mean puncture time was 5 minutes 43 seconds ± 2 minutes 7 seconds with MR imaging-enhanced fluoroscopic guidance and 5 minutes 14 seconds ± 2 minutes 25 seconds with real-time MR imaging guidance (P = .68).

CONCLUSION

Both MR imaging-enhanced fluoroscopic guidance and real-time MR imaging guidance demonstrated reasonable and similar accuracy in guiding needle placement to selected targets in phantoms and animals.

Augmented reality visualization with use of image overlay technology for MR imaging-guided interventions: assessment of performance in cadaveric shoulder and hip arthrography at 1.5 T

PURPOSE

To prospectively assess overlay technology in providing accurate and efficient targeting for magnetic resonance (MR) imaging-guided shoulder and hip joint arthrography.

MATERIALS AND METHODS

A prototype augmented reality image overlay system was used in conjunction with a clinical 1.5-T MR imager. A total of 24 shoulder joint and 24 hip joint injections were planned in 12 human cadavers. Two operators (A and B) participated, each performing procedures on different cadavers using image overlay guidance. MR imaging was used to confirm needle positions, monitor injections, and perform MR arthrography. Accuracy was assessed according to the rate of needle adjustment, target error, and whether the injection was intraarticular. Efficiency was assessed according to arthrography procedural time. Operator differences were assessed with comparison of accuracy and procedure times between the operators. Mann-Whitney U test and Fisher exact test were used to assess group differences.

RESULTS

Forty-five arthrography procedures (23 shoulders, 22 hips) were performed. Three joints had prostheses and were excluded. Operator A performed 12 shoulder and 12 hip injections. Operator B performed 11 shoulder and 10 hip injections. Needle adjustment rate was 13% (six of 45; one for operator A and five for operator B). Target error was 3.1 mm±1.2 (standard deviation) (operator A, 2.9 mm±1.4; operator B, 3.5 mm±0.9). Intraarticular injection rate was 100% (45 of 45). The average arthrography time was 14 minutes (range, 6-27 minutes; 12 minutes [range, 6-25 minutes] for operator A and 16 minutes [range, 6-27 min] for operator B). Operator differences were not significant with regard to needle adjustment rate (P=.08), target error (P=.07), intraarticular injection rate (P>.99), and arthrography time (P=.22).

CONCLUSION

Image overlay technology provides accurate and efficient MR guidance for successful shoulder and hip arthrography in human cadavers.

Informatics in radiology: Intuitive user interface for 3D image manipulation using augmented reality and a smartphone as a remote control

Although widely used as a pointing device on personal computers (PCs), the mouse was originally designed for control of two-dimensional (2D) cursor movement and is not suited to complex three-dimensional (3D) image manipulation. Augmented reality (AR) is a field of computer science that involves combining the physical world and an interactive 3D virtual world; it represents a new 3D user interface (UI) paradigm. A system for 3D and four-dimensional (4D) image manipulation has been developed that uses optical tracking AR integrated with a smartphone remote control. The smartphone is placed in a hard case (jacket) with a 2D printed fiducial marker for AR on the back. It is connected to a conventional PC with an embedded Web camera by means of WiFi. The touch screen UI of the smartphone is then used as a remote control for 3D and 4D image manipulation. Using this system, the radiologist can easily manipulate 3D and 4D images from computed tomography and magnetic resonance imaging in an AR environment with high-quality image resolution. Pilot assessment of this system suggests that radiologists will be able to manipulate 3D and 4D images in the reading room in the near future. Supplemental material available at http://radiographics.rsna.org/lookup/suppl/doi:10.1148/rg.324115086/-/DC1.

A guide to stereoscopic 3D displays in medicine

Stereoscopic displays can potentially improve many aspects of medicine. However, weighing the advantages and disadvantages of such displays remains difficult, and more insight is needed to evaluate whether stereoscopic displays are worth adopting. In this article, we begin with a review of monocular and binocular depth cues. We then apply this knowledge to examine how stereoscopic displays can potentially benefit diagnostic imaging, medical training, and surgery. It is apparent that the binocular depth information afforded by stereo displays 1) aid the detection of diagnostically relevant shapes, orientations, and positions of anatomical features, especially when monocular cues are absent or unreliable; 2) help novice surgeons orient themselves in the surgical landscape and perform complicated tasks; and 3) improve the three-dimensional anatomical understanding of students with low visual-spatial skills. The drawbacks of stereo displays are also discussed, including extra eyewear, potential three-dimensional misperceptions, and the hurdle of overcoming familiarity with existing techniques. Finally, we list suggested guidelines for the optimal use of stereo displays. We provide a concise guide for medical practitioners who want to assess the potential benefits of stereo displays before adopting them.

Percutaneous transhepatic cholangiodrainage under real-time MRI guidance: initial experience in an animal model

AIMS

To assess percutaneous transhepatic cholangiodrainage (PTCD) under real-time MRI-guidance and compare it to procedures performed under fluoroscopy.

METHODS

We developed an in vitro model for MRI-guided and conventional PTCD, using an animal organ set including liver and bile ducts placed in an MRI-compatible box and tested it in a 1.0-Tesla open MRI-scanner. Prototype 18G needles and guide wires, standard guide wires, dilatation bougies, and drainages were used (MRI-compatible). MRI-visualization was by means of a bFFE real-time sequence using a surface coil (Flex-L). Outcome measurements were success rates and time needed for bile duct puncture using real-time MRI-guidance versus conventional radiologic methods in the model. Cannulation and drainage placement were also analysed.

RESULTS

Fifty MRI-guided experiments were performed, leading to rapid (mean: 43s, range: 15-72s) and successful puncture and cannulation in 96% of procedures. Median drainage placement time was 321.5s (range: 241-411s). In 35 control experiments under fluoroscopy, puncture success was 69%, whereas times were significantly longer (mean 273s, range 45-631s).

CONCLUSIONS

Initial in vitro experience shows that PTCD can be successfully and rapidly performed under real-time MRI-guidance and demonstrates improved performance compared to the conventional radiologic approach.

MRI-guided musculoskeletal soft tissue interventions

This article highlights some of current state-of-the-art applications of interventional magnetic resonance imaging (MRI) technology pertaining to the musculoskeletal soft tissues. The rationale for the use of these techniques is to provide modes of minimally invasive diagnosis and/or therapy for a subset of patients whose lesions are not approachable by the traditional modes of interventional radiology and to introduce methods to mark subtle and infiltrative lesions to improve the outcomes of subsequent surgery or radiation therapy. These techniques build on the inherent attributes of MRI, particularly the high soft tissue contrast that made MRI the current mainstay diagnostic modality to identify and characterize musculoskeletal soft tissue lesions. The application of MRI technology to the musculoskeletal system, particularly for lesions related to the appendicular skeleton, does not typically suffer from the complexity related to involuntary organ motion. In addition, MRI-compatible versions of most of the needed instruments and devices for these interventions are currently available on commercial basis. Although musculoskeletal applications were not adopted early during the development of interventional MRI technology, we are likely to observe an increasing use of this technology for musculoskeletal soft tissue applications in the future.