Holographic mixed reality for planning transcatheter aortic valve replacement

BACKGROUND

Using three-dimensional (3D) modalities for optimal pre-procedure planning in transcatheter aortic valve replacement (TAVR) is critical for procedural success. However, current methods rely on visualizing images on a two-dimensional screen, using shading and colors to create the illusion of 3D, potentially impeding the accurate comprehension of the actual anatomy structures. In contrast, a new Mixed Reality (MxR) based software enables accurate 3D visualization, imaging manipulation, and quantification of measurements.

AIMS

The study aims to evaluate the feasibility, reproducibility, and accuracy of dimensions of the aortic valve complex as measured with a new holographic MxR software (ARTICOR®, Artiness srl, Milano, Italy) compared to a widely used software for pre-operative sizing and planning (3mensio Medical Imaging BV, Bilthoven, The Netherlands).

METHODS

This retrospective, observational, double-center study enrolled 100 patients with severe aortic stenosis who underwent cardiac computed tomography (CCT) before TAVR. The CCT datasets of volumetric aortic valve images were analyzed using 3Mensio and newly introduced MxR-based software.

RESULTS

Ninety-eight percent of the CCT datasets were successfully converted into holographic models. A higher level of agreement between the two software systems was observed for linear metrics (short, long, and average diameter). In comparison, agreement was lower for area, perimeter, and annulus-to-coronary ostia distance measurements. Notably, the annulus area, annular perimeter, left ventricular outflow tract (LVOT) area, and LVOT perimeter were significantly and consistently smaller with the MxR-based software compared to the 3Mensio. Excellent interobserver reliability was demonstrated for most measurements, especially for direct linear measurements.

CONCLUSIONS

Linear measurements of the aortic valve complex using MxR-based software are reproducible compared to the standard CCT dataset analyzed with 3Mensio. MxR-based software could represent an accurate tool for the pre-procedural planning of TAVR.

Using a novel three-dimensional holographic technology to perform open vascular surgery procedures

Augmented reality technology has been introduced during recent years into everyday clinical practice. Several surgical specialties have begun using such technology for preoperative planning as well as intraoperatively. Regarding vascular surgery, a limited number of reports have described the benefits, mainly for endovascular procedures. We aim to present a novel three-dimensional holographic system we used to perform an open vascular procedure.

Augmented Reality Navigation System for Biliary Interventional Procedures With Dynamic Respiratory Motion Correction

OBJECTIVE

Biliary interventional procedures require physicians to track the interventional instrument tip (Tip) precisely with X-ray image. However, Tip positioning relies heavily on the physicians' experience due to the limitations of X-ray imaging and the respiratory interference, which leads to biliary damage, prolonged operation time, and increased X-ray radiation.

METHODS

We construct an augmented reality (AR) navigation system for biliary interventional procedures. It includes system calibration, respiratory motion correction and fusion navigation. Firstly, the magnetic and 3D computed tomography (CT) coordinates are aligned through system calibration. Secondly, a respiratory motion correction method based on manifold regularization is proposed to correct the misalignment of the two coordinates caused by respiratory motion. Thirdly, the virtual biliary, liver and Tip from CT are overlapped to the corresponding position of the patient for dynamic virtual-real fusion.

RESULTS

Our system is respectively evaluated and achieved an average alignment error of 0.75 ± 0.17 mm and 2.79 ± 0.46 mm on phantoms and patients. The navigation experiments conducted on phantoms achieve an average Tip positioning error of 0.98 ± 0.15 mm and an average fusion error of 1.67 ± 0.34 mm after correction.

CONCLUSION

Our system can automatically register the Tip to the corresponding location in CT, and dynamically overlap the 3D virtual model onto patients to provide accurate and intuitive AR navigation.

SIGNIFICANCE

This study demonstrates the clinical potential of our system by assisting physicians during biliary interventional procedures. Our system enables dynamic visualization of virtual model on patients, reducing the reliance on contrast agents and X-ray usage.

The Influence of Individualized Three-Dimensional Holographic Models on Patients' Knowledge Qualified for Intervention in the Treatment of Peripheral Arterial Disease (PAD)

We sought to determine the role of the patient-specific, three-dimensional (3D) holographic vascular model in patient medical knowledge and its influence on obtaining a more conscious informed consent process for percutaneous balloon angioplasty (PTA). Patients with peripheral arterial disease who had been scheduled for PTA were enrolled in the study. Information regarding the primary disease, planned procedure, and informed consent was recorded in typical fashion. Subsequently, the disease and procedure details were presented to the patient, showing the patients their individual model. A patient and medical supervisor equipped with mixed reality headsets could both simultaneously manipulate the hologram using gestures. The holographic 3D model had been created on a scale of 1:1 based on computed tomography scans. The patient's knowledge was tested by the completion of a questionnaire before and after the interaction in a mixed reality environment. Seventy-nine patients manipulated arterial holograms in mixed reality head-mounted devices. Before the 3D holographic artery model interaction, the mean ± standard deviation score of the knowledge test was 2.95 ± 1.21 points. After the presentation, the score had increased to 4.39 ± 0.82, with a statistically significant difference (p = 0.0000) between the two scores. Using a Likert scale from 1 to 5, the patients had scored the use of the 3D holographic model at 3.90 points regarding its usefulness in comprehending their medical condition; at 4.04 points regarding the evaluation of the holograms as helpful in understanding the course of surgery; and rated the model at 1.99 points in reducing procedure-related stress. Using a nominal scale (know or don't know), the patients had self-assessed their knowledge of the procedure before and after the 3D model presentation, with a score of 6.29 ± 2.01 and 8.39 ± 1.54, respectively. The study group tolerated the use of head-mounted devices. Only one patient had nausea and dizziness, while four patients experienced transient eye pain. The 3D holographic arterial model aided in the understanding of patients' knowledge regarding the disease and procedure, making the informed consent process more conscious. The holograms improved the patient's self-consciousness. Mixed reality headset-related complications were rare and within acceptable rates.

Mixed reality for the assessment of aortoiliac anatomy in patients with abdominal aortic aneurysm prior to open and endovascular repair: Feasibility and interobserver agreement

OBJECTIVES

The objective is to evaluate the feasibility and interobserver agreement of a Mixed Reality Viewer (MRV) in the assessment of aortoiliac vascular anatomy of abdominal aortic aneurysm (AAA) patients.

METHODS

Fifty preoperative computed tomography angiographies (CTAs) of AAA patients were included. CTAs were assessed in a mixed reality (MR) environment with respect to aortoiliac anatomy according to a standardized protocol by two experienced observers (Mixed Reality Viewer, MRV, Brainlab AG, Germany). Additionally, all CTAs were independently assessed applying the same protocol by the same observers using a conventional DICOM viewer on a two-dimensional screen with multi-planar reconstructions (Conventional viewer, CV, GE Centricity PACS RA1000 Workstation, GE, United States). The protocol included four sets of items: calcification, dilatation, patency, and tortuosity as well as the number of lumbar and renal arteries. Interobserver agreement (IA, Cohen's Kappa, κ) was calculated for every item set.

RESULTS

All CTAs could successfully be displayed in the MRV (100%). The MRV demonstrated equal or better IA in the assessment of anterior and posterior calcification (κMRV: 0.68 and 0.61, κCV: 0.33 and 0.45, respectively) as well as tortuosity (κMRV: 0.60, κCV: 0.48) and dilatation (κMRV: 0.68, κCV: 0.67). The CV demonstrated better IA in the assessment of patency (κMRV: 0.74, κCV: 0.93). The CV also identified significantly more lumbar arteries (CV: 379, MRV: 239, p < 0.01).

CONCLUSIONS

The MRV is a feasible imaging viewing technology in clinical routine. Future efforts should aim at improving hologram quality and enabling accurate registration of the hologram with the physical patient.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

Virtual and augmented reality in critical care medicine: the patient's, clinician's, and researcher's perspective

Virtual reality (VR) and augmented reality (AR) are aspiring, new technologies with increasing use in critical care medicine. While VR fully immerses the user into a virtual three-dimensional space, AR adds overlaid virtual elements into a real-world environment. VR and AR offer great potential to improve critical care medicine for patients, relatives and health care providers. VR may help to ameliorate anxiety, stress, fear, and pain for the patient. It may assist patients in mobilisation and rehabilitation and can improve communication between all those involved in the patient's care. AR can be an effective tool to support continuous education of intensive care medicine providers, and may complement traditional learning methods to acquire key practical competences such as central venous line placement, cardiopulmonary resuscitation, extracorporeal membrane oxygenation device management or endotracheal intubation. Currently, technical, human, and ethical challenges remain. The adaptation and integration of VR/AR modalities into useful clinical applications that can be used routinely on the ICU is challenging. Users may experience unwanted side effects (so-called "cybersickness") during VR/AR sessions, which may limit its applicability. Furthermore, critically ill patients are one of the most vulnerable patient groups and warrant special ethical considerations if new technologies are to be introduced into their daily care. To date, most studies involving AR/VR in critical care medicine provide only a low level of evidence due to their research design. Here we summarise background information, current developments, and key considerations that should be taken into account for future scientific investigations in this field.

Mixed Reality in der Gefäßchirurgie – ein Scoping Review

Hintergrund „Mixed Reality“ (MR) erlaubt die Projektion von virtuellen Objekten in das Sichtfeld des Anwenders durch ein Head-mounted Display (HMD). Im gefäßchirurgischen Behandlungsspektrum könnten MR-Anwendungen in Zukunft einen Nutzen darstellen. Im folgenden Scoping Review soll eine Orientierung über die aktuelle Anwendung der genannten Technologien im Bereich der Gefäßchirurgie gegeben und Forschungsziele für die Zukunft definiert werden. Material und Methoden Es erfolgte eine systematische Literaturrecherche in PubMed (MEDLINE) mit den Suchbegriffen „aorta“, „intervention“, „endovsacular intervention“, „vascular surgery“, „aneurysm“, „endovascular“, „vascular access“ jeweils in Kombination mit „mixed reality“ oder „augmented reality“. Die Suche erfolgte nach PRISMA-Leitlinie (Preferred Reporting Items for Systematic reviews and Meta-Analyses) für Scoping Reviews. Ergebnisse Aus 547 Literaturstellen konnten 8 relevante Studien identifiziert werden. Die Suchergebnisse konnten in 2 Anwendungskategorien eingeteilt werden: (1) MR mit dem Ziel des Informationsmanagements und zur Verbesserung der periprozeduralen Ergonomie gefäßchirurgischer Eingriffe (n = 3) sowie (2) MR mit dem Ziel der intraoperativen Navigation bei gefäßchirurgischen Eingriffen (n = 5). Die Registrierung des physischen Patienten mit dem virtuellen Objekt und das Tracking von Instrumenten in der MR-Umgebung zur intraoperativen Navigation ist dabei im Fokus des wissenschaftlichen Interesses und konnte technisch erfolgreich am Phantom- und Tiermodell gezeigt werden. Die bisher vorgestellten Methoden sind jedoch mit hohem infrastrukturellem Aufwand und relevanten Limitationen verbunden. Schlussfolgerung Der Einsatz von MR im Bereich der Gefäßchirurgie ist grundsätzlich vielversprechend. Für die Zukunft sollten alternative, pragmatische Registrierungsmethoden mit entsprechender Quantifizierung des Positionierungsfehlers angestrebt werden. Die entwickelten Soft- und Hardwarelösungen sollten auf das Anforderungsprofil der Gefäßchirurgie angepasst werden. Das elektromagnetische Instrumenten-Tracking erscheint als sinnvolle, komplementäre Technologie zur Umsetzung der MR-assistierten Navigation.

Augmented Reality in Vascular and Endovascular Surgery: Scoping Review

Background

Technological advances have transformed vascular intervention in recent decades. In particular, improvements in imaging and data processing have allowed for the development of increasingly complex endovascular and hybrid interventions. Augmented reality (AR) is a subject of growing interest in surgery, with the potential to improve clinicians’ understanding of 3D anatomy and aid in the processing of real-time information. This study hopes to elucidate the potential impact of AR technology in the rapidly evolving fields of vascular and endovascular surgery.

Objective

The aim of this review is to summarize the fundamental concepts of AR technologies and conduct a scoping review of the impact of AR and mixed reality in vascular and endovascular surgery.

Methods

A systematic search of MEDLINE, Scopus, and Embase was performed in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. All studies written in English from inception until January 8, 2021, were included in the search. Combinations of the following keywords were used in the systematic search string: (“augmented reality” OR “hololens” OR “image overlay” OR “daqri” OR “magic leap” OR “immersive reality” OR “extended reality” OR “mixed reality” OR “head mounted display”) AND (“vascular surgery” OR “endovascular”). Studies were selected through a blinded process between 2 investigators (JE and AS) and assessed using data quality tools.

Results

AR technologies have had a number of applications in vascular and endovascular surgery. Most studies (22/32, 69%) used 3D imaging of computed tomography angiogram–derived images of vascular anatomy to augment clinicians’ anatomical understanding during procedures. A wide range of AR technologies were used, with heads up fusion imaging and AR head-mounted displays being the most commonly applied clinically. AR applications included guiding open, robotic, and endovascular surgery while minimizing dissection, improving procedural times, and reducing radiation and contrast exposure.

Conclusions

AR has shown promising developments in the field of vascular and endovascular surgery, with potential benefits to surgeons and patients alike. These include reductions in patient risk and operating times as well as in contrast and radiation exposure for radiological interventions. Further technological advances are required to overcome current limitations, including processing capacity and vascular deformation by instrumentation.

Bioengineering, augmented reality, and robotic surgery in vascular surgery: A literature review

Biomedical engineering integrates a variety of applied sciences with life sciences to improve human health and reduce the invasiveness of surgical procedures. Technological advances, achieved through biomedical engineering, have contributed to significant improvements in the field of vascular and endovascular surgery. This paper aims to review the most cutting-edge technologies of the last decade involving the use of augmented reality devices and robotic systems in vascular surgery, highlighting benefits and limitations. Accordingly, two distinct literature surveys were conducted through the PubMed database: the first review provides a comprehensive assessment of augmented reality technologies, including the different techniques available for the visualization of virtual content (11 papers revised); the second review collects studies with bioengineering content that highlight the research trend in robotic vascular surgery, excluding works focused only on the clinical use of commercially available robotic systems (15 papers revised). Technological flow is constant and further advances in imaging techniques and hardware components will inevitably bring new tools for a clinical translation of innovative therapeutic strategies in vascular surgery.

Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

A look to the future: Pandemic-induced digital technologies in vascular surgery

Like many areas of medicine, vascular surgery has been transformed by the COVID-19 (coronavirus disease 2019) pandemic. Public health precautions to minimize disease transmission have led to reduced attendance at hospitals and clinics in elective and emergency settings; fewer face-to-face and hands-on clinical interactions; and increased reliance on telemedicine, virtual attendance, investigations, and digital therapeutics. However, a “silver lining” to the COVID-19 pandemic may be the mainstream acceptance and acceleration of telemedicine, remote monitoring, digital health technology, and three-dimensional technologies, such as three-dimensional printing and virtual reality, by connecting health care providers to patients in a safe, reliable, and timely manner, and supplanting face-to-face surgical simulation and training. This review explores the impact of these changes in the delivery of vascular surgical care.

Virtual and Augmented Reality in Cardiovascular Care: State-of-the-Art and Future Perspectives

Applications of virtual reality (VR) and augmented reality (AR) assist both health care providers and patients in cardiovascular education, complementing traditional learning methods. Interventionalists have successfully used VR to plan difficult procedures and AR to facilitate complex interventions. VR/AR has already been used to treat patients, during interventions in rehabilitation programs and in immobilized intensive care patients. There are numerous additional potential applications in the catheterization laboratory. By using AR, interventionalists could combine visual fluoroscopy information projected and registered on the patient body with data derived from preprocedural imaging and live fusion of different imaging modalities such as fluoroscopy with echocardiography. Persistent technical challenges to overcome include the integration of different imaging modalities into VR/AR and the harmonization of data flow and interfaces. Cybersickness might exclude some patients and users from the potential benefits of VR/AR. Critical ethical considerations arise in the application of VR/AR in vulnerable patients. In addition, digital applications must not distract physicians from the patient. It is our duty as physicians to participate in the development of these innovations to ensure a virtual health reality benefit for our patients in a real-world setting. The purpose of this review is to summarize the current and future role of VR and AR in different fields within cardiology, its challenges, and perspectives.

Three-Dimensional Holographic Guidance, Navigation, and Control (3D-GNC) for Endograft Positioning in Porcine Aorta: Feasibility Comparison With 2-Dimensional X-Ray Fluoroscopy

OBJECTIVES

Intraprocedural deployment of endovascular devices during complex aortic repair with 2-dimensional (2D) x-ray fluoroscopic guidance poses challenges in terms of accurate delivery system positioning and increased risk of x-ray radiation exposure with prolonged fluoroscopy times, particularly in unfavorable anatomy. The objective of this study was to assess feasibility of using an augmented reality (AR) system to position and orient a modified aortic endograft delivery system in comparison with standard fluoroscopy.

MATERIALS AND METHODS

The 3-dimensional guidance, navigation, and control (3D-GNC) prototype system was developed for eventual integration with the Intra-Operative Positioning System (IOPS, Centerline Biomedical, Cleveland, OH) to project spatially registered 3D holographic representations of the subject-specific aorta for intraoperative guidance and coupled with an electromagnetically (EM) tracked delivery system for intravascular navigation. Numerical feedback for controlling the endograft landing zone distance and ostial alignment was holographically projected on the operative field. Visualization of the holograms was provided via a commercially available AR headset. A Zenith Spiral-Z AAA limb stent-graft was modified with a scallop, 6 degree-of-freedom EM sensor for tracking, and radiopaque markers for fluoroscopic visualization. In vivo, 10 interventionalists independently positioned and oriented the delivery system to the ostia of renal or visceral branch vessels in anesthetized swine via open femoral artery access using 3D-GNC and standard fluoroscopic guidance. Procedure time, fluoroscopy time, cumulative air kerma, and contrast material volume were recorded for each technique. Positioning and orientation accuracy was determined by measuring the target landing-zone distance error (δ_LZE) and the scallop-ostium angular alignment error (θ_SOE) using contrast-enhanced cone beam computed tomography imaging after each positioning for each technique. Mean, standard deviation, and standard error are reported for the performance variables, and Student's t tests were used to evaluate statistically significant differences in performance mean values of 3D-GNC and fluoroscopy.

RESULTS

Technical success for the use of 3D-GNC to orient and position the endovascular device at each renal-visceral branch ostium was 100%. 3D-GNC resulted in 56% decrease in procedure time in comparison with standard fluoroscopic guidance (p<0.001). The 3D-GNC system was used without fluoroscopy or contrast-dye administration. Positioning accuracy was comparable for both techniques (p=0.86), while overall orientation accuracy was improved with the 3D-GNC system by 41.5% (p=0.008).

CONCLUSIONS

The holographic 3D-GNC system demonstrated improved accuracy of aortic stent-graft positioning with significant reductions in fluoroscopy time, contrast-dye administration, and procedure time.

Applications of Head-Mounted Displays and Smart Glasses in Vascular Surgery

OBJECTIVES

Advances in virtual, augmented and mixed reality have led to the development of wearable technologies including head mounted displays (HMD) and smart glasses. While there is a growing interest on their potential applications in health, only a few studies have addressed so far their use in vascular surgery. The aim of this review was to summarize the fundamental notions associated with these technologies and to discuss potential applications and current limits for their use in vascular surgery.

METHODS

A comprehensive literature review was performed to introduce the fundamental concepts and provide an overview of applications of HMD and smart glasses in surgery.

RESULTS

HMD and smart glasses demonstrated a potential interest for the education of surgeons including anatomical teaching, surgical training, teaching and telementoring. Applications for pre-surgical planning have been developed in general and cardiac surgery and could be transposed for a use in vascular surgery. The use of wearable technologies in the operating room has also been investigated in both general and cardiovascular surgery and demonstrated its potential interest for image-guided surgery and data collection.

CONCLUSION

Studies performed so far represent a proof of concept of the interest of HMD and smart glasses in vascular surgery for education of surgeons and for surgical practice. Although these technologies exhibited encouraging results for applications in vascular surgery, technical improvements and further clinical research in large series are required before hoping using them in daily clinical practice.