Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease

Clinical validation of 3D virtual modelling for laparoscopic complete mesocolic excision with central vascular ligation for proximal colon cancer

INTRODUCTION

Laparoscopic Complete Mesocolic Excision (CME) with Central Vascular Ligation (CVL) in colon cancer surgery has not been broadly adopted in part because of safety concerns. Pre-operative 3-D virtual modelling (3DVM) may help but needs validation.

METHODS

3DVM were routinely constructed from CT mesenteric angiograms (CTMA) using a commercial service (Visible Patient, Strasbourg, France) for consecutive patients during our CMECVL learning curve over three years. 3DVMs were independently checked versus CTMA and operative findings. CMECVL outcomes were compared versus other patients undergoing standard mesocolic excision (SME) surgery laparoscopically in the same hospital as control. Stakeholders were studied regarding 3DVM use and usefulness (including detail retention) versus CTMA and a physical 3D-printed model.

RESULTS

26 patients underwent 3DVM with intraoperative display during laparoscopic CMECVL within existing workflows. 3DVM accuracy was 96 % re arteriovenous variations at patient level versus CTMA/intraoperative findings including accessory middle colic artery identification in three patients. Twenty-two laparoscopic CMECVL with 3DVM cases were compared with 49 SME controls (age 69 ± 10 vs 70.9 ± 11 years, 55 % vs 53 % males). There were no intraoperative complications with CMECVL and similar 30-day postoperative morbidity (30 % vs 29 %), hospital stay (9 ± 3 vs 12 ± 13 days), 30-day readmission (6 % vs 4 %) and reoperation (0 % vs 4 %) rates. Intraoperative times were longer (215.7 ± 43.9 vs 156.9 ± 52.9 min, p=<0.01) but decreased significantly over time. 3DVM surveys (n = 98, 20 surgeons, 48 medical students, 30 patients/patient relatives) and comparative study revealed majority endorsement (90 %) and favour (87 %).

CONCLUSION

3DVM use was positively validated for laparoscopic CMECVL and valued by clinicians, students, and patients alike.

Cardiac anatomic digital twins: findings from a single national centre

Aims

New three-dimensional cardiac visualization technologies are increasingly employed for anatomic digital twins in pre-operative planning. However, the role and influence of extended reality (virtual, augmented, or mixed) within heart team settings remain unclear. We aimed to assess the impact of mixed reality visualization of the intracardiac anatomy on surgical decision-making in patients with complex heart defects.

Methods and results

Between September 2020 and December 2022, we recruited 50 patients and generated anatomic digital twins and visualized them in mixed reality. These anatomic digital twins were presented to the heart team after initial decisions were made using standard visualization methods. Changes in the surgical strategy were recorded. Additionally, heart team members rated their mixed reality experience through a questionnaire, and post-operative outcomes were registered. Anatomic digital twins changed the initially decided upon surgical strategies for 68% of cases. While artificial intelligence facilitated the rapid creation of digital anatomic twins, manual corrections were always necessary.

Conclusion

In conclusion, mixed reality anatomic digital twins added information to standard visualization methods and significantly influenced surgical planning, with evidence that these strategies can be implemented safely without additional risk.

Applications and advances of immersive technology in cardiology

Different forms of immersive technology, such as Virtual Reality (VR) and Augmented Reality (AR), are getting increasingly invested in medicine. Advances in head-mounted display technology, processing, and rendering power have demonstrated the increasing utility of immersive technology in medicine and the healthcare environment. There are a growing number of publications on using immersive technology in cardiology. We reviewed the articles published within the last decade that reported case studies or research that uses or investigates the application of immersive technology in the broad field of cardiology - from education to preoperative planning and intraoperative guidance. We summarized the advantages and disadvantages of using AR and VR for various application categories. Our review highlights the need for a robust assessment of the effectiveness of the methods and discusses the technical limitations that hinder the complete integration of AR and VR in cardiology, including cost-effectiveness and regulatory compliance. Despite the limitations and gaps that have inhibited us from benefiting from immersive technologies' full potential in cardiology settings to date, its promising, impactful future for standard cardiovascular care is undoubted.

Utilization of facial fat grafting augmented reality guidance system in facial soft tissue defect reconstruction

BACKGROUND

Successfully restoring facial contours continues to pose a significant challenge for surgeons. This study aims to utilize head-mounted display-based augmented reality (AR) navigation technology for facial soft tissue defect reconstruction and to evaluate its accuracy and effectiveness, exploring its feasibility in craniofacial surgery.

METHODS

Hololens 2 was utilized to construct the AR guidance system for facial fat grafting. Twenty artificial cases with facial soft tissue defects were randomly assigned to Group A and Group B, undergoing filling surgeries with the AR guidance system and conventional methods, respectively. All postoperative three-dimensional models were superimposed onto virtual plans to evaluate the accuracy of the system versus conventional filling methods. Additionally, procedure completion time was recorded to assess system efficiency relative to conventional methods.

RESULTS

The error in facial soft tissue defect reconstruction assisted by the system in Group A was 2.09 ± 0.56 mm, significantly lower than the 3.23 ± 1.15 mm observed with conventional methods in Group B (p < 0.05). Additionally, the time required for facial defect filling reconstruction using the system in Group A was 25.45 ± 2.58 min, markedly shorter than the 37.05 ± 3.34 min needed with conventional methods in Group B (p < 0.05).

CONCLUSION

The visual navigation offered by the fat grafting AR guidance system presents obvious advantages in facial soft tissue defect reconstruction, facilitating enhanced precision and efficiency in these filling procedures.

Application of Mixed/Augmented Reality in Interventional Cardiology

This review explores the transformative applications of augmented reality (AR) and mixed reality (MR) technologies in interventional cardiology. The integration of these cutting-edge systems offers unprecedented potential to enhance visualization, guidance, and outcomes during complex cardiac interventional procedures. This review examines four key domains: (1) medical AR/MR systems and technological foundations; (2) clinical applications across procedures like TAVI, PCI, and electrophysiology mapping; (3) ongoing technology development and validation efforts; and (4) educational and training applications for fostering essential skills. By providing an in-depth analysis of the benefits, challenges, and future directions, this work elucidates the paradigm shift catalyzed by AR and MR in advancing interventional cardiology practices. Through meticulous exploration of technological, clinical, and educational implications, this review underscores the pivotal role of these innovative technologies in optimizing procedural guidance, improving patient outcomes, and driving innovation in cardiovascular care.

Patient-specific tissue engineered vascular graft for aortic arch reconstruction

Objectives

The complexity of aortic arch reconstruction due to diverse 3-dimensional geometrical abnormalities is a major challenge. This study introduces 3-dimensional printed tissue-engineered vascular grafts, which can fit patient-specific dimensions, optimize hemodynamics, exhibit antithrombotic and anti-infective properties, and accommodate growth.

Methods

We procured cardiac magnetic resonance imaging with 4-dimensional flow for native porcine anatomy (n = 10), from which we designed tissue-engineered vascular grafts for the distal aortic arch, 4 weeks before surgery. An optimal shape of the curved vascular graft was designed using computer-aided design informed by computational fluid dynamics analysis. Grafts were manufactured and implanted into the distal aortic arch of porcine models, and postoperative cardiac magnetic resonance imaging data were collected. Pre- and postimplant hemodynamic data and histology were analyzed.

Results

Postoperative magnetic resonance imaging of all pigs with 1:1 ratio of polycaprolactone and poly-L-lactide-co-ε-caprolactone demonstrated no specific dilatation or stenosis of the graft, revealing a positive growth trend in the graft area from the day after surgery to 3 months later, with maintaining a similar shape. The peak wall shear stress of the polycaprolactone/poly-L-lactide-co-ε-caprolactone graft portion did not change significantly between the day after surgery and 3 months later. Immunohistochemistry showed endothelization and smooth muscle layer formation without calcification of the polycaprolactone/poly-L-lactide-co-ε-caprolactone graft.

Conclusions

Our patient-specific polycaprolactone/poly-L-lactide-co-ε-caprolactone tissue-engineered vascular grafts demonstrated optimal anatomical fit maintaining ideal hemodynamics and neotissue formation in a porcine model. This study provides a proof of concept of patient-specific tissue-engineered vascular grafts for aortic arch reconstruction.

Cardiovascular Computed Tomography in the Diagnosis of Cardiovascular Disease: Beyond Lumen Assessment

Cardiovascular CT is being widely used in the diagnosis of cardiovascular disease due to the rapid technological advancements in CT scanning techniques. These advancements include the development of multi-slice CT, from early generation to the latest models, which has the capability of acquiring images with high spatial and temporal resolution. The recent emergence of photon-counting CT has further enhanced CT performance in clinical applications, providing improved spatial and contrast resolution. CT-derived fractional flow reserve is superior to standard CT-based anatomical assessment for the detection of lesion-specific myocardial ischemia. CT-derived 3D-printed patient-specific models are also superior to standard CT, offering advantages in terms of educational value, surgical planning, and the simulation of cardiovascular disease treatment, as well as enhancing doctor-patient communication. Three-dimensional visualization tools including virtual reality, augmented reality, and mixed reality are further advancing the clinical value of cardiovascular CT in cardiovascular disease. With the widespread use of artificial intelligence, machine learning, and deep learning in cardiovascular disease, the diagnostic performance of cardiovascular CT has significantly improved, with promising results being presented in terms of both disease diagnosis and prediction. This review article provides an overview of the applications of cardiovascular CT, covering its performance from the perspective of its diagnostic value based on traditional lumen assessment to the identification of vulnerable lesions for the prediction of disease outcomes with the use of these advanced technologies. The limitations and future prospects of these technologies are also discussed.

Revaiah P, Stanuch M, Skalski A, Onuma Y, Serruys PW. Feasibility and accuracy of real-time 3D-holographic graft length measurements

Aims

Mixed reality (MR) holograms can display high-definition images while preserving the user's situational awareness. New MR software can measure 3D objects with gestures and voice commands; however, these measurements have not been validated. We aimed to assess the feasibility and accuracy of using 3D holograms for measuring the length of coronary artery bypass grafts.

Methods and results

An independent core lab analyzed follow-up computer tomography coronary angiograms performed 30 days after coronary artery bypass grafting in 30 consecutive cases enrolled in the FASTTRACK CABG trial. Two analysts, blinded to clinical information, performed holographic reconstruction and measurements using the CarnaLife Holo software (Medapp, Krakow, Poland). Inter-observer agreement was assessed in the first 20 cases. Another analyst performed the validation measurements using the CardIQ W8 CT system (GE Healthcare, Milwaukee, Wisconsin). Seventy grafts (30 left internal mammary artery grafts, 31 saphenous vein grafts, and 9 right internal mammary artery grafts) were measured. Holographic measurements were feasible in 97.1% of grafts and took 3 minutes 36 s ± 50.74 s per case. There was an excellent inter-observer agreement [interclass correlation coefficient (ICC) 0.99 (0.97-0.99)]. There was no significant difference between the total graft length on hologram and CT [187.5 mm (157.7-211.4) vs. 183.1 mm (156.8-206.1), P = 0.50], respectively. Hologram and CT measurements are highly correlated (r = 0.97, P < 0.001) with an excellent agreement [ICC 0.98 (0.97-0.99)].

Conclusion

Real-time holographic measurements are feasible, quick, and accurate even for tortuous bypass grafts. This new methodology can empower clinicians to visualize and measure 3D images by themselves and may provide insights for procedural strategy.

Merging virtual and physical experiences: extended realities in cardiovascular medicine

Technological advancement and the COVID-19 pandemic have brought virtual learning and working into our daily lives. Extended realities (XR), an umbrella term for all the immersive technologies that merge virtual and physical experiences, will undoubtedly be an indispensable part of future clinical practice. The intuitive and three-dimensional nature of XR has great potential to benefit healthcare providers and empower patients and physicians. In the past decade, the implementation of XR into cardiovascular medicine has flourished such that it is now integrated into medical training, patient education, pre-procedural planning, intra-procedural visualization, and post-procedural care. This review article discussed how XR could provide innovative care and complement traditional practice, as well as addressing its limitations and considering its future perspectives.

The assessment of presence and performance in an AR environment for motor imitation learning: A case-study on violinists

The acquisition of advanced gestures is a challenge in various domains of proficient sensorimotor performance. For example, orchestral violinists must move in sync with the lead violinist's gestures. To help train these gestures, an educational music play-back system was developed using a HoloLens 2 simulated AR environment and an avatar representation of the lead violinist. This study aimed to investigate the impact of using a 2D or 3D representation of the lead violinist's avatar on students' learning experience in the AR environment. To assess the learning outcome, the study employed a longitudinal experiment design, in which eleven participants practiced two pieces of music in four trials, evenly spaced over a month. Participants were asked to mimic the avatar's gestures as closely as possible when it came to using the bow, including bowing, articulations, and dynamics. The study compared the similarities between the avatar's gestures and those of the participants at the biomechanical level, using motion capture measurements, as well as the smoothness of the participants' movements. Additionally, presence and perceived difficulty were assessed using questionnaires. The results suggest that using a 3D representation of the avatar leads to better gesture resemblance and a higher experience of presence compared to a 2D representation. The 2D representation, however, showed a learning effect, but this was not observed in the 3D condition. The findings suggest that the 3D condition benefits from stereoscopic information that enhances spatial cognition, making it more effective in relation to sensorimotor performance. Overall, the 3D condition had a greater impact on performance than on learning. This work concludes with recommendations for future efforts directed towards AR-based advanced gesture training to address the challenges related to measurement methodology and participants' feedback on the AR application.

Use of a mixed reality system for navigational mapping during cardiac electrophysiological testing does not prolong case duration: A subanalysis from the Cardiac Augmented REality study

Background

CommandEP™ is a mixed reality (MXR) system for cardiac electrophysiological (EP) procedures that provides a real-time 3-dimensional digital image of cardiac geometry and catheter locations. In a previous study, physicians using the system demonstrated improved navigational accuracy. This study investigated the impact of the CommandEP system on EP procedural times compared to the standard-of-care electroanatomic mapping system (EAMS) display.

Objective

The purpose of this retrospective case-controlled analysis was to evaluate the impact of a novel MXR interface on EP procedural times compared to a case-matched cohort.

Methods

Cases from the Cardiac Augmented REality (CARE) study were matched for diagnosis and weight using a contemporary cohort. Procedural time was compared from the roll-in and full implementation cohort. During routine EP procedures, operators performed tasks during the postablation waiting phase, including creation of cardiac geometry and 5-point navigation under 2 conditions: (1) EAMS first; and (2) CommandEP.

Results

From a total of 16 CARE study patients, the 10 full implementation patients were matched to a cohort of 20 control patients (2 controls:1 CARE, matched according to pathology and age/weight). No statistical difference in total case times between CARE study patients vs control group (118 ± 29 minutes vs 97 ± 20 minutes; P = .07) or fluoroscopy times (6 ± 4 minutes vs 7 ± 6 minutes; P = .9). No significant difference in case duration for CARE study patients comparing roll-in vs full-implementation cohort (121 ± 26 minutes vs 118 ± 29 minutes; P = .96). CommandEP wear time during cases was significantly longer in full implementation cases (53 ± 24 minutes vs 24 ± 5 minutes; P = .0009). During creation of a single cardiac geometry, no significant time difference was noted between CommandEP vs EAMS (284 ± 45 seconds vs 268 ± 43 seconds; P = .1) or fluoroscopy use (9 ± 19 seconds vs 6 ± 18 seconds; P = .25). During point navigation tasks, there was no difference in total time (CommandEP 31 ± 14 seconds vs EAMS 28 ± 15 seconds; P = .16) or fluoroscopy time (CommandEP 0 second vs EAMS 0 second).

Conclusion

MXR did not prolong overall procedural time compared to a matched cohort. There was no prolongation in study task completion time. Future studies with experienced CommandEP users directly assessing procedural time and task completion time in a randomized study population would be of interest.

Exploring the Potential of Three-Dimensional Imaging, Printing, and Modeling in Pediatric Surgical Oncology: A New Era of Precision Surgery

Pediatric surgical oncology is a technically challenging field that relies on CT and MRI as the primary imaging tools for surgical planning. However, recent advances in 3D reconstructions, including Cinematic Rendering, Volume Rendering, 3D modeling, Virtual Reality, Augmented Reality, and 3D printing, are increasingly being used to plan complex cases bringing new insights into pediatric tumors to guide therapeutic decisions and prognosis in different pediatric surgical oncology areas and locations including thoracic, brain, urology, and abdominal surgery. Despite this, challenges to their adoption remain, especially in soft tissue-based specialties such as pediatric surgical oncology. This work explores the main innovative imaging reconstruction techniques, 3D modeling technologies (CAD, VR, AR), and 3D printing applications through the analysis of three real cases of the most common and surgically challenging pediatric tumors: abdominal neuroblastoma, thoracic inlet neuroblastoma, and a bilateral Wilms tumor candidate for nephron-sparing surgery. The results demonstrate that these new imaging and modeling techniques offer a promising alternative for planning complex pediatric oncological cases. A comprehensive analysis of the advantages and limitations of each technique has been carried out to assist in choosing the optimal approach.

Extended reality for procedural planning and guidance in structural heart disease - a review of the state-of-the-art

Extended reality (XR), which encompasses virtual, augmented and mixed reality, is an emerging medical imaging display platform which enables intuitive and immersive interaction in a three-dimensional space. This technology holds the potential to enhance understanding of complex spatial relationships when planning and guiding cardiac procedures in congenital and structural heart disease moving beyond conventional 2D and 3D image displays. A systematic review of the literature demonstrates a rapid increase in publications describing adoption of this technology. At least 33 XR systems have been described, with many demonstrating proof of concept, but with no specific mention of regulatory approval including some prospective studies. Validation remains limited, and true clinical benefit difficult to measure. This review describes and critically appraises the range of XR technologies and its applications for procedural planning and guidance in structural heart disease while discussing the challenges that need to be overcome in future studies to achieve safe and effective clinical adoption.

The HoloLens in medicine: A systematic review and taxonomy

The HoloLens (Microsoft Corp., Redmond, WA), a head-worn, optically see-through augmented reality (AR) display, is the main player in the recent boost in medical AR research. In this systematic review, we provide a comprehensive overview of the usage of the first-generation HoloLens within the medical domain, from its release in March 2016, until the year of 2021. We identified 217 relevant publications through a systematic search of the PubMed, Scopus, IEEE Xplore and SpringerLink databases. We propose a new taxonomy including use case, technical methodology for registration and tracking, data sources, visualization as well as validation and evaluation, and analyze the retrieved publications accordingly. We find that the bulk of research focuses on supporting physicians during interventions, where the HoloLens is promising for procedures usually performed without image guidance. However, the consensus is that accuracy and reliability are still too low to replace conventional guidance systems. Medical students are the second most common target group, where AR-enhanced medical simulators emerge as a promising technology. While concerns about human-computer interactions, usability and perception are frequently mentioned, hardly any concepts to overcome these issues have been proposed. Instead, registration and tracking lie at the core of most reviewed publications, nevertheless only few of them propose innovative concepts in this direction. Finally, we find that the validation of HoloLens applications suffers from a lack of standardized and rigorous evaluation protocols. We hope that this review can advance medical AR research by identifying gaps in the current literature, to pave the way for novel, innovative directions and translation into the medical routine.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND

Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike.

OBJECTIVE

In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions.

METHODS

A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment.

RESULTS

A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education.

CONCLUSIONS

The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority.

Clinical Applications of Mixed Reality and 3D Printing in Congenital Heart Disease

Understanding the anatomical features and generation of realistic three-dimensional (3D) visualization of congenital heart disease (CHD) is always challenging due to the complexity and wide spectrum of CHD. Emerging technologies, including 3D printing and mixed reality (MR), have the potential to overcome these limitations based on 2D and 3D reconstructions of the standard DICOM (Digital Imaging and Communications in Medicine) images. However, very little research has been conducted with regard to the clinical value of these two novel technologies in CHD. This study aims to investigate the usefulness and clinical value of MR and 3D printing in assisting diagnosis, medical education, pre-operative planning, and intraoperative guidance of CHD surgeries through evaluations from a group of cardiac specialists and physicians. Two cardiac computed tomography angiography scans that demonstrate CHD of different complexities (atrial septal defect and double outlet right ventricle) were selected and converted into 3D-printed heart models (3DPHM) and MR models. Thirty-four cardiac specialists and physicians were recruited. The results showed that the MR models were ranked as the best modality amongst the three, and were significantly better than DICOM images in demonstrating complex CHD lesions (mean difference (MD) = 0.76, p = 0.01), in enhancing depth perception (MD = 1.09, p = 0.00), in portraying spatial relationship between cardiac structures (MD = 1.15, p = 0.00), as a learning tool of the pathology (MD = 0.91, p = 0.00), and in facilitating pre-operative planning (MD = 0.87, p = 0.02). The 3DPHM were ranked as the best modality and significantly better than DICOM images in facilitating communication with patients (MD = 0.99, p = 0.00). In conclusion, both MR models and 3DPHM have their own strengths in different aspects, and they are superior to standard DICOM images in the visualization and management of CHD.

The Expanding Uses of Medical Extended Reality in the Cardiac Catheterization Laboratory: Pre-procedural Planning, Intraprocedural Guidance, and Intraprocedural Navigation

The use of innovative imaging practices in the field of interventional cardiology and electrophysiology has led to significant progress in both diagnostic and therapeutic capabilities. 3D reconstructions of 2D images allows a proceduralist to develop a superior understanding of patient anatomy. Medical extended reality (MXR) technologies employ 3D interactive images for the user to improve depth perception and spatial awareness. Although MXR procedural navigation is a relatively new concept, the potential for use within interventional cardiology and EP is significant with the eventual goal of improving patient outcomes and reducing patient harm. This review article will discuss the current landscape of MXR use in the catheterization lab including pre-procedural planning, intraprocedural planning and intraprocedural guidance in diagnostic cardiac catheterization, valvar and coronary interventions, electrophysiology studies, and device implants.

Location matters: Offset in tissue-engineered vascular graft implantation location affects wall shear stress in porcine models

Objective

Although surgical simulation using computational fluid dynamics has advanced, little is known about the accuracy of cardiac surgical procedures after patient-specific design. We evaluated the effects of discrepancies in location for patient-specific simulation and actual implantation on hemodynamic performance of patient-specific tissue-engineered vascular grafts (TEVGs) in porcine models.

Methods

Magnetic resonance angiography and 4-dimensional (4D) flow data were acquired in porcine models (n = 11) to create individualized TEVGs. Graft shapes were optimized and manufactured by electrospinning bioresorbable material onto a metal mandrel. TEVGs were implanted 1 or 3 months postimaging, and postoperative magnetic resonance angiography and 4D flow data were obtained and segmented. Displacement between intended and observed TEVG position was determined through center of mass analysis. Hemodynamic data were obtained from 4D flow analysis. Displacement and hemodynamic data were compared using linear regression.

Results

Patient-specific TEVGs were displaced between 1 and 8 mm during implantation compared with their surgically simulated, intended locations. Greater offset between intended and observed position correlated with greater wall shear stress (WSS) in postoperative vasculature (P < .01). Grafts that were implanted closer to their intended locations showed decreased WSS.

Conclusions

Patient-specific TEVGs are designed for precise locations to help optimize hemodynamic performance. However, if TEVGs were implanted far from their intended location, worse WSS was observed. This underscores the importance of not only patient-specific design but also precision-guided implantation to optimize hemodynamics in cardiac surgery and increase reproducibility of surgical simulation.

The use of mixed reality in dentistry

Augmented reality (AR) was first described in the literature in the 1990s. It has been shown as a futuristic concept in television, film and media, and now in the twenty-first century has become a reality. AR is defined as an interactive experience of a real-world environment where the object that resides in the real world is enhanced by computer-generated perceptual information.Microsoft HoloLens is a mixed reality device which has the capability to provide a real-time, three-dimensional platform using multiple sensors and holographic processing to display information and even simulate a virtual world. With rapidly evolving technology and virtual learning on the increase, the HoloLens technology can be used as a vital tool for dental education and surgical planning. However, within dentistry at present, there is limited research regarding its benefits and potential.The authors would like to demonstrate the use of HoloLens in three common oral surgery procedures and how it can be used to distinguish anatomy and benefit surgical planning, aid in patient communication and play a role in dental education.

Case Report: Application of Mixed Reality Combined With A Surgical Template for Precise Periapical Surgery

Objective

The etiology of apical diseases is diverse, and most are due to incomplete root canal therapy. The common clinical manifestations include gingival abscess, fistula and bone destruction. The currently existing limitation of procedures is that surgeons cannot visually evaluate the surgical areas. We sought to combine mixed reality (MR) technology with a 3-dimensional (3D) printed surgical template to achieve visualization in apical surgery. Notably, no reports have described this application.

Methods

We created visual 3D (V3D) files and transferred them into the HoloLens system. We explained the surgical therapy plan to the patient using a mixed reality head-mounted display (MR-HMD). Then, the 3D information was preliminarily matched with the operative area, and the optimal surgical approach was determined by combining this information with 3D surgical guide plate technology.

Results

We successfully developed a suitable surgical workflow and confirmed the optimal surgical approach from the buccal side. We completely exposed the apical lesion and removed the inflammatory granulation tissue.

Conclusion

We are the first group to use the MR technique in apical surgery. We integrated the MR technique with a 3D surgical template to successfully accomplish the surgery. Desirable outcomes using minimally invasive therapy could be achieved with the MR technique.

Imaging-Based, Patient-Specific Three-Dimensional Printing to Plan, Train, and Guide Cardiovascular Interventions: A Systematic Review and Meta-Analysis

BACKGROUND

To tailor cardiovascular interventions, the use of three-dimensional (3D), patient-specific phantoms (3DPSP) encompasses patient education, training, simulation, procedure planning, and outcome-prediction.

AIM

This systematic review and meta-analysis aims to investigate the current and future perspective of 3D printing for cardiovascular interventions.

METHODS

We systematically screened articles on Medline and EMBASE reporting the prospective use of 3DPSP in cardiovascular interventions by using combined search terms. Studies that compared intervention time depending on 3DPSP utilisation were included into a meta-analysis.

RESULTS

We identified 107 studies that prospectively investigated a total of 814 3DPSP in cardiovascular interventions. Most common settings were congenital heart disease (CHD) (38 articles, 6 comparative studies), left atrial appendage (LAA) occlusion (11 articles, 5 comparative, 1 randomised controlled trial [RCT]), and aortic disease (10 articles). All authors described 3DPSP as helpful in assessing complex anatomic conditions, whereas poor tissue mimicry and the non-consideration of physiological properties were cited as limitations. Compared to controls, meta-analysis of six studies showed a significant reduction of intervention time in LAA occlusion (n=3 studies), and surgery due to CHD (n=3) if 3DPSPs were used (Cohen's d=0.54; 95% confidence interval, 0.13 to 0.95; p=0.001), however heterogeneity across studies should be taken into account.

CONCLUSIONS

3DPSP are helpful to plan, train, and guide interventions in patients with complex cardiovascular anatomy. Benefits for patients include reduced intervention time with the potential for lower radiation exposure and shorter mechanical ventilation times. More evidence and RCTs including clinical endpoints are needed to warrant adoption of 3DPSP into routine clinical practice.

CT in Transcatheter-delivered Treatment of Valvular Heart Disease

Minimally invasive strategies to treat valvular heart disease have emerged over the past 2 decades. The use of transcatheter aortic valve replacement in the treatment of severe aortic stenosis, for example, has recently expanded from high- to low-risk patients and became an alternative treatment for those with prohibitive surgical risk. With the increase in transcatheter strategies, multimodality imaging, including echocardiography, CT, fluoroscopy, and cardiac MRI, are used. Strategies for preprocedural imaging strategies vary depending on the targeted valve. Herein, an overview of preprocedural imaging strategies and their postprocessing approaches is provided, with a focus on CT. Transcatheter aortic valve replacement is reviewed, as well as less established minimally invasive treatments of the mitral and tricuspid valves. In addition, device-specific details and the goals of CT imaging are discussed. Future imaging developments, such as peri-procedural fusion imaging, machine learning for image processing, and mixed reality applications, are presented.

A Novel 3D Visualized Operative Procedure in the Single-Stage Complete Repair With Unifocalization of Pulmonary Atresia With Ventricular Septal Defect and Major Aortopulmonary Collateral Arteries

Objectives

Pulmonary atresia with ventricular septal defect and major aortopulmonary collateral arteries (PA/VSD/MAPCAs) is a relatively rare, complex, and heterogeneous congenital heart disease. As one of the effective treatments, the midline unifocalization strategy still remains complicated and challenging due to the diverse forms of MAPCAs and pulmonary arteries. The purpose of this study is to summarize our experience of a novel three-dimensional (3D) visualized operative procedure in the single-stage complete repair with unifocalization and to clarify the benefits it may bring to us.

Methods

We described our experience of the 3D visualized operative procedure such as 3D printing, virtual reality (VR), and mixed reality (MR) technology in patients with PA/VSD/MAPCAs who underwent a single-stage complete repair with unifocalization. The data from the patients who underwent this procedure (3D group) and those who underwent the conventional procedure (conventional group) were compared.

Results

The conventional and 3D groups included 11 patients from September 2011 to December 2017 and 9 from January 2018 to March 2021, respectively. The baseline characteristics such as age, body weight, preoperative saturation, the anatomy of the pulmonary arteries and MAPCAs, the Nakata index, and TNPAI had no statistical significance. All 9 patients in the 3D group were operated only through a median sternotomy, while 8 cases (72.7%) in the conventional group needed another posterolateral thoracotomy (p = 0.001). In the 3D group, the CPB time was shorter (93.2 ± 63.8 vs. 145.1 ± 68.4 min, p = 0.099), and the median pre-CPB time per MAPCAs was significantly shorter [25.7 (14.0, 46.3) vs. 65 (41.3, 75.0) min, p = 0.031]. There was no early death in the 3D group, while there were 3 in the conventional group (0 vs. 27.3%, p = 0.218).

Conclusion

The novel 3D visualized operative procedure may help improve the performance of the single-stage complete repair with the midline unifocalization of PA/VSD/MAPCAs and help shorten the dissecting time of the MAPCAs. It may promote the routine and successful application of this strategy in more centers.

Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

Applications of Mixed Reality Technology in Orthopedics Surgery: A Pilot Study

Objective: The aim of this study is to explore the potential of mixed reality (MR) technology in the visualization of orthopedic surgery.

Methods: The visualization system with MR technology is widely used in orthopedic surgery. The system is composed of a 3D imaging workstation, a cloud platform, and an MR space station. An intelligent segmentation algorithm is adopted on the 3D imaging workstation to create a 3D anatomical model with zooming and rotation effects. This model is then exploited for efficient 3D reconstruction of data for computerized tomography (CT) and magnetic resonance imaging (MRI). Additionally, the model can be uploaded to the cloud platform for physical parameter tuning, model positioning, rendering and high-dimensional display. Using Microsoft’s HoloLens glasses in combination with the MR system, we project and view 3D holograms in real time under different clinical scenarios. After each procedure, nine surgeons completed a Likert-scale questionnaire on communication and understanding, spatial awareness and effectiveness of MR technology use. In addition to that, the National Aeronautics and Space Administration Task Load Index (NASA-TLX) is also used to evaluate the workload of MR hologram support.

Results: 1) MR holograms can clearly show the 3D structures of bone fractures, which improves the understanding of different fracture types and the design of treatment plans; 2) Holograms with three-dimensional lifelike dynamic features provide an intuitive communication tool among doctors and also between doctors and patients; 3) During surgeries, a full lesion hologram can be obtained and blended in real time with a patient’s virtual 3D digital model in order to give surgeons superior visual guidance through novel high-dimensional “perspectives” of the surgical area; 4) Hologram-based magnetic navigation improves the accuracy and safety of the screw placement in orthopaedics surgeries; 5) The combination of mixed reality cloud platform and telemedicine system based on 5G provides a new technology platform for telesurgery collaboration. Results of qualitative study encourage the usage of MR technology for orthopaedics surgery. Analysis of the Likert-scale questionnaire shows that MR adds significant value to understanding and communication, spatial awareness, learning and effectiveness. Based on the NASA TLX-scale questionnaire results, mixed reality scored significantly lower under the “mental,” “temporal,” “performance,” and “frustration” categories compared to usual 2D.

Conclusion: The integration of MR technology in orthopaedic surgery reduces the dependence on surgeons’ experience and provides personalized 3D visualization models for accurate diagnosis and treatment of orthopaedic abnormalities. This integration is clearly one of the prominent future development directions in medical surgery.

Spatially resolved transcriptomics in immersive environments

Spatially resolved transcriptomics is an emerging class of high-throughput technologies that enable biologists to systematically investigate the expression of genes along with spatial information. Upon data acquisition, one major hurdle is the subsequent interpretation and visualization of the datasets acquired. To address this challenge, VR-Cardiomics is presented, which is a novel data visualization system with interactive functionalities designed to help biologists interpret spatially resolved transcriptomic datasets. By implementing the system in two separate immersive environments, fish tank virtual reality (FTVR) and head-mounted display virtual reality (HMD-VR), biologists can interact with the data in novel ways not previously possible, such as visually exploring the gene expression patterns of an organ, and comparing genes based on their 3D expression profiles. Further, a biologist-driven use-case is presented, in which immersive environments facilitate biologists to explore and compare the heart expression profiles of different genes.

Three-Dimensional Virtual and Printed Prototypes in Complex Congenital and Pediatric Cardiac Surgery-A Multidisciplinary Team-Learning Experience

Three-dimensional (3D) virtual modeling and printing advances individualized medicine and surgery. In congenital cardiac surgery, 3D virtual models and printed prototypes offer advantages of better understanding of complex anatomy, hands-on preoperative surgical planning and emulation, and improved communication within the multidisciplinary team and to patients. We report our single center team-learning experience about the realization and validation of possible clinical benefits of 3D-printed models in surgical planning of complex congenital cardiac surgery. CT-angiography raw data were segmented into 3D-virtual models of the heart-great vessels. Prototypes were 3D-printed as rigid "blood-volume" and flexible "hollow". The accuracy of the models was evaluated intraoperatively. Production steps were realized in the framework of a clinical/research partnership. We produced 3D prototypes of the heart-great vessels for 15 case scenarios (nine males, median age: 11 months) undergoing complex intracardiac repairs. Parity between 3D models and intraoperative structures was within 1 mm range. Models refined diagnostics in 13/15, provided new anatomic information in 9/15. As a team-learning experience, all complex staged redo-operations (13/15; Aristotle-score mean: 10.64 ± 1.95) were rehearsed on the 3D models preoperatively. 3D-printed prototypes significantly contributed to an improved/alternative operative plan on the surgical approach, modification of intracardiac repair in 13/15. No operative morbidity/mortality occurred. Our clinical/research partnership provided coverage for the extra time/labor and material/machinery not financed by insurance. 3D-printed models provided a team-learning experience and contributed to the safety of complex congenital cardiac surgeries. A clinical/research partnership may open avenues for bioprinting of patient-specific implants.

A Virtual Reality System for Improved Image-Based Planning of Complex Cardiac Procedures

The intricate nature of congenital heart disease requires understanding of the complex, patient-specific three-dimensional dynamic anatomy of the heart, from imaging data such as three-dimensional echocardiography for successful outcomes from surgical and interventional procedures. Conventional clinical systems use flat screens, and therefore, display remains two-dimensional, which undermines the full understanding of the three-dimensional dynamic data. Additionally, the control of three-dimensional visualisation with two-dimensional tools is often difficult, so used only by imaging specialists. In this paper, we describe a virtual reality system for immersive surgery planning using dynamic three-dimensional echocardiography, which enables fast prototyping for visualisation such as volume rendering, multiplanar reformatting, flow visualisation and advanced interaction such as three-dimensional cropping, windowing, measurement, haptic feedback, automatic image orientation and multiuser interactions. The available features were evaluated by imaging and nonimaging clinicians, showing that the virtual reality system can help improve the understanding and communication of three-dimensional echocardiography imaging and potentially benefit congenital heart disease treatment.

Patient-specific 3-dimensional lung modelling: next-generation imaging helps guide precise surgical resection

Recent strides have allowed the consolidation of routine imaging modalities with highly accurate reconstruction software to aid the operating surgeon generate patient-specific three-dimensional models. In this preliminary report, we describe our initial experience using a patient-specific reconstruction software to guide surgical resection for 2 patients with non-small-cell lung cancer. Digital imaging and communications in medicine data from patient chest CT scans was configured into IQQA BodyImaging Lung software to generate highly accurate maps of airways, vessels and segments as well as estimates of lung volumes. Models generated aided us in planning appropriate lung cancer resection procedures.

3D Printing for Cardiovascular Applications: From End-to-End Processes to Emerging Developments

3D printing as a means of fabrication has seen increasing applications in medicine in the last decade, becoming invaluable for cardiovascular applications. This rapidly developing technology has had a significant impact on cardiovascular research, its clinical translation and education. It has expanded our understanding of the cardiovascular system resulting in better devices, tools and consequently improved patient outcomes. This review discusses the latest developments and future directions of generating medical replicas ('phantoms') for use in the cardiovascular field, detailing the end-to-end process from medical imaging to capture structures of interest, to production and use of 3D printed models. We provide comparisons of available imaging modalities and overview of segmentation and post-processing techniques to process images for printing, detailed exploration of latest 3D printing methods and materials, and a comprehensive, up-to-date review of milestone applications and their impact within the cardiovascular domain across research, clinical use and education. We then provide an in-depth exploration of future technologies and innovations around these methods, capturing opportunities and emerging directions across increasingly realistic representations, bioprinting and tissue engineering, and complementary virtual and mixed reality solutions. The next generation of 3D printing techniques allow patient-specific models that are increasingly realistic, replicating properties, anatomy and function.

Potential Efficacy of Multimodal Mixed Reality in Epilepsy Surgery

BACKGROUND

Mixed reality (MR) technology, which can fuse things in real and virtual space in real time, has been used mainly for simulation in neurosurgical training.

OBJECTIVE

To develop MR technology into multimodal MR for intraoperative guidance during epilepsy surgery.

METHODS

A 33-yr-old male patient suffered from intractable general tonic seizures due to left temporal meningoencephalocele. Preoperative scalp electroencephalograms localized interictal epileptic activity on the left temporal lobe. Iomazenil single photon emission tomography revealed temporal lobe lateralization. Magnetic resonance imaging (MRI) demonstrated left basal temporal meningoencephalocele extending into the pterygopalatine fossa through a bone defect at the base of the greater sphenoid wing. A 3-dimensional model was created for MR based on multimodal data including computed tomography, MRI tractography, and digital subtraction angiography, which enabled 3-dimensional visualization of abnormal subcortical fiber connections between the meningoencephalocele and the epileptic focus.

RESULTS

By using intraoperative multimodal MR, we were able to safely remove the meningoencephalocele and perform epileptic focus resection. The patient was seizure-free postoperatively, and no adverse effects were noted.

CONCLUSION

Intraoperative multimodal MR was a feasible and effective technique, and it can be applied for a wide range of epilepsy surgeries.

Non-invasive Imaging in the Evaluation of Cardiac Shunts for Interventional Closure

Multimodality imaging provides important information to guide patient selection and pre-procedural decision making for shunt lesions in CHD. While echocardiography, CT, and CMR are well-established, 3D printing and now virtual reality imaging are beginning to show promise.

Clinical Value of Virtual Reality versus 3D Printing in Congenital Heart Disease

Both three-dimensional (3D) printing and virtual reality (VR) are reported as being superior to the current visualization techniques in conveying more comprehensive visualization of congenital heart disease (CHD). However, little is known in terms of their clinical value in diagnostic assessment, medical education, and preoperative planning of CHD. This cross-sectional study aims to address these by involving 35 medical practitioners to subjectively evaluate VR visualization of four selected CHD cases in comparison with the corresponding 3D printed heart models (3DPHM). Six questionnaires were excluded due to incomplete sections, hence a total of 29 records were included for the analysis. The results showed both VR and 3D printed heart models were comparable in terms of the degree of realism. VR was perceived as more useful in medical education and preoperative planning compared to 3D printed heart models, although there was no significant difference in the ratings (p = 0.54 and 0.35, respectively). Twenty-one participants (72%) indicated both the VR and 3DPHM provided additional benefits compared to the conventional medical imaging visualizations. This study concludes the similar clinical value of both VR and 3DPHM in CHD, although further research is needed to involve more cardiac specialists for their views on the usefulness of these tools.

Virtual reality three-dimensional echocardiographic imaging for planning surgical atrioventricular valve repair

OBJECTIVES

To investigate how virtual reality (VR) imaging impacts decision-making in atrioventricular valve surgery.

METHODS

This was a single-center retrospective study involving 15 children and adolescents, median age 6 years (range, 0.33-16) requiring surgical repair of the atrioventricular valves between the years 2016 and 2019. The patients' preoperative 3-dimesnional (3D) echocardiographic data were used to create 3D visualization in a VR application. Five pediatric cardiothoracic surgeons completed a questionnaire formulated to compare their surgical decisions regarding the cases after reviewing conventionally presented 2-dimesnional and 3D echocardiographic images and again after visualization of 3D echocardiograms using the VR platform. Finally, intraoperative findings were shared with surgeons to confirm assessment of the pathology.

RESULTS

In 67% of cases presented with VR, surgeons reported having "more" or "much more" confidence in their understanding of each patient's pathology and their surgical approach. In all but one case, surgeons were at least as confident after reviewing the VR compared with standard imaging. The case where surgeons reported to be least confident on VR had the worst technical quality of data used. After viewing patient cases on VR, surgeons reported that they would have made minor modifications to surgical approach in 53% and major modifications in 7% of cases.

CONCLUSIONS

The main impact of viewing imaging on VR is the improved clarity of the anatomical structures. Surgeons reported that this would have impacted the surgical approach in the majority of cases. Poor-quality 3D echocardiographic data were associated with a negative impact of VR visualization; thus. quality assessment of imaging is necessary before projecting in a VR format.

Teaching with Disruptive Technology: The Use of Augmented, Virtual, and Mixed Reality (HoloLens) for Disease Education

Modern technologies are often utilised in schools or universities with a variety of educational goals in mind. Of particular interest is the enhanced interactivity and engagement offered by mixed reality devices such as the HoloLens, as well as the ability to explore anatomical models of disease using augmented and virtual realities. As the students are required to learn an ever-increasing number of diseases within a university health science or medical degree, it is crucial to consider which technologies provide value to educators and students. This chapter explores the opportunities for using modern disruptive technologies to teach a curriculum surrounding disease. For relevant examples, a focus will be placed on asthma as a respiratory disease which is increasing in prevalence, and stroke as a neurological and cardiovascular disease. The complexities of creating effective educational curricula around these diseases will be explored, along with the benefits of using augmented reality and mixed reality as viable teaching technologies in a range of use cases.

A Projector-Based Augmented Reality Navigation System for Computer-Assisted Surgery

In the medical field, guidance to follow the surgical plan is crucial. Image overlay projection is a solution to link the surgical plan with the patient. It realizes augmented reality (AR) by projecting computer-generated image on the surface of the target through a projector, which can visualize additional information to the scene. By overlaying anatomical information or surgical plans on the surgery area, projection helps to enhance the surgeon's understanding of the anatomical structure, and intuitively visualizes the surgical target and key structures of the operation, and avoid the surgeon's sight diversion between monitor and patient. However, it still remains a challenge to project the surgical navigation information on the target precisely and efficiently. In this study, we propose a projector-based surgical navigation system. Through the gray code-based calibration method, the projector can be calibrated with a camera and then be integrated with an optical spatial locator, so that the navigation information of the operation can be accurately projected onto the target area. We validated the projection accuracy of the system through back projection, with average projection error of 3.37 pixels in x direction and 1.51 pixels in y direction, and model projection with an average position error of 1.03 ± 0.43 mm, and carried out puncture experiments using the system with correct rate of 99%, and qualitatively analyzed the system's performance through the questionnaire. The results demonstrate the efficacy of our proposed AR system.

A three-dimensional printing navigational template combined with mixed reality technique for localizing pulmonary nodules

OBJECTIVES

Localizing non-palpable pulmonary nodules is challenging for thoracic surgeons. Here, we investigated the accuracy of three-dimensional (3D) printing technology combined with mixed reality (MR) for localizing ground glass opacity-dominant pulmonary nodules.

METHODS

In this single-arm study, we prospectively enrolled patients with small pulmonary nodules (<2 cm) that required accurate localization. A 3D-printing physical navigational template was designed based on the reconstruction of computed tomography images, and a 3D model was generated through the MR glasses. We set the deviation distance as the primary end point for efficacy evaluation. Clinicopathological and surgical data were obtained for further analysis.

RESULTS

Sixteen patients with 17 non-palpable pulmonary nodules were enrolled in this study. Sixteen nodules were localized successfully (16/17; 94.1%) using this novel approach with a median deviation of 9 mm. The mean time required for localization was 25 ± 5.2 min. For the nodules in the upper/middle and lower lobes, the median deviation was 6 mm (range, 0-12.0) and 16 mm (range, 15.0-20.0), respectively. The deviation difference between the groups was significant (Z = -2.957, P = 0.003). The pathological evaluation of resection margins was negative.

CONCLUSIONS

The 3D printing navigational template combined with MR can be a feasible approach for localizing pulmonary nodules.

Applications of Head-Mounted Displays and Smart Glasses in Vascular Surgery

OBJECTIVES

Advances in virtual, augmented and mixed reality have led to the development of wearable technologies including head mounted displays (HMD) and smart glasses. While there is a growing interest on their potential applications in health, only a few studies have addressed so far their use in vascular surgery. The aim of this review was to summarize the fundamental notions associated with these technologies and to discuss potential applications and current limits for their use in vascular surgery.

METHODS

A comprehensive literature review was performed to introduce the fundamental concepts and provide an overview of applications of HMD and smart glasses in surgery.

RESULTS

HMD and smart glasses demonstrated a potential interest for the education of surgeons including anatomical teaching, surgical training, teaching and telementoring. Applications for pre-surgical planning have been developed in general and cardiac surgery and could be transposed for a use in vascular surgery. The use of wearable technologies in the operating room has also been investigated in both general and cardiovascular surgery and demonstrated its potential interest for image-guided surgery and data collection.

CONCLUSION

Studies performed so far represent a proof of concept of the interest of HMD and smart glasses in vascular surgery for education of surgeons and for surgical practice. Although these technologies exhibited encouraging results for applications in vascular surgery, technical improvements and further clinical research in large series are required before hoping using them in daily clinical practice.

Using Mixed Reality (MR) to Improve On-Site Design Experience in Community Planning

In recent years, designing in existing environments has been consistently emphasized in community planning. However, practicing such on-site design is not easy for designers, because the current technical conditions do not allow virtual design objects into real environments for 3D visualization and interaction. Thus, designers’ intuitive design perceptions, accurate design judgments, and convenient design decisions are hardly supported. This paper explores the possibilities of using mixed reality (MR) technology to improve designers’ on-site design experiences in community planning. For this, we introduced an MR design support system (MR-DSS) for the interactive on-site 3D visualization of virtual design objects. With the MR-DSS, we performed a design experiment with sixteen participants in a typical on-site design scene of community planning. The results showed that the MR technology could provide designers with intuitive design perceptions, accurate design judgments, and convenient design decisions, thus effectively improving their on-site design experiences.

Stereoscopic virtual reality does not improve knowledge acquisition of congenital heart disease

Advances in virtual reality have made it possible for clinicians and trainees to interact with 3D renderings of hearts with congenital heart disease in 3D stereoscopic vision. No study to date has assessed whether this technology improved instruction compared to standard 2D interfaces. The purpose of this study was to assess whether stereoscopic virtual reality improves congenital heart disease anatomy education. Subjects in a prospective, blinded, randomized trial completed a pre-test assessing factual and visuospatial knowledge of common atrioventricular canal and were randomized to an intervention or control group based on their score. The intervention group used a 3D virtual reality (VR) headset to visualize a lecture with 3D heart models while the control group used a desktop (DT) computer interface with the same models. Subjects took a post-test and provided subjective feedback. 51 subjects were enrolled, 24 in the VR group & 27 in the DT group. The median score difference for VR subjects was 12 (IQR 9-13.3), compared to 10 (IQR 7.5-12) in the DT group. No difference in score improvement was found (p = 0.11). VR subjects' impression of the ease of use of their interface was higher than DT subjects (median 8 vs 7, respectively, p = 0.01). VR subjects' impression of their understanding of the subject matter was higher than desktop subjects (median 7 vs 5, respectively, p = 0.01). There was no statistically significant difference in the knowledge acquisition observed between the stereoscopic virtual reality group and the monoscopic desktop-based group. Participants in virtual reality reported a better learning experience and self-assessment suggesting virtual reality may increase learner engagement in understanding congenital heart disease.

Mixed Reality Visualization of Radiation Dose for Health Professionals and Patients in Interventional Radiology

For interventional radiology, dose management has persisted as a crucially important issue to reduce radiation exposure to patients and medical staff. This study designed a real-time dose visualization system for interventional radiology designed with mixed reality technology and Monte Carlo simulation. An earlier report described a Monte-Carlo-based estimation system, which simulates a patient's skin dose and air dose distributions, adopted for our system. We also developed a system of acquiring fluoroscopic conditions to input them into the Monte Carlo system. Then we combined the Monte Carlo system with a wearable device for three-dimensional holographic visualization. The estimated doses were transferred sequentially to the device. The patient's dose distribution was then projected on the patient body. The visualization system also has a mechanism to detect one's position in a room to estimate the user's exposure dose to detect and display the exposure level. Qualitative tests were conducted to evaluate the workload and usability of our mixed reality system. An end-to-end system test was performed using a human phantom. The acquisition system accurately recognized conditions that were necessary for real-time dose estimation. The dose hologram represents the patient dose. The user dose was changed correctly, depending on conditions and positions. The perceived overall workload score (33.50) was lower than the scores reported in the literature for medical tasks (50.60) for computer activities (54.00). Mixed reality dose visualization is expected to improve exposure dose management for patients and health professionals by exhibiting the invisible radiation exposure in real space.

Cinematic Rendering in Mixed-Reality Holograms: A New 3D Preoperative Planning Tool in Pediatric Heart Surgery

Cinematic rendering (CR) is based on a new algorithm that creates a photo-realistic three-dimensional (3D) picture from cross-sectional images. Previous studies have shown its positive impact on preoperative planning. To date, CR presentation has only been possible on 2D screens which limited natural 3D perception. To depict CR-hearts spatially, we used mixed-reality technology and mapped corresponding hearts as holograms in 3D space. Our aim was to assess the benefits of CR-holograms in the preoperative planning of cardiac surgery. Including 3D prints allowed a direct comparison of two spatially resolved display methods. Twenty-six patients were recruited between February and September 2019. CT or MRI was used to visualize the patient's heart preoperatively. The surgeon was shown the anatomy in cross-sections on a 2D screen, followed by spatial representations as a 3D print and as a high-resolution hologram. The holographic representation was carried out using mixed-reality glasses (HoloLens®). To create the 3D prints, corresponding structures were segmented to create STL files which were printed out of resin. In 22 questions, divided in 5 categories (3D-imaging effect, representation of pathology, structure resolution, cost/benefit ratio, influence on surgery), the surgeons compared each spatial representation with the 2D method, using a five-level Likert scale. The surgical preparation time was assessed by comparing retrospectively matched patient pairs, using a paired t-test. CR-holograms surpassed 2D-monitor imaging in all categories. CR-holograms were superior to 3D prints in all categories (mean Likert scale 4.4 ± 1.0 vs. 3.7 ± 1.3, P < 0.05). Compared to 3D prints it especially improved the depth perception (4.7 ± 0.7 vs. 3.7 ± 1.2) and the representation of the pathology (4.4 ± 0.9 vs. 3.6 ± 1.2). 3D imaging reduced the intraoperative preparation time (n = 24, 59 ± 23 min vs. 73 ± 43 min, P < 0.05). In conclusion, the combination of an extremely photo-realistic presentation via cinematic rendering and the spatial presentation in 3D space via mixed-reality technology allows a previously unattained level of comprehension of anatomy and pathology in preoperative planning.

Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics

Purpose

This study presents a novel surgical navigation tool developed in mixed reality environment for orthopaedic surgery. Joint and skeletal deformities affect all age groups and greatly reduce the range of motion of the joints. These deformities are notoriously difficult to diagnose and to correct through surgery.

Method

We have developed a surgical tool which integrates surgical instrument tracking and augmented reality through a head mounted display. This allows the surgeon to visualise bones with the illusion of possessing “X-ray” vision. The studies presented below aim to assess the accuracy of the surgical navigation tool in tracking a location at the tip of the surgical instrument in holographic space.

Results

Results show that the average accuracy provided by the navigation tool is around 8 mm, and qualitative assessment by the orthopaedic surgeons provided positive feedback in terms of the capabilities for diagnostic use.

Conclusions

More improvements are necessary for the navigation tool to be accurate enough for surgical applications, however, this new tool has the potential to improve diagnostic accuracy and allow for safer and more precise surgeries, as well as provide for better learning conditions for orthopaedic surgeons in training.

Development and Pre-Clinical Analysis of Spatiotemporal-Aware Augmented Reality in Orthopedic Interventions

Suboptimal interaction with patient data and challenges in mastering 3D anatomy based on ill-posed 2D interventional images are essential concerns in image-guided therapies. Augmented reality (AR) has been introduced in the operating rooms in the last decade; however, in image-guided interventions, it has often only been considered as a visualization device improving traditional workflows. As a consequence, the technology is gaining minimum maturity that it requires to redefine new procedures, user interfaces, and interactions. The main contribution of this paper is to reveal how exemplary workflows are redefined by taking full advantage of head-mounted displays when entirely co-registered with the imaging system at all times. The awareness of the system from the geometric and physical characteristics of X-ray imaging allows the exploration of different human-machine interfaces. Our system achieved an error of 4.76 ± 2.91mm for placing K-wire in a fracture management procedure, and yielded errors of 1.57 ± 1.16° and 1.46 ± 1.00° in the abduction and anteversion angles, respectively, for total hip arthroplasty (THA). We compared the results with the outcomes from baseline standard operative and non-immersive AR procedures, which had yielded errors of [4.61mm, 4.76°, 4.77°] and [5.13mm, 1.78°, 1.43°], respectively, for wire placement, and abduction and anteversion during THA. We hope that our holistic approach towards improving the interface of surgery not only augments the surgeon's capabilities but also augments the surgical team's experience in carrying out an effective intervention with reduced complications and provide novel approaches of documenting procedures for training purposes.

Current and Future Applications of Virtual, Augmented, and Mixed Reality in Cardiothoracic Surgery

BACKGROUND

This review aims to examine the existing literature to address currently used virtual, augmented, and mixed reality modalities in the areas of preoperative surgical planning, intraoperative guidance, and postoperative management in the field of cardiothoracic surgery. In addition, this innovative technology provides future perspectives and potential benefits for cardiothoracic surgeons, trainees, and patients.

METHODS

A targeted, non-systematic literature assessment was performed within the Medline and Google Scholar databases to help identify current trends and to provide better understanding of the current state-of-the-art extended reality (XR) modalities in cardiothoracic surgery. Related articles published up to July 2020, are included in the review.

RESULTS

XR is a novel technique gaining increasing application in cardiothoracic surgery. It provides a three-dimensional (3D) and realistic view of structures and environments and offers the user the ability to interact with digital projections of surgical targets. Recent studies showed the validity and benefits of XR applications in cardiothoracic surgery. Examples include XR-guided pre-operative planning, intraoperative guidance and navigation, post-operative pain and rehabilitation management, surgical simulation, and patient education.

CONCLUSIONS

XR is gaining interest in the field of cardiothoracic surgery. In particular, there are promising roles for XR applications in televirtuality, surgical planning, surgical simulation, and perioperative management. However, future refinement and research is needed to further implement XR in the aforementioned settings within cardiothoracic surgery.