Clinical application of navigation surgery using augmented reality in the abdominal field

Beyond the visible: preliminary evaluation of the first wearable augmented reality assistance system for pancreatic surgery

PURPOSE

The retroperitoneal nature of the pancreas, marked by minimal intraoperative organ shifts and deformations, makes augmented reality (AR)-based systems highly promising for pancreatic surgery. This study presents preliminary data from a prospective study aiming to develop the first wearable AR assistance system, ARAS, for pancreatic surgery and evaluating its usability, accuracy, and effectiveness in enhancing the perioperative outcomes of patients.

METHODS

We developed ARAS as a two-phase system for a wearable AR device to aid surgeons in planning and operation. This system was used to visualize and register patient-specific 3D anatomical models during the surgery. The location and precision of the registered 3D anatomy were evaluated by assessing the arterial pulse and employing Doppler and duplex ultrasonography. The usability, accuracy, and effectiveness of ARAS were assessed using a five-point Likert scale questionnaire.

RESULTS

Perioperative outcomes of five patients underwent various pancreatic resections with ARAS are presented. Surgeons rated ARAS as excellent for preoperative planning. All structures were accurately identified without any noteworthy errors. Only tumor identification decreased after the preparation phase, especially in patients who underwent pancreaticoduodenectomy because of the extensive mobilization of peripancreatic structures. No perioperative complications related to ARAS were observed.

CONCLUSIONS

ARAS shows promise in enhancing surgical precision during pancreatic procedures. Its efficacy in preoperative planning and intraoperative vascular identification positions it as a valuable tool for pancreatic surgery and a potential educational resource for future surgical residents.

Enhancing visualization and guidance in general surgery: a comprehensive and narrative review of the current cutting-edge technologies and future perspectives

BACKGROUND

In the last decade, there has been a great effort in developing new technologies to enhance surgical visualization and guidance. This comprehensive and narrative review aimed to provide a wide and extensive overview of the current state of the art on this topic and their near-future perspectives linked to the development of artificial intelligence (AI), by focusing on the most recent and relevant literature.

METHODS

A comprehensive and narrative review of the literature was performed by searching specific terms on PubMed/MEDLINE, Scopus, and Embase databases, assessing the current state of the art on this topic.

RESULTS

Fluorescence-guided surgery, contrast-enhanced ultrasound (CEUS), ultra-high frequency ultrasound (UHFUS), photoacoustic imaging (PAI), and augmented reality (AR) are boosting the field of image-guided techniques as the rapid development of AI in surgery is promising a more automated decision-making and surgical movements in the operating room.

CONCLUSION

Fluorescence-guided surgery, CEUS, UHFUS, PAI, and AR are becoming crucial to give surgeons a new level of information during the intervention, with the right timing and sequence, and represent the future of surgery. As many more controlled studies are needed to validate the employment of these technologies, the next generation of surgeons must become more familiar with the basics of AI to better incorporate new tools into the daily surgical practice of the future.

Is Augmented Reality Technology Effective in Locating the Apex of Teeth Undergoing Apicoectomy Procedures?

This study seeks to assess the accuracy of apical location using an augmented reality (AR) device with a free-hand method. Sixty (60) osteotomy site preparations were randomly assigned to one of two study groups: A. AR device (AR) (n = 30), and B. conventional free-hand method (FHM) (n = 30). Preoperative CBCT scans and intraoral scans were taken and uploaded to specialized implant-planning software to virtually plan preparations for the apical location osteotomy sites. The planning software was then used to automatically segment the teeth in each experimental model for their complete visualization using the AR device. A CBCT scan was carried out postoperatively after conducting the apical location procedures. The subsequent datasets were imported into therapeutic software to analyze the coronal, apical, and angular deviations. The Mann-Whitney non-parametric test was used. There were no statistically significant differences identified at the coronal (p = 0.1335), apical (p = 0.2401), and angular deviations (p = 0.4849) between the AR and FHM study groups. The augmented reality technique did not show a statistically significant accuracy of osteotomies for apical location when compared with the conventional free-hand method.

Preclinical Application of Augmented Reality in Pediatric Craniofacial Surgery: An Accuracy Study

Background: Augmented reality (AR) allows the overlapping and integration of virtual information with the real environment. The camera of the AR device reads the object and integrates the virtual data. It has been widely applied to medical and surgical sciences in recent years and has the potential to enhance intraoperative navigation. Materials and methods: In this study, the authors aim to assess the accuracy of AR guidance when using the commercial HoloLens 2 head-mounted display (HMD) in pediatric craniofacial surgery. The Authors selected fronto-orbital remodeling (FOR) as the procedure to test (specifically, frontal osteotomy and nasal osteotomy were considered). Six people (three surgeons and three engineers) were recruited to perform the osteotomies on a 3D printed stereolithographic model under the guidance of AR. By means of calibrated CAD/CAM cutting guides with different grooves, the authors measured the accuracy of the osteotomies that were performed. We tested accuracy levels of ±1.5 mm, ±1 mm, and ±0.5 mm. Results: With the HoloLens 2, the majority of the individuals involved were able to successfully trace the trajectories of the frontal and nasal osteotomies with an accuracy level of ±1.5 mm. Additionally, 80% were able to achieve an accuracy level of ±1 mm when performing a nasal osteotomy, and 52% were able to achieve an accuracy level of ±1 mm when performing a frontal osteotomy, while 61% were able to achieve an accuracy level of ±0.5 mm when performing a nasal osteotomy, and 33% were able to achieve an accuracy level of ±0.5 mm when performing a frontal osteotomy. Conclusions: despite this being an in vitro study, the authors reported encouraging results for the prospective use of AR on actual patients.

Visualization, registration and tracking techniques for augmented reality guided surgery: a review

Augmented reality (AR) surgical navigation has developed rapidly in recent years. This paper reviews and analyzes the visualization, registration, and tracking techniques used in AR surgical navigation systems, as well as the application of these AR systems in different surgical fields. The types of AR visualization are divided into two categories ofin situvisualization and nonin situvisualization. The rendering contents of AR visualization are various. The registration methods include manual registration, point-based registration, surface registration, marker-based registration, and calibration-based registration. The tracking methods consist of self-localization, tracking with integrated cameras, external tracking, and hybrid tracking. Moreover, we describe the applications of AR in surgical fields. However, most AR applications were evaluated through model experiments and animal experiments, and there are relatively few clinical experiments, indicating that the current AR navigation methods are still in the early stage of development. Finally, we summarize the contributions and challenges of AR in the surgical fields, as well as the future development trend. Despite the fact that AR-guided surgery has not yet reached clinical maturity, we believe that if the current development trend continues, it will soon reveal its clinical utility.

A novel motionless calibration method for augmented reality surgery navigation system based on optical tracker

Augmented reality (AR) surgery navigation systems display the pre-operation planned virtual model at the accurate position in the real surgical scene to assist the operation. Accurate calibration of the mapping relationship between the virtual coordinate and the real world is the key to the virtual-real fusion effect. Former calibration methods require the doctor user to conduct complex manual procedures before usage. This paper introduces a novel motionless virtual-real calibration method. The method only requires to take a mixed reality image containing both virtual and real marker balls using the built-in forward camera of the AR glasses. The mapping relationship between the virtual and real spaces is calculated by using the camera coordinate system as a transformation medium. The composition and working process of the AR navigation system is introduced, and then the mathematical principle of the calibration is designed. The feasibility of the proposed calibration scheme is verified with a verification experiment, and the average registration accuracy of the scheme is around 5.80mm, which is of same level of formerly reported methods. The proposed method is convenient and rapid to implement, and the calibration accuracy is not dependent on the user experience. Further, it can potentially realize the real-time update of the registration transformation matrix, which can improve the AR fusion accuracy when the AR glasses moves. This motionless calibration method has great potential to be applied in future clinical navigation research.

Augmented Reality Surgical Navigation System for External Ventricular Drain

Augmented reality surgery systems are playing an increasing role in the operating room, but applying such systems to neurosurgery presents particular challenges. In addition to using augmented reality technology to display the position of the surgical target position in 3D in real time, the application must also display the scalpel entry point and scalpel orientation, with accurate superposition on the patient. To improve the intuitiveness, efficiency, and accuracy of extra-ventricular drain surgery, this paper proposes an augmented reality surgical navigation system which accurately superimposes the surgical target position, scalpel entry point, and scalpel direction on a patient’s head and displays this data on a tablet. The accuracy of the optical measurement system (NDI Polaris Vicra) was first independently tested, and then complemented by the design of functions to help the surgeon quickly identify the surgical target position and determine the preferred entry point. A tablet PC was used to display the superimposed images of the surgical target, entry point, and scalpel on top of the patient, allowing for correct scalpel orientation. Digital imaging and communications in medicine (DICOM) results for the patient’s computed tomography were used to create a phantom and its associated AR model. This model was then imported into the application, which was then executed on the tablet. In the preoperative phase, the technician first spent 5–7 min to superimpose the virtual image of the head and the scalpel. The surgeon then took 2 min to identify the intended target position and entry point position on the tablet, which then dynamically displayed the superimposed image of the head, target position, entry point position, and scalpel (including the scalpel tip and scalpel orientation). Multiple experiments were successfully conducted on the phantom, along with six practical trials of clinical neurosurgical EVD. In the 2D-plane-superposition model, the optical measurement system (NDI Polaris Vicra) provided highly accurate visualization (2.01 ± 1.12 mm). In hospital-based clinical trials, the average technician preparation time was 6 min, while the surgeon required an average of 3.5 min to set the target and entry-point positions and accurately overlay the orientation with an NDI surgical stick. In the preparation phase, the average time required for the DICOM-formatted image processing and program import was 120 ± 30 min. The accuracy of the designed augmented reality optical surgical navigation system met clinical requirements, and can provide a visual and intuitive guide for neurosurgeons. The surgeon can use the tablet application to obtain real-time DICOM-formatted images of the patient, change the position of the surgical entry point, and instantly obtain an updated surgical path and surgical angle. The proposed design can be used as the basis for various augmented reality brain surgery navigation systems in the future.

A particle filter approach to dynamic kidney pose estimation in robotic surgical exposure

PURPOSE

Traditional soft tissue registration methods require direct intraoperative visualization of a significant portion of the target anatomy in order to produce acceptable surface alignment. Image guidance is therefore generally not available during the robotic exposure of structures like the kidneys which are not immediately visualized upon entry into the abdomen. This paper proposes guiding surgical exposure with an iterative state estimator that assimilates small visual cues into an a priori anatomical model as exposure progresses, thereby evolving pose estimates for the occluded structures of interest.

METHODS

Intraoperative surface observations of a right kidney are simulated using endoscope tracking and preoperative tomography from a representative robotic partial nephrectomy case. Clinically relevant random perturbations of the true kidney pose are corrected using this sequence of observations in a particle filter framework to estimate an optimal similarity transform for fitting a patient-specific kidney model at each step. The temporal response of registration error is compared against that of serial rigid coherent point drift (CPD) in both static and simulated dynamic surgical fields, and for varying levels of observation persistence.

RESULTS

In the static case, both particle filtering and persistent CPD achieved sub-5 mm accuracy, with CPD processing observations 75% faster. Particle filtering outperformed CPD in the dynamic case under equivalent computation times due to the former requiring only minimal persistence.

CONCLUSION

This proof-of-concept simulation study suggests that Bayesian state estimation may provide a viable pathway to image guidance for surgical exposure in the abdomen, especially in the presence of dynamic intraoperative tissue displacement and deformation.

Utility of optical see-through head mounted displays in augmented reality-assisted surgery: A systematic review

This article presents a systematic review of optical see-through head mounted display (OST-HMD) usage in augmented reality (AR) surgery applications from 2013 to 2020. Articles were categorised by: OST-HMD device, surgical speciality, surgical application context, visualisation content, experimental design and evaluation, accuracy and human factors of human-computer interaction. 91 articles fulfilled all inclusion criteria. Some clear trends emerge. The Microsoft HoloLens increasingly dominates the field, with orthopaedic surgery being the most popular application (28.6%). By far the most common surgical context is surgical guidance (n=58) and segmented preoperative models dominate visualisation (n=40). Experiments mainly involve phantoms (n=43) or system setup (n=21), with patient case studies ranking third (n=19), reflecting the comparative infancy of the field. Experiments cover issues from registration to perception with very different accuracy results. Human factors emerge as significant to OST-HMD utility. Some factors are addressed by the systems proposed, such as attention shift away from the surgical site and mental mapping of 2D images to 3D patient anatomy. Other persistent human factors remain or are caused by OST-HMD solutions, including ease of use, comfort and spatial perception issues. The significant upward trend in published articles is clear, but such devices are not yet established in the operating room and clinical studies showing benefit are lacking. A focused effort addressing technical registration and perceptual factors in the lab coupled with design that incorporates human factors considerations to solve clear clinical problems should ensure that the significant current research efforts will succeed.

Augmented Reality and Image-Guided Robotic Liver Surgery

Artificial intelligence makes surgical resection easier and safer, and, at the same time, can improve oncological results. The robotic system fits perfectly with these more or less diffused technologies, and it seems that this benefit is mutual. In liver surgery, robotic systems help surgeons to localize tumors and improve surgical results with well-defined preoperative planning or increased intraoperative detection. Furthermore, they can balance the absence of tactile feedback and help recognize intrahepatic biliary or vascular structures during parenchymal transection. Some of these systems are well known and are already widely diffused in open and laparoscopic hepatectomies, such as indocyanine green fluorescence or ultrasound-guided resections, whereas other tools, such as Augmented Reality, are far from being standardized because of the high complexity and elevated costs. In this paper, we review all the experiences in the literature on the use of artificial intelligence systems in robotic liver resections, describing all their practical applications and their weaknesses.

Current trends in three-dimensional visualization and real-time navigation as well as robot-assisted technologies in hepatobiliary surgery

With the continuous development of digital medicine, minimally invasive precision and safety have become the primary development trends in hepatobiliary surgery. Due to the specificity and complexity of hepatobiliary surgery, traditional preoperative imaging techniques such as computed tomography and magnetic resonance imaging cannot meet the need for identification of fine anatomical regions. Imaging-based three-dimensional (3D) reconstruction, virtual simulation of surgery and 3D printing optimize the surgical plan through preoperative assessment, improving the controllability and safety of intraoperative operations, and in difficult-to-reach areas of the posterior and superior liver, assistive robots reproduce the surgeon’s natural movements with stable cameras, reducing natural vibrations. Electromagnetic navigation in abdominal surgery solves the problem of conventional surgery still relying on direct visual observation or preoperative image assessment. We summarize and compare these recent trends in digital medical solutions for the future development and refinement of digital medicine in hepatobiliary surgery.

Development of recognised position-guided navigation system

BACKGROUND

Previously, we developed an image-guided navigation system (IG-NS) incorporating augmented reality technology. Nevertheless, the system could still only aid the operator by presenting imagery and was short of achieving the goal of developing a real navigation system. Therefore, we developed a recognised position-guided navigation system (RP-NS) and herein reported the functionality and usefulness of this system in a phantom model for clinical applications.

METHODS

We developed RP-NS which was reconstructed by adding the positional recognition and instruction functions with the cautions by displaying the images on the monitor with a voice to the IG-NS. We evaluated accuracy of positional recognition and instruction functions using phantom model. By utilising the chronological recording of the tip position of the surgical apparatus, the surgical precision of the operators was assessed. Finally, the feasibility of improvements in surgical precision using this system was evaluated.

RESULTS

The RP-NS indicated an accuracy of the position recognition functions with an error of 2.7 mm. The surgeons could perform partial hepatectomies within mean value of 7.5% error as compared with calculated volume according to the instruction. Improvements in surgical precision using this system were obtained on the surgeons with different levels.

CONCLUSIONS

The RP-NS was highly effective as a navigation system owing to precise positional recognition and adequate instruction functions. Therefore, these results indicate that the use of this system may complement differences in proficiency, and numerically evaluate surgical skills and analyse tendencies of surgeons.

Performance of image guided navigation in laparoscopic liver surgery - A systematic review

BACKGROUND

Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations.

METHODS

Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format.

RESULTS

Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes.

CONCLUSIONS

Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.

Augmented reality technology in image-guided therapy: State-of-the-art review

Image-guided therapies have been on the rise in recent years as they can achieve higher accuracy and are less invasive than traditional methods. By combining augmented reality technology with image-guided therapy, more organs, and tissues can be observed by surgeons to improve surgical accuracy. In this review, 233 publications (dated from 2015 to 2020) on the design and application of augmented reality-based systems for image-guided therapy, including both research prototypes and commercial products, were considered for review. Based on their functions and applications. Sixteen studies were selected. The engineering specifications and applications were analyzed and summarized for each study. Finally, future directions and existing challenges in the field were summarized and discussed.

Augmented reality navigation method for recontouring surgery of craniofacial fibrous dysplasia

The objective of this study is to introduce the application of augmented reality (AR) navigation system developed by the authors in recontouring surgery of craniofacial fibrous dysplasia. Five consecutive patients with craniofacial fibrous dysplasia were enrolled. Through three-dimensional (3D) simulation, a virtual plan was designed to reconstruct the normal anatomical contour of the deformed region. Surgical recontouring was achieved with the assistance of the AR navigation system. The accuracy of the surgical procedure was assessed by superimposing the post-operative 3D craniomaxillofacial model onto the virtual plan. The pre-operative preparation time and operation time were also counted. In all patients, AR navigation was performed successfully, with a mean ± SD of the errors of 1.442 ± 0.234 mm. The operative time of the patients ranged from 60 to 80 min. The pre-operative preparation time was 20 min for each patient. All the patients showed uneventful healing without any complications, in addition to satisfaction with the post-operative aesthetics. Using our AR navigation system in recontouring surgery can provide surgeons with a comprehensive and intuitive view of the recontouring border, as well as the depth, in real time. This method could improve the efficiency and safety of craniofacial fibrous dysplasia recontouring procedures.

Use of augmented reality navigation to optimise the surgical management of craniofacial fibrous dysplasia

The aim of this study was to apply an augmented reality (AR) navigation technique based on a head- mounted display in the treatment of craniofacial fibrous dysplasia and to explore the feasibility and the value of AR in craniofacial surgery. With preoperative planning and three-dimensional simulation, the normal anatomical contours of the deformed area were recreated by superimposing the unaffected side on to the affected side. We completed the recontouring procedures in real time with the aid of an AR navigation system. The surgical outcome was assessed by superimposing the postoperative computed tomographic images on to the preoperative virtual plan. The preparation and operation times were recorded. With intraoperative AR guidance, facial bone recontouring was performed uneventfully in all cases. The mean (SD) discrepancy between the actual surgical reduction and preoperative planning was 1.036 (0.081) mm (range: 0.913 (0.496) to 1.165 (0.498) mm). The operation time ranged from 50 to 80 minutes, with an average of 66.4 minutes. The preoperative preparation time ranged from 26 to 36 minutes, with a mean of 29.6 minutes. AR navigation-assisted facial bone recontouring is a valuable treatment modality in managing craniomaxillofacial fibrous dysplasia and shows benefits in improving the efficiency and safety of this complicated procedure.

Augmented Reality-Assisted Pancreaticoduodenectomy with Superior Mesenteric Vein Resection and Reconstruction

Introduction

Pancreaticoduodenectomy (PD) with superior mesenteric vein (SMV) reconstruction are often required to achieve complete (R0) resection for pancreatic head cancer (PHC) with tumor invasion of the SMV. Augmented reality (AR) technology can be used to assist in determining the extent of SMV involvement by superimposing virtual 3-dimensional (3D) images of the pancreas and regional vasculature on the surgical field.

Materials and Methods

Three patients with PHC and tumor invasion of the SMV underwent AR-assisted PD with SMV resection and reconstruction following preoperative computed tomography scanning. Preoperative imaging data were used to reconstruct 3D images of anatomical structures, including the tumor, portal vein (PV), SMV, and splenic vein (SV). Using AR software installed on a smart phone, the reconstructed 3D images were superimposed on the surgical field as viewed in a smart phone display to provide intermittent navigational assistance to the surgeon in identifying the boundaries of PHC tumor invasion for resection of the vessels involved.

Result

All patients successfully completed the operation. Intraoperative AR applications displayed virtual images of the pancreas, SMV, bile duct, common hepatic artery (CHA), and superior mesenteric artery (SMA). Two patients required end-to-end anastomosis for reconstruction of the SMV. One patient required allogenic vascular bypass to reconstruct the SMV-PV juncture with concomitant reconstruction of the SV-SMV confluence by end-to-side anastomosis of the SV and bypass vessel. Postoperative pathology confirmed R0 resections for all patients.

Conclusion

AR navigation technology based on preoperative CT image data can assist surgeons performing PD with SMV resection and reconstruction.

Using Virtual Reality in Biomedical Engineering Education

This study explored virtual reality (VR) as an educational tool to offer immersive and experiential learning environments to biomedical engineering (BME) students. VR and traditional two-dimensional (2D) videos were created and used to teach required communication skills to BME students' while working with clinical partners in healthcare settings. The videos of interdisciplinary teams (engineering and nursing students) tackling medical device-related problems, similar to those commonly observed in healthcare settings, were shown to BME students. Student surveys indicated that, through VR videos, they felt more immersed in real-world clinical scenarios while learning about the clinical problems, each team-member's areas of expertise, their roles and responsibilities, and how an interdisciplinary team operated collectively to solve a problem in the presented settings. Students with a prior in-person immersion experience, in the presented settings, reported VR videos to serve as a possible alternative to in-person immersion and a useful tool for their preparedness for real-world clinical immersion. We concluded that VR holds promise as an educational tool to offer simulated clinical scenarios that are effective in training BME students for interprofessional collaborations.

Pancreatic Cancer Imaging: A New Look at an Old Problem

Computed tomography is the most commonly used imaging modality to detect and stage pancreatic cancer. Previous advances in pancreatic cancer imaging have focused on optimizing image acquisition parameters and reporting standards. However, current state-of-the-art imaging approaches still misdiagnose some potentially curable pancreatic cancers and do not provide prognostic information or inform optimal management strategies beyond stage. Several recent developments in pancreatic cancer imaging, including artificial intelligence and advanced visualization techniques, are rapidly changing the field. The purpose of this article is to review how these recent advances have the potential to revolutionize pancreatic cancer imaging.

Augmented Reality: A Systematic Review of Its Benefits and Challenges in E-learning Contexts

Augmented reality (AR) has received increasing attention in the research literature as a fundamental pedagogical tool that can enhance learning at most educational levels. In academic contexts, this technology permits the superimposing of three-dimensional images onto the real environment. Although AR has been found to enhance learning in the academic environment, no systematic review of it has been conducted to identify, evaluate, and summarize empirical findings on its advantages and challenges in e-learning contexts. Hence, a systematic review of the research literature was conducted on the use of AR in e-learning contexts, with a focus on the key benefits and challenges related to its adoption and implementation. Electronic searches on databases, including Springer, Science Direct, EBSCO, and Google Scholar, were performed to retrieve relevant journal articles; 28 studies were included after they were screened using the inclusion and exclusion criteria. The key benefits of using AR in e-learning included support of kinesthetic (tactile) learning, collaborative learning, distance/remote learning, learner-centered learning, and creative learning. Studies also reported that AR enhanced students’ engagement, motivation, attention/focus, interactivity, verbal participation, concentration, knowledge retention, and spatial abilities, as well as information accessibility. The findings suggest that challenges associated with AR in e-learning include information and cognitive overload, lack of experience in using the technology, resistance from teachers, complex technology, costly technology, and technical issues, such as connectivity problems.

VR4Health: Personalized teaching and learning anatomy using VR

Virtual Reality (VR) is being integrated into many different areas of our lives, from industrial engineering to video-games, and also including teaching and education. We have several examples where VR has been used to engage students and facilitate their 3D spatial understanding, but can VR help also teachers? What are the benefits teachers can obtain on using VR applications? In this paper we present an application (VR4Health) designed to allow students to directly inspect 3D models of several human organs by using Virtual Reality systems. The application is designed to be used in an HMD device autonomously as a self-learning tool and also reports information to teachers in order that he/she becomes aware of what the students do and can redirect his/her work to the concrete necessities of the student. We evaluate both the students’ and the teachers’ perception by doing an experiment and asking them to fill-in a questionnaire at the end of the experiment.

Comparison of manual and semi-automatic registration in augmented reality image-guided liver surgery: a clinical feasibility study

BACKGROUND

The laparoscopic approach to liver resection may reduce morbidity and hospital stay. However, uptake has been slow due to concerns about patient safety and oncological radicality. Image guidance systems may improve patient safety by enabling 3D visualisation of critical intra- and extrahepatic structures. Current systems suffer from non-intuitive visualisation and a complicated setup process. A novel image guidance system (SmartLiver), offering augmented reality visualisation and semi-automatic registration has been developed to address these issues. A clinical feasibility study evaluated the performance and usability of SmartLiver with either manual or semi-automatic registration.

METHODS

Intraoperative image guidance data were recorded and analysed in patients undergoing laparoscopic liver resection or cancer staging. Stereoscopic surface reconstruction and iterative closest point matching facilitated semi-automatic registration. The primary endpoint was defined as successful registration as determined by the operating surgeon. Secondary endpoints were system usability as assessed by a surgeon questionnaire and comparison of manual vs. semi-automatic registration accuracy. Since SmartLiver is still in development no attempt was made to evaluate its impact on perioperative outcomes.

RESULTS

The primary endpoint was achieved in 16 out of 18 patients. Initially semi-automatic registration failed because the IGS could not distinguish the liver surface from surrounding structures. Implementation of a deep learning algorithm enabled the IGS to overcome this issue and facilitate semi-automatic registration. Mean registration accuracy was 10.9 ± 4.2 mm (manual) vs. 13.9 ± 4.4 mm (semi-automatic) (Mean difference - 3 mm; p = 0.158). Surgeon feedback was positive about IGS handling and improved intraoperative orientation but also highlighted the need for a simpler setup process and better integration with laparoscopic ultrasound.

CONCLUSION

The technical feasibility of using SmartLiver intraoperatively has been demonstrated. With further improvements semi-automatic registration may enhance user friendliness and workflow of SmartLiver. Manual and semi-automatic registration accuracy were comparable but evaluation on a larger patient cohort is required to confirm these findings.

Application of computer-assisted navigation in mandibular angle osteotomy

Objective

To compare the effectiveness, accuracy, and surgical safety of a navigation technique with those of a traditional technique for intraoperative mandibular angle osteotomy.

Methods

Forty-three postsurgical patients with mandibular angle hypertrophy who were admitted to our Department from June 2014 to June 2017 were retrospectively reviewed. Of these patients, 23 underwent mandibular angle osteotomy using computer-assisted navigation (navigation group), and 20 underwent osteotomy using a traditional technique (traditional group). Postoperative computed tomography images were analyzed by three-dimensional software. Each patient’s facial proportion indices were measured using Mimics 19.0 software, and statistical comparisons and analyses were performed preoperatively and postoperatively.

Results

The postoperative facial contour morphology and facial proportion were improved in both groups; the navigation group showed greater improvement. The difference between the predicted and postoperative values was smaller in the navigation group than traditional group. The postoperative shape of the mandibular angle sample was similar to the preoperative predicted shape in the navigation group. No complications occurred in the navigation group, but paresthesia occurred in 17% of patients in the traditional group.

Conclusions

Mandibular angle osteotomy aided with computer-assisted navigation is more effective, accurate, and safe than the traditional technique and represents a promising clinical approach.

Precision of a Novel Craniofacial Surgical Navigation System Based on Augmented Reality Using an Occlusal Splint as a Registration Strategy

The authors have developed a novel augmented reality (AR)-based navigation system (NS) for craniofacial surgery. In this study, the authors aimed to measure the precision of the system and further analyze the primary influencing factors of the precision. The drilling of holes into the mandibles of ten beagle dogs was performed under the AR-based NS, and the precision was analyzed by comparing the deviation between the preoperational plan and the surgical outcome. The AR-based NS was successfully applied to quickly and precisely drill holes in the mandibles. The mean positional deviation between the preoperative design and intraoperative navigation was 1.29 ± 0.70 mm for the entry points and 2.47 ± 0.66 mm for the end points, and the angular deviation was 1.32° ± 1.17°. The precision linearly decreased with the distance from the marker. In conclusion, the precision of this system could satisfy clinical requirements, and this system may serve as a helpful tool for improving the precision in craniofacial surgery.

Investigating accuracy of 3D printed liver models with computed tomography

Background

The aim of this study was to evaluate the accuracy of three-dimensional (3D) printed liver models developed by a cost-effective approach for establishing validity of using these models in a clinical setting.

Methods

Fifteen patients undergoing laparoscopic liver resection in a single surgical department were included. Patient-specific, 1-1 scale 3D printed liver models including the liver, tumor, and vasculature were created from contrast-enhanced computed tomography (CT) images using a cost-effective approach. The 3D models were subsequently CT scanned, 3D image post-processing was performed, and these 3D computer models (MCT) were compared to the original 3D models created from the original patient images (PCT). 3D computer models of each type were co-registered using a point set registration method. 3D volume measurements of the liver and lesions were calculated and compared for each set. In addition, Hausdorff distances were calculated and surface quality was compared by generated heatmaps.

Results

The median liver volume in MCT was 1,281.84 [interquartile range (IQR) =296.86] cm³, and 1,448.03 (IQR =413.23) cm³ in PCT. Analysis of differences between surfaces showed that the median value of mean Hausdorff distances for liver parenchyma was 1.92 mm. Bland-Altman plots revealed no significant bias in liver volume and diameters of hepatic veins and tumor location. Median errors of all measured vessel diameters were smaller than CT slice height. There was a slight trend towards undersizing anatomical structures, although those errors are most likely due to source imaging.

Conclusions

We have confirmed the accuracy of 3D printed liver models created by using the low-cost method. 3D models are useful tools for pre-operative planning and intra-operative guidance. Future research in this field should continue to move towards clinical trials for assessment of the impact of these models on pre-surgical planning decisions and perioperative outcomes.

Preliminary study on application of augmented reality visualization in robotic thyroid surgery

Purpose

Increased robotic surgery is attended by increased reports of complications, largely due to limited operative view and lack of tactile sense. These kinds of obstacles, which seldom occur in open surgery, are challenging for beginner surgeons. To enhance robotic surgery safety, we created an augmented reality (AR) model of the organs around the thyroid glands, and tested the AR model applicability in robotic thyroidectomy.

Methods

We created AR images of the thyroid gland, common carotid arteries, trachea, and esophagus using preoperative CT images of a thyroid carcinoma patient. For a preliminary test, we overlaid the AR images on a 3-dimensional printed model at five different angles and evaluated its accuracy using Dice similarity coefficient. We then overlaid the AR images on the real-time operative images during robotic thyroidectomy.

Results

The Dice similarity coefficients ranged from 0.984 to 0.9908, and the mean of the five different angles was 0.987. During the entire process of robotic thyroidectomy, the AR images were successfully overlaid on the real-time operative images using manual registration.

Conclusion

We successfully demonstrated the use of AR on the operative field during robotic thyroidectomy. Although there are currently limitations, the use of AR in robotic surgery will become more practical as the technology advances and may contribute to the enhancement of surgical safety.

Perception enhancement using importance-driven hybrid rendering for augmented reality based endoscopic surgical navigation

Misleading depth perception may greatly affect the correct identification of complex structures in image-guided surgery. In this study, we propose a novel importance-driven hybrid rendering method to enhance perception for navigated endoscopic surgery. First, the volume structures are enhanced using gradient-based shading to reduce the color information in low-priority regions and improve the distinctions between complicated structures. Second, an importance sorting method based on the order-independent transparency rendering is introduced to intensify the perception of multiple surfaces. Third, volume data are adaptively truncated and emphasized with respect to the perspective orientation and the illustration of critical information for viewing range extension. Various experimental results prove that with the combination of volume and surface rendering, our method can effectively improve the depth distinction of multiple objects both in simulated and clinical scenes. Our importance-driven surface rendering method demonstrates improved average performance and statistical significance as rated by 15 participants (five clinicians and ten non-clinicians) on a five-point Likert scale. Further, the average frame rate of hybrid rendering with thin-layer sectioning reaches 42 fps. Given that the process of the hybrid rendering is fully automatic, it can be utilized in real-time surgical navigation to improve the rendering efficiency and information validity.

High Speed, High Density Intraoperative 3D Optical Topographical Imaging with Efficient Registration to MRI and CT for Craniospinal Surgical Navigation

Intraoperative image-guided surgical navigation for craniospinal procedures has significantly improved accuracy by providing an avenue for the surgeon to visualize underlying internal structures corresponding to the exposed surface anatomy. Despite the obvious benefits of surgical navigation, surgeon adoption remains relatively low due to long setup and registration times, steep learning curves, and workflow disruptions. We introduce an experimental navigation system utilizing optical topographical imaging (OTI) to acquire the 3D surface anatomy of the surgical cavity, enabling visualization of internal structures relative to exposed surface anatomy from registered preoperative images. Our OTI approach includes near instantaneous and accurate optical measurement of >250,000 surface points, computed at >52,000 points-per-second for considerably faster patient registration than commercially available benchmark systems without compromising spatial accuracy. Our experience of 171 human craniospinal surgical procedures, demonstrated significant workflow improvement (41 s vs. 258 s and 794 s, p < 0.05) relative to benchmark navigation systems without compromising surgical accuracy. Our advancements provide the cornerstone for widespread adoption of image guidance technologies for faster and safer surgeries without intraoperative CT or MRI scans. This work represents a major workflow improvement for navigated craniospinal procedures with possible extension to other image-guided applications.

Augmented reality in dentistry: a current perspective

Augmentation reality technology offers virtual information in addition to that of the real environment and thus opens new possibilities in various fields. The medical applications of augmentation reality are generally concentrated on surgery types, including neurosurgery, laparoscopic surgery and plastic surgery. Augmentation reality technology is also widely used in medical education and training. In dentistry, oral and maxillofacial surgery is the primary area of use, where dental implant placement and orthognathic surgery are the most frequent applications. Recent technological advancements are enabling new applications of restorative dentistry, orthodontics and endodontics. This review briefly summarizes the history, definitions, features, and components of augmented reality technology and discusses its applications and future perspectives in dentistry.

A Novel Augmented Reality-Based Navigation System in Perforator Flap Transplantation - A Feasibility Study

BACKGROUND

In perforator flap transplantation, dissection of the perforator is an important but difficult procedure because of the high variability in vascular anatomy. Preoperative imaging techniques could provide substantial information about vascular anatomy; however, it cannot provide direct guidance for surgeons during the operation. In this study, a navigation system (NS) was established to overlie a vascular map on surgical sites to further provide a direct guide for perforator flap transplantation.

METHODS

The NS was established based on computed tomographic angiography and augmented reality techniques. A virtual vascular map was reconstructed according to computed tomographic angiography data and projected onto real patient images using ARToolKit software. Additionally, a screw-fixation marker holder was created to facilitate registration. With the use of a tracking and display system, we conducted the NS on an animal model and measured the system error on a rapid prototyping model.

RESULTS

The NS assistance allowed for correct identification, as well as a safe and precise dissection of the perforator. The mean value of the system error was determined to be 3.474 ± 1.546 mm.

CONCLUSIONS

Augmented reality-based NS can provide precise navigation information by directly displaying a 3-dimensional individual anatomical virtual model onto the operative field in real time. It will allow rapid identification and safe dissection of a perforator in free flap transplantation surgery.

Enhanced Visualization: From Intraoperative Tissue Differentiation to Augmented Reality

Background

Optimal visualization of the operative field and methods that additionally provide supportive optical information form the basis for target-directed and successful surgery. This article strives to give an overview of current enhanced visualization techniques in visceral surgery and to highlight future developments.

Methods

The article was written as a comprehensive review on this topic and is based on a MEDLINE search and ongoing research from our own group and from other working groups.

Results

Various techniques for enhanced visualization are described comprising augmented reality, unspecific and targeted staining methods, and optical modalities such as narrow-band imaging. All facilitate our surgical performance; however, due to missing randomized controlled studies for most of the innovations reported on, the available evidence is low.

Conclusion

Many new visualization technologies are emerging with the aim to improve our perception of the surgical field leading to less invasive, target-oriented, and elegant treatment forms that are of significant benefit to our patients.

Virtual and Augmented Reality Systems for Renal Interventions: A Systematic Review

PURPOSE

Many virtual and augmented reality systems have been proposed to support renal interventions. This paper reviews such systems employed in the treatment of renal cell carcinoma and renal stones.

METHODS

A systematic literature search was performed. Inclusion criteria were virtual and augmented reality systems for radical or partial nephrectomy and renal stone treatment, excluding systems solely developed or evaluated for training purposes.

RESULTS

In total, 52 research papers were identified and analyzed. Most of the identified literature (87%) deals with systems for renal cell carcinoma treatment. About 44% of the systems have already been employed in clinical practice, but only 20% in studies with ten or more patients. Main challenges remaining for future research include the consideration of organ movement and deformation, human factor issues, and the conduction of large clinical studies.

CONCLUSION

Augmented and virtual reality systems have the potential to improve safety and outcomes of renal interventions. In the last ten years, many technical advances have led to more sophisticated systems, which are already applied in clinical practice. Further research is required to cope with current limitations of virtual and augmented reality assistance in clinical environments.

Recent Development of Augmented Reality in Surgery: A Review

Introduction

The development augmented reality devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency, safety, and cost and to enhance surgical training. However, the awareness of possibilities of augmented reality is generally low. This review evaluates whether augmented reality can presently improve the results of surgical procedures.

Methods

We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.” Results. The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review.

Conclusions

The present literature suggest an increasing interest of surgeons regarding employing augmented reality into surgery leading to improved safety and efficacy of surgical procedures. Many studies showed that the performance of newly devised augmented reality systems is comparable to traditional techniques. However, several problems need to be addressed before augmented reality is implemented into the routine practice.

Registration and fusion quantification of augmented reality based nasal endoscopic surgery

This paper quantifies the registration and fusion display errors of augmented reality-based nasal endoscopic surgery (ARNES). We comparatively investigated the spatial calibration process for front-end endoscopy and redefined the accuracy level of a calibrated endoscope by using a calibration tool with improved structural reliability. We also studied how registration accuracy was combined with the number and distribution of the deployed fiducial points (FPs) for positioning and the measured registration time. A physically integrated ARNES prototype was customarily configured for performance evaluation in skull base tumor resection surgery with an innovative approach of dynamic endoscopic vision expansion. As advised by surgical experts in otolaryngology, we proposed a hierarchical rendering scheme to properly adapt the fused images with the required visual sensation. By constraining the rendered sight in a known depth and radius, the visual focus of the surgeon can be induced only on the anticipated critical anatomies and vessel structures to avoid misguidance. Furthermore, error analysis was conducted to examine the feasibility of hybrid optical tracking based on point cloud, which was proposed in our previous work as an in-surgery registration solution. Measured results indicated that the error of target registration for ARNES can be reduced to 0.77 ± 0.07 mm. For initial registration, our results suggest that a trade-off for a new minimal time of registration can be reached when the distribution of five FPs is considered. For in-surgery registration, our findings reveal that the intrinsic registration error is a major cause of performance loss. Rigid model and cadaver experiments confirmed that the scenic integration and display fluency of ARNES are smooth, as demonstrated by three clinical trials that surpassed practicality.

Intraoperative Evaluation of Body Surface Improvement by an Augmented Reality System That a Clinician Can Modify

BACKGROUND

Augmented reality (AR) technology that can combine computer-generated images with a real scene has been reported in the medical field recently. We devised the AR system for evaluation of improvements of the body surface, which is important for plastic surgery.

METHODS

We constructed an AR system that is easy to modify by combining existing devices and free software. We superimposed the 3-dimensional images of the body surface and the bone (obtained from VECTRA H1 and CT) onto the actual surgical field by Moverio BT-200 smart glasses and evaluated improvements of the body surface in 8 cases.

RESULTS

In all cases, the 3D image was successfully projected on the surgical field. Improvement of the display method of the 3D image made it easier to distinguish the different shapes in the 3D image and surgical field, making comparison easier. In a patient with fibrous dysplasia, the symmetrized body surface image was useful for confirming improvement of the real body surface. In a patient with complex facial fracture, the simulated bone image was useful as a reference for reduction. In a patient with an osteoma of the forehead, simultaneously displayed images of the body surface and the bone made it easier to understand these positional relationships.

CONCLUSIONS

This study confirmed that AR technology is helpful for evaluation of the body surface in several clinical applications. Our findings are not only useful for body surface evaluation but also for effective utilization of AR technology in the field of plastic surgery.

A novel augmented reality system for displaying inferior alveolar nerve bundles in maxillofacial surgery

Augmented reality systems can combine virtual images with a real environment to ensure accurate surgery with lower risk. This study aimed to develop a novel registration and tracking technique to establish a navigation system based on augmented reality for maxillofacial surgery. Specifically, a virtual image is reconstructed from CT data using 3D software. The real environment is tracked by the augmented reality (AR) software. The novel registration strategy that we created uses an occlusal splint compounded with a fiducial marker (OSM) to establish a relationship between the virtual image and the real object. After the fiducial marker is recognized, the virtual image is superimposed onto the real environment, forming the “integrated image” on semi-transparent glass. Via the registration process, the integral image, which combines the virtual image with the real scene, is successfully presented on the semi-transparent helmet. The position error of this navigation system is 0.96 ± 0.51 mm. This augmented reality system was applied in the clinic and good surgical outcomes were obtained. The augmented reality system that we established for maxillofacial surgery has the advantages of easy manipulation and high accuracy, which can improve surgical outcomes. Thus, this system exhibits significant potential in clinical applications.

New simple image overlay system using a tablet PC for pinpoint identification of the appropriate site for anastomosis in peripheral arterial reconstruction

PURPOSE

To evaluate the accuracy and utility of a new image overlay system using a tablet PC for patients undergoing peripheral arterial reconstruction.

METHODS

Eleven limbs treated with distal bypass surgery were studied. Three-dimensional images obtained by processing a preoperative contrast-enhanced computed tomography scan were superimposed onto the back-camera images of a tablet PC. We used this system to pinpoint a planned distal anastomotic site preoperatively and to make a precise incision directly above it during surgery. We used a branch artery near the distal anastomotic site as a reference point and the accuracy of the system was validated by comparing its results with the intraoperative findings. The precision of the system was also compared with that of a preoperative ultrasonographic examination.

RESULTS

Both the image overlay system and ultrasonography (US) accurately identified the target branch artery in all except one limb. In that limb, which had a very small reference branch artery, preoperative US wrongly identified another branch, whereas the image overlay system located the target branch with an error of 10 mm.

CONCLUSIONS

Our image overlay system was easy to use and allowed us to precisely identify a target artery preoperatively. Therefore, this system could be helpful for pinpointing the most accurate incision site during surgery.

[Importance of preoperative and intraoperative imaging for operative strategies]

Recent advances in preoperative and postoperative imaging have an increasing influence on surgical decision-making and make more complex surgical interventions possible. This improves the possibilities for frequently occurring challenges and promoting improved functional and oncological outcome. This manuscript reviews the role of preoperative and intraoperative imaging in surgery. Various techniques are explained based on examples from hepatobiliary surgery and neurosurgery, in particular real-time procedures, such as the online use of augmented reality and in vivo fluorescence, as well as new and promising optical techniques including imaging of intrinsic signals and vibrational spectroscopy.

Augmented Reality for Assistance of Total Knee Replacement

The aim of this work was the development of a surgical assistance system based on augmented reality to support joint replacement procedures and implantation of prosthetic knee. Images of the scene were captured in order to detect the visual markers located on the lateral surface of the patient’s leg for overlapping the 3D models of the prosthesis and the joint, as well as the tool used by the medical specialist. With the marker identification, it was possible to compute its position and orientation for locating the virtual models, obtaining a monitoring system for giving accurate information about the procedure. Also it can be used as training platform for surgeons, without having volunteers or patients for performing real surgeries; instead they can train in a virtual environment. The results have shown an efficient system in terms of cost-benefit relation, taking into account the materials used for developing the system; nevertheless, the accuracy of the algorithm decreases according to the distance between the markers.

Computer vision and augmented reality in gastrointestinal endoscopy

Augmented reality (AR) is an environment-enhancing technology, widely applied in the computer sciences, which has only recently begun to permeate the medical field. Gastrointestinal endoscopy—which relies on the integration of high-definition video data with pathologic correlates—requires endoscopists to assimilate and process a tremendous amount of data in real time. We believe that AR is well positioned to provide computer-guided assistance with a wide variety of endoscopic applications, beginning with polyp detection. In this article, we review the principles of AR, describe its potential integration into an endoscopy set-up, and envisage a series of novel uses. With close collaboration between physicians and computer scientists, AR promises to contribute significant improvements to the field of endoscopy.