Tracking Accuracy of a Stereo Camera-Based Augmented Reality Navigation System for Orthognathic Surgery

Predicting the Future Focus of Orthognathic Surgery: Outcome-Driven Planning and Treatment With Function, Esthetics, and Occlusion as Key Indicators

As an international group of orthognathic surgeons, we believe the next big thing in orthognathic surgery will be a clinical and research focus on patient-oriented outcomes and improved quality of life. We expect to see advances in diagnosis and treatment planning, materials development, and patient management.

Advanced outcomes of mixed reality usage in orthognathic surgery: a systematic review

INTRODUCTION

Orthognathic surgery (OGS) is a highly sophisticated surgical technique that aims to repair a variety of skeletal and dental abnormalities, including misaligned jaws and teeth. It requires precise preoperative preparation and advanced surgical skills, which are typically learned through years of practical experience in operating rooms or laboratory-based surgical training facilities utilizing cadavers or models. The traditional physical hands-on method of surgical training is still used at OGS. However, this method requires a longer time of preparation. Currently, mixed reality (MR)-a combination of virtual reality and augmented reality technology-is an innovation of OGS. The present study aimed to present a comprehensive review of studies that assessed the advantages of utilizing mixed reality technology in OGS.

METHODS

A modified Population, Intervention, Comparison, Outcome strategy was performed using a combination of electronic (PubMed, Cochrane, Embase) and manual searches between 2013 and 2023 exploring mixed reality (MR) technology in OGS in the last 10 years. The inclusion criteria were limited to the patient and study model focusing on the clinical application of MR and the associated field of OGS.

RESULT

The initial search indicated 1731 studies, of which 17 studies were included for analysis. The main results indicated that the use of MR technology in OGS led to high accuracy and time reduction as primary outcomes and cost-effectiveness and skill improvement as secondary outcomes. The review firmly concluded that MR technology exhibited a positive impact on students, trainees, and oromaxillofacial surgeons. However, due to the heterogeneity of the included studies, meta-analyses could not be performed. Collectively, these findings provide strong evidence for the advantages of MR technology in orthognathic surgery.

CONCLUSION

MR technology significantly improves OGS planning efficiency by providing pre-surgical information and serving as an intraoperative navigation tool, reducing surgical time without compromising outcomes. Virtual training using MR technology exerts a positive impact on knowledge and skill improvement for OGS. This innovative technology will revolutionize the healthcare system and enhance patient care.

First implementation of an innovative infra-red camera system integrated into a mobile CBCT scanner for applicator tracking in brachytherapy-Initial performance characterization

PURPOSE

To enable a real-time applicator guidance for brachytherapy, we used for the first time infra-red tracking cameras (OptiTrack, USA) integrated into a mobile cone-beam computed tomography (CBCT) scanner (medPhoton, Austria). We provide the first description of this prototype and its performance evaluation.

METHODS

We performed assessments of camera calibration and camera-CBCT registration using a geometric calibration phantom. For this purpose, we first evaluated the effects of intrinsic parameters such as camera temperature or gantry rotations on the tracked marker positions. Afterward, calibrations with various settings (sample number, field of view coverage, calibration directions, calibration distances, and lighting conditions) were performed to identify the requirements for achieving maximum tracking accuracy based on an in-house phantom. The corresponding effects on camera-CBCT registration were determined as well by comparing tracked marker positions to the positions determined via CBCT. Long-term stability was assessed by comparing tracking and a ground-truth on a weekly basis for 6 weeks.

RESULTS

Robust tracking with positional drifts of 0.02 ± 0.01 mm was feasible using the system after a warm-up period of 90 min. However, gantry rotations affected the tracking and led to inaccuracies of up to 0.70 mm. We identified that 4000 samples and full coverage were required to ensure a robust determination of marker positions and camera-CBCT registration with geometric deviations of 0.18 ± 0.03 mm and 0.42 ± 0.07 mm, respectively. Long-term stability showed deviations of more than two standard deviations from the initial calibration after 3 weeks.

CONCLUSION

We implemented for the first time a standalone combined camera-CBCT system for tracking in brachytherapy. The system showed high potential for establishing corresponding workflows.

Novel Use of Virtual Reality and Augmented Reality in Temporomandibular Total Joint Replacement Using Stock Prosthesis

This technical innovation demonstrates the use of ImmersiveTouch Virtual Reality (VR) and Augmented Reality (AR)-guided total temporomandibular joint replacement (TJR) using Biomet stock prosthesis in 2 patients with condylar degeneration. TJR VR planning includes condylar resection, prosthesis selection and positioning, and interference identification. AR provides real-time guidance for osteotomies, placement of prostheses and fixation screws, occlusion verification, and flexibility to modify the surgical course. Radiographic analysis demonstrated high correspondence between the preoperative plan and postoperative result. The average differences in the positioning of the condylar and fossa prosthesis are 1.252 ± 0.269 mm and 1.393 ± 0.335 mm, respectively. The main challenges include a steep learning curve, intraoperative technical difficulties, added surgical time, and additional costs. In conclusion, the case report demonstrates the advantages of implementing AR and VR technology in TJR's using stock prostheses as a pilot study. Further clinical trials are needed prior to this innovation becoming a mainstream practice.

Complex Craniofacial Cases through Augmented Reality Guidance in Surgical Oncology: A Technical Report

Augmented reality (AR) is a promising technology to enhance image guided surgery and represents the perfect bridge to combine precise virtual planning with computer-aided execution of surgical maneuvers in the operating room. In craniofacial surgical oncology, AR brings to the surgeon's sight a digital, three-dimensional representation of the anatomy and helps to identify tumor boundaries and optimal surgical paths. Intraoperatively, real-time AR guidance provides surgeons with accurate spatial information, ensuring accurate tumor resection and preservation of critical structures. In this paper, the authors review current evidence of AR applications in craniofacial surgery, focusing on real surgical applications, and compare existing literature with their experience during an AR and navigation guided craniofacial resection, to subsequently analyze which technological trajectories will represent the future of AR and define new perspectives of application for this revolutionizing technology.

The use of virtual reality and augmented reality in oral and maxillofacial surgery: A narrative review

OBJECTIVE

The purpose of this article is to review the current uses of virtual reality (VR) and augmented reality (AR) in oral and maxillofacial surgery. We discuss the use of VR/AR in educational training, surgical planning, advances in hardware and software, and the implementation of VR/AR in this field.

STUDY DESIGN

A retrospective comprehensive review search of PubMed, Web of Science, Embase, and Cochrane Library was conducted. The search resulted in finding 313 English articles in the last 10 years.

RESULTS

A total of 38 articles were selected after a meticulous review of the aims, objectives, and methodology by 2 independent reviewers.

CONCLUSIONS

Virtual reality/AR technology offers significant potential in various aspects, including student education, resident evaluation, surgical planning, and overall surgical implementation. However, its widespread adoption in practice is hindered by factors such as the need for further research, cost concerns, unfamiliarity among current educators, and the necessity for technological improvement. Furthermore, residency programs hold a unique position to influence the future of oral and maxillofacial surgery. As VR/AR has demonstrated substantial benefits in resident education and other applications, residency programs have much to gain by integrating these emerging technologies into their curricula.

Exploring Deep Cervical Compartments in Head and Neck Surgical Oncology through Augmented Reality Vision: A Proof of Concept

BACKGROUND

Virtual surgical planning allows surgeons to meticulously define surgical procedures by creating a digital replica of patients' anatomy. This enables precise preoperative assessment, facilitating the selection of optimal surgical approaches and the customization of treatment plans. In neck surgery, virtual planning has been significantly underreported compared to craniofacial surgery, due to a multitude of factors, including the predominance of soft tissues, the unavailability of intraoperative navigation and the complexity of segmenting such areas. Augmented reality represents the most innovative approach to translate virtual planning for real patients, as it merges the digital world with the surgical field in real time. Surgeons can access patient-specific data directly within their field of view, through dedicated visors. In head and neck surgical oncology, augmented reality systems overlay critical anatomical information onto the surgeon's visual field. This aids in locating and preserving vital structures, such as nerves and blood vessels, during complex procedures. In this paper, the authors examine a series of patients undergoing complex neck surgical oncology procedures with prior virtual surgical planning analysis. For each patient, the surgical plan was imported in Hololens headset to allow for intraoperative augmented reality visualization. The authors discuss the results of this preliminary investigation, tracing the conceptual framework for an increasing AR implementation in complex head and neck surgical oncology procedures.

A Custom-Made Surgical Guide for Accurate Enucleation of Nasopalatine Duct Cysts: A Technical Note and Case Report

Nasopalatine cysts are common nonodontogenic cysts that occur in the maxilla. During the nucleation of large cysts extending to the floor of the nasal cavity, care must be taken to avoid damage to the nasal mucosa. In the present report, an innovative custom-made surgical guide made by a Three-dimensional printer is introduced for accurate enucleation surgery. The patient's cone-beam computerized tomography and dental model scan data were obtained, and a tooth-supported type of surgical guide was designed containing a circular plate structure showing the size of the cystic region, an indicator that showed the position of the bottom of the cyst, and a sliding stopper that was used to accurately indicate the position of the deepest cyst wall. The surgical tool enabled us to indicate the accurate size, location of the cysts, and approach direction. Although effective and accurate navigation systems have become increasingly available, the cost-effective and accurate computer-aided design/computer-aided manufacturing surgical guide system introduced in the present report could support the safe enucleation of large nasopalatine duct cysts.

Collaborative Control Method and Experimental Research on Robot-Assisted Craniomaxillofacial Osteotomy Based on the Force Feedback and Optical Navigation

Surgical robot has advantages in high accuracy and stability. But during the robot-assisted bone surgery, the lack of force information from surgical area and incapability of intervention from surgeons become the obstacle. The aim of the study is to introduce a collaborative control method based on the force feedback and optical navigation, which may optimally combine the excellent performance of surgical robot with clinical experiences of surgeons.

Effect of marker position and size on the registration accuracy of HoloLens in a non-clinical setting with implications for high-precision surgical tasks

Purpose

Emerging holographic headsets can be used to register patient-specific virtual models obtained from medical scans with the patient’s body. Maximising accuracy of the virtual models’ inclination angle and position (ideally, ≤ 2° and ≤ 2 mm, respectively, as in currently approved navigation systems) is vital for this application to be useful. This study investigated the accuracy with which a holographic headset registers virtual models with real-world features based on the position and size of image markers.

Methods

HoloLens^® and the image-pattern-recognition tool Vuforia Engine™ were used to overlay a 5-cm-radius virtual hexagon on a monitor’s surface in a predefined position. The headset’s camera detection of an image marker (displayed on the monitor) triggered the rendering of the virtual hexagon on the headset’s lenses. 4 × 4, 8 × 8 and 12 × 12 cm image markers displayed at nine different positions were used. In total, the position and dimensions of 114 virtual hexagons were measured on photographs captured by the headset’s camera.

Results

Some image marker positions and the smallest image marker (4 × 4 cm) led to larger errors in the perceived dimensions of the virtual models than other image marker positions and larger markers (8 × 8 and 12 × 12 cm). ≤ 2° and ≤ 2 mm errors were found in 70.7% and 76% of cases, respectively.

Conclusion

Errors obtained in a non-negligible percentage of cases are not acceptable for certain surgical tasks (e.g. the identification of correct trajectories of surgical instruments). Achieving sufficient accuracy with image marker sizes that meet surgical needs and regardless of image marker position remains a challenge.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11548-021-02354-9.

Error Analysis of Robot-Assisted Orthognathic Surgery

OBJECTIVE

Orthognathic surgery is an effective method to correct the dentomaxillofacial deformities. The aim of the study is to introduce the robot-assisted orthognathic surgery and demonstrate the accuracy and feasibility of robot-assisted osteotomy in transferring the preoperative virtual surgical planning (VSP) into the intraoperative phase.

METHODS

The CMF robot system, a craniomaxillofacial surgical robot system was developed, consisted of a robotic arm with 6 degrees of freedom, a self-developed end-effector, and an optical localizer. The individualized end-effector was installed with reciprocating saw so that it could perform osteotomy. The study included control and experimental groups. In control group, under the guidance of navigation system, surgeon performed the osteotomies on 3 skull models. In experimental group, according to the preoperative VSP, the robot completed the osteotomies on 3 skull models automatically with assistance of navigation. Statistical analysis was carried out to evaluate the accuracy and feasibility of robot-assisted orthognathic surgery and compare the errors between robot-assisted automatic osteotomy and navigation-assisted manual osteotomy.

RESULTS

All the osteotomies were successfully completed. The overall osteotomy error was 1.07 ± 0.19 mm in the control group, and 1.12 ± 0.20 mm in the experimental group. No significant difference in osteotomy errors was found in the robot-assisted osteotomy groups (P = 0.353). There was consistence of errors between robot-assisted automatic osteotomy and navigation-assisted manual osteotomy.

CONCLUSION

In robot-assisted orthognathic surgery, the robot can complete an osteotomy according to the preoperative VSP and transfer a preoperative VSP into the actual surgical operation with good accuracy and feasibility.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Augmented reality in craniomaxillofacial surgery: added value and proposed recommendations through a systematic review of the literature

This systematic review provides an overview of augmented reality (AR) and its benefits in craniomaxillofacial surgery in an attempt to answer the question: Is AR beneficial for craniomaxillofacial surgery? This review includes a description of the studies conducted, the systems used and their technical characteristics. The search was performed in four databases: PubMed, Cochrane Library, Embase, and Web of Science. All journal articles published during the past 11 years related to AR, mixed reality, craniomaxillofacial, and surgery were considered in this study. From a total of 7067 articles identified using AR- and surgery-related keywords, 39 articles were finally selected. Based on these articles, a classification of study types, surgery types, devices used, metrics reported, and benefits were collected. The findings of this review indicate that AR could provide various benefits, addressing the challenges of conventional navigation systems, such as hand-eye coordination and depth perception. However, three main concerns were raised while performing this study: (1) it is complicated to aggregate the metrics reported in the articles, (2) it is difficult to obtain statistical value from the current studies, and (3) user evaluation studies are lacking. This article concludes with recommendations for future studies by addressing the latter points.

The use of 3D virtual surgical planning and computer aided design in reconstruction of maxillary surgical defects

PURPOSE OF REVIEW

The present review describes the latest development of 3D virtual surgical planning (VSP) and computer aided design (CAD) for reconstruction of maxillary defects with an aim of fully prosthetic rehabilitation. The purpose is to give an overview of different methods that use CAD in maxillary reconstruction in patients with head and neck cancer.

RECENT FINDINGS

3D VSP enables preoperative planning of resection margins and osteotomies. The current 3D VSP workflow is expanded with multimodal imaging, merging decision supportive information. Development of more personalized implants is possible using CAD, individualized virtual muscle modelling and topology optimization. Meanwhile the translation of the 3D VSP towards surgery is improved by techniques like intraoperative imaging and augmented reality. Recent improvements of preoperative 3D VSP enables surgical reconstruction and/or prosthetic rehabilitation of the surgical defect in one combined procedure.

SUMMARY

With the use of 3D VSP and CAD, ablation surgery, reconstructive surgery, and prosthetic rehabilitation can be planned preoperatively. Many reconstruction possibilities exist and a choice depends on patient characteristics, tumour location and experience of the surgeon. The overall objective in patients with maxillary defects is to follow a prosthetic-driven reconstruction with the aim to restore facial form, oral function, and do so in accordance with the individual needs of the patient.

Frameless Patient Tracking With Adhesive Optical Skin Markers for Augmented Reality Surgical Navigation in Spine Surgery

STUDY DESIGN

Observational study.

OBJECTIVE

The aim of this study was to evaluate the accuracy of a new frameless reference marker system for patient tracking by analyzing the effect of vertebral position within the surgical field.

SUMMARY OF BACKGROUND DATA

Most modern navigation systems for spine surgery rely on a dynamic reference frame attached to a vertebra for tracking the patient. This solution has the drawback of being bulky and obstructing the surgical field, while requiring that the dynamic reference frame is moved between vertebras to maintain accuracy.

METHODS

An augmented reality surgical navigation (ARSN) system with intraoperative cone beam computed tomography (CBCT) capability was installed in a hybrid operating room. The ARSN system used input from four video cameras for tracking adhesive skin markers placed around the surgical field. The frameless reference marker system was evaluated first in four human cadavers, and then in 20 patients undergoing navigated spine surgery. In each CBCT, the impact of vertebral position in the surgical field on technical accuracy was analyzed. The technical accuracy of the inserted pedicle devices was determined by measuring the distance between the planned position and the placed pedicle device, at the bone entry point.

RESULTS

The overall mean technical accuracy was 1.65 ± 1.24 mm at the bone entry point (n = 366). There was no statistically significant difference in technical accuracy between levels within CBCTs (P ≥ 0.12 for all comparisons). Linear regressions showed that null- to negligible parts of the effect on technical accuracy could be explained by the number of absolute levels away from the index vertebrae (r ≤ 0.007 for all, β ≤ 0.071 for all).

CONCLUSION

The frameless reference marker system based on adhesive skin markers is unobtrusive and affords the ARSN system a high accuracy throughout the navigated surgical field, independent of vertebral position.

LEVEL OF EVIDENCE

3.

Three-dimensional virtual surgical planning in the oncologic treatment of the mandible

OBJECTIVES

In case of surgical removal of oral squamous cell carcinomas, a resection of mandibular bone is frequently part of the treatment. Nowadays, such resections frequently include the application of 3D virtual surgical planning (VSP) and guided surgery techniques. In this paper, current methods for 3D VSP leads for optimisation of the workflow, and patient-specific application of guides and implants are reviewed.

RECENT FINDINGS

Current methods for 3D VSP enable multi-modality fusion of images. This fusion of images is not restricted to a specific software package or workflow. New strategies for 3D VSP in Oral and Maxillofacial Surgery include finite element analysis, deep learning and advanced augmented reality techniques. These strategies aim to improve the treatment in terms of accuracy, predictability and safety.

CONCLUSIONS

Application of the discussed novel technologies and strategies will improve the accuracy and safety of mandibular resection and reconstruction planning. Accurate, easy-to-use, safe and efficient three-dimensional VSP can be applied for every patient with malignancies needing resection of the mandible.