Augmented reality for image guidance in transoral robotic surgery

A review on organ deformation modeling approaches for reliable surgical navigation using augmented reality

Augmented Reality (AR) holds the potential to revolutionize surgical procedures by allowing surgeons to visualize critical structures within the patient's body. This is achieved through superimposing preoperative organ models onto the actual anatomy. Challenges arise from dynamic deformations of organs during surgery, making preoperative models inadequate for faithfully representing intraoperative anatomy. To enable reliable navigation in augmented surgery, modeling of intraoperative deformation to obtain an accurate alignment of the preoperative organ model with the intraoperative anatomy is indispensable. Despite the existence of various methods proposed to model intraoperative organ deformation, there are still few literature reviews that systematically categorize and summarize these approaches. This review aims to fill this gap by providing a comprehensive and technical-oriented overview of modeling methods for intraoperative organ deformation in augmented reality in surgery. Through a systematic search and screening process, 112 closely relevant papers were included in this review. By presenting the current status of organ deformation modeling methods and their clinical applications, this review seeks to enhance the understanding of organ deformation modeling in AR-guided surgery, and discuss the potential topics for future advancements.

Towards safer robot-assisted surgery: A markerless augmented reality framework

Robot-assisted surgery is rapidly developing in the medical field, and the integration of augmented reality shows the potential to improve the operation performance of surgeons by providing more visual information. In this paper, we proposed a markerless augmented reality framework to enhance safety by avoiding intra-operative bleeding, which is a high risk caused by collision between surgical instruments and delicate blood vessels (arteries or veins). Advanced stereo reconstruction and segmentation networks are compared to find the best combination to reconstruct the intra-operative blood vessel in 3D space for registration with the pre-operative model, and the minimum distance detection between the instruments and the blood vessel is implemented. A robot-assisted lymphadenectomy is emulated on the da Vinci Research Kit in a dry lab, and ten human subjects perform this operation to explore the usability of the proposed framework. The result shows that the augmented reality framework can help the users to avoid the dangerous collision between the instruments and the delicate blood vessel while not introducing an extra load. It provides a flexible framework that integrates augmented reality into the medical robotic platform to enhance safety during surgery.

Enhancing Surgical Vision: Augmented Reality in Otolaryngology-Head and Neck Surgery

Augmented reality (AR) technology has become widely established in otolaryngology-head and neck surgery. Over the past 20 years, numerous AR systems have been investigated and validated across the subspecialties, both in cadaveric and in live surgical studies. AR displays projected through head-mounted devices, microscopes, and endoscopes, most commonly, have demonstrated utility in preoperative planning, intraoperative guidance, and improvement of surgical decision-making. Specifically, they have demonstrated feasibility in guiding tumor margin resections, identifying critical structures intraoperatively, and displaying patient-specific virtual models derived from preoperative imaging, with millimetric accuracy. This review summarizes both established and emerging AR technologies, detailing how their systems work, what features they offer, and their clinical impact across otolaryngology subspecialties. As AR technology continues to advance, its integration holds promise for enhancing surgical precision, simulation training, and ultimately, improving patient outcomes.

Perception of Young European Otolaryngologists toward Transoral Robotic Surgery in Head and Neck Oncology and Surgery

Background: To investigate the perception of young European otolaryngologists (OTOs), i.e., head and neck surgeons, toward transoral robotic surgery (TORS). Methods: Members of the Young Confederation of European Otorhinolaryngology-Head and Neck Surgery and Young Otolaryngologists of International Federation of Otorhinolaryngological Societies were surveyed about TORS perception and practice. Results: The survey was completed by 120 young OTOS (26%). The most important barriers to TORS were robot availability (73%), cost (69%), and lack of training (37%). The participants believed that the main benefits include better surgical filed view (64%), shorter hospital stay (62%), and better postoperative outcomes (61%) than the conventional approach. Head and neck surgeons considered cT1-T2 oropharyngeal cancers (94%), resection of base of tongue for sleep apnea (86%), or primary unknown cancer (76%) as the most appropriate indications. A total of 67% of TORS surgeons assessed themselves as adequately trained in TORS. Conclusions: Young European OTOs report positive perception, adoption, and knowledge of TORS. The cost-related unavailability and the lack of training or access are reported to be the most important barriers for the spread of TORS.

Development of an augmented reality guidance system for head and neck cancer resection

The use of head-mounted augmented reality (AR) for surgeries has grown rapidly in recent years. AR aids in intraoperative surgical navigation through overlaying three-dimensional (3D) holographic reconstructions of medical data. However, performing AR surgeries on complex areas such as the head and neck region poses challenges in terms of accuracy and speed. This study explores the feasibility of an AR guidance system for resections of positive tumour margins in a cadaveric specimen. The authors present an intraoperative solution that enables surgeons to upload and visualize holographic reconstructions of resected cadaver tissues. The solution involves using a 3D scanner to capture detailed scans of the resected tissue, which are subsequently uploaded into our software. The software converts the scans of resected tissues into specimen holograms that are viewable through a head-mounted AR display. By re-aligning these holograms with cadavers with gestures or voice commands, surgeons can navigate the head and neck tumour site. This workflow can run concurrently with frozen section analysis. On average, the authors achieve an uploading time of 2.98 min, visualization time of 1.05 min, and re-alignment time of 4.39 min, compared to the 20 to 30 min typical for frozen section analysis. The authors achieve a mean re-alignment error of 3.1 mm. The authors' software provides a foundation for new research and product development for using AR to navigate complex 3D anatomy in surgery.

Towards transcervical ultrasound image guidance for transoral robotic surgery

PURPOSE

Trans-oral robotic surgery (TORS) using the da Vinci surgical robot is a new minimally-invasive surgery method to treat oropharyngeal tumors, but it is a challenging operation. Augmented reality (AR) based on intra-operative ultrasound (US) has the potential to enhance the visualization of the anatomy and cancerous tumors to provide additional tools for decision-making in surgery.

METHODS

We propose a US-guided AR system for TORS, with the transducer placed on the neck for a transcervical view. Firstly, we perform a novel MRI-to-transcervical 3D US registration study, comprising (i) preoperative MRI to preoperative US registration, and (ii) preoperative to intraoperative US registration to account for tissue deformation due to retraction. Secondly, we develop a US-robot calibration method with an optical tracker and demonstrate its use in an AR system that displays anatomy models in the surgeon's console in real-time.

RESULTS

Our AR system achieves a projection error from the US to the stereo cameras of 27.14 and 26.03 pixels (image is 540[Formula: see text]960) in a water bath experiment. The average target registration error (TRE) for MRI to 3D US is 8.90 mm for the 3D US transducer and 5.85 mm for freehand 3D US, and the TRE for pre-intra operative US registration is 7.90 mm.

CONCLUSION

We demonstrate the feasibility of each component of the first complete pipeline for MRI-US-robot-patient registration for a proof-of-concept transcervical US-guided AR system for TORS. Our results show that trans-cervical 3D US is a promising technique for TORS image guidance.

Intraoperative Imaging Techniques to Improve Surgical Resection Margins of Oropharyngeal Squamous Cell Cancer: A Comprehensive Review of Current Literature

Inadequate resection margins in head and neck squamous cell carcinoma surgery necessitate adjuvant therapies such as re-resection and radiotherapy with or without chemotherapy and imply increasing morbidity and worse prognosis. On the other hand, taking larger margins by extending the resection also leads to avoidable increased morbidity. Oropharyngeal squamous cell carcinomas (OPSCCs) are often difficult to access; resections are limited by anatomy and functionality and thus carry an increased risk for close or positive margins. Therefore, there is a need to improve intraoperative assessment of resection margins. Several intraoperative techniques are available, but these often lead to prolonged operative time and are only suitable for a subgroup of patients. In recent years, new diagnostic tools have been the subject of investigation. This study reviews the available literature on intraoperative techniques to improve resection margins for OPSCCs. A literature search was performed in Embase, PubMed, and Cochrane. Narrow band imaging (NBI), high-resolution microendoscopic imaging, confocal laser endomicroscopy, frozen section analysis (FSA), ultrasound (US), computed tomography scan (CT), (auto) fluorescence imaging (FI), and augmented reality (AR) have all been used for OPSCC. NBI, FSA, and US are most commonly used and increase the rate of negative margins. Other techniques will become available in the future, of which fluorescence imaging has high potential for use with OPSCC.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Augmented Reality and Intraoperative Navigation in Sinonasal Malignancies: A Preclinical Study

Objective

To report the first use of a novel projected augmented reality (AR) system in open sinonasal tumor resections in preclinical models and to compare the AR approach with an advanced intraoperative navigation (IN) system.

Methods

Four tumor models were created. Five head and neck surgeons participated in the study performing virtual osteotomies. Unguided, AR, IN, and AR + IN simulations were performed. Statistical comparisons between approaches were obtained. Intratumoral cut rate was the main outcome. The groups were also compared in terms of percentage of intratumoral, close, adequate, and excessive distances from the tumor. Information on a wearable gaze tracker headset and NASA Task Load Index questionnaire results were analyzed as well.

Results

A total of 335 cuts were simulated. Intratumoral cuts were observed in 20.7%, 9.4%, 1.2,% and 0% of the unguided, AR, IN, and AR + IN simulations, respectively (p < 0.0001). The AR was superior than the unguided approach in univariate and multivariate models. The percentage of time looking at the screen during the procedures was 55.5% for the unguided approaches and 0%, 78.5%, and 61.8% in AR, IN, and AR + IN, respectively (p < 0.001). The combined approach significantly reduced the screen time compared with the IN procedure alone.

Conclusion

We reported the use of a novel AR system for oncological resections in open sinonasal approaches, with improved margin delineation compared with unguided techniques. AR improved the gaze-toggling drawback of IN. Further refinements of the AR system are needed before translating our experience to clinical practice.

Case-specific three-dimensional hologram with a mixed reality technique for tumor resection in otolaryngology

OBJECTIVE

We report our first experience of using a case-specific three-dimensional (3D) hologram for tumor resection in otolaryngology to show the proof of concept. In addition, a questionnaire was administered to assess the usefulness of the mixed reality technique in otolaryngology.

METHODS

A case-specific 3D hologram was developed from enhanced images of dynamic computed tomography, with reference to contrast-enhanced magnetic resonance images, and used for preoperative planning and intraoperative image reference. To evaluate the usefulness of the 3D hologram with head mount displays (HMDs), 18 attendings and resident otolaryngologists completed a questionnaire with the Likert scale.

RESULTS

The case-specific 3D hologram on HMDs was successfully used by means of easy gesture-handling without any monitors preoperatively and intraoperatively. The experience of picturing the tumor localization and evaluating the surgical approach was statistically better using the 3D hologram on HMDs than using the computer images (P < .01). Similarly, the holograms were observed to be better for intraoperative application and surgical education than computer images (P < .01).

CONCLUSION

We demonstrated the use of a case-specific 3D hologram for tumor resection in otolaryngology. The technology may be useful for preoperative planning and intraoperative image reference, especially for challenging cases, and surgical education.

LEVEL OF EVIDENCE

NA.

Imaging-based navigation technologies in head and neck surgery

PURPOSE OF REVIEW

Image guided navigation has had significant impact in head and neck surgery, and has been most prolific in endonasal surgeries. Although conventional image guidance involves static computed tomography (CT) images attained in the preoperative setting, the continual evolution of surgical navigation technologies is fast expanding to incorporate both real-time data and bioinformation that allows for improved precision in surgical guidance. With the rapid advances in technologies, this article allows for a timely review of the current and developing techniques in surgical navigation for head and neck surgery.

RECENT FINDINGS

Current advances for cross-sectional-based image-guided surgery include fusion of CT with other imaging modalities (e.g., magnetic resonance imaging and positron emission tomography) as well as the uptake in intraoperative real-time 'on the table' imaging (e.g., cone-beam CT). These advances, together with the integration of virtual/augmented reality, enable potential enhancements in surgical navigation. In addition to the advances in radiological imaging, the development of optical modalities such as fluorescence and spectroscopy techniques further allows the assimilation of biological data to improve navigation particularly for head and neck surgery.

SUMMARY

The steady development of radiological and optical imaging techniques shows great promise in changing the paradigm of head and neck surgery.

Intraoperative 3-dimensional Projection of Blood Vessels on Body Surface Using an Augmented Reality System

Preoperative understanding of the running pattern of blood vessels is an important factor to approach surgical fields safely. In 2 cases where the vascular abnormalities were estimated, we projected the blood vessels onto the surgical field using an augmented reality device HoloLens. A splint was made to allow the patient to be fixed while undergoing computed tomographic angiography. Three-dimensional (3D) data on the blood vessels, skin surfaces, bones, and the 3 chosen points for alignment were segmented and then projected onto the body surfaces as holograms using the HoloLens. Two types of projection for holograms were used: projection type 1-where the body contours were projected as a line, and projection type 2-where the body surface was projected as meshed skin type. By projecting projection type 2 rather than projection type 1, we gained a better understanding of the 3D anatomic findings and deformation characteristics, including the anatomic blood vessel variation and positional relationships between the organs and body surfaces. To some extent, we could make sure that the depth perception can be obtained by recognizing the bone, vessels, or tumor inside the meshed skin surface. Our new method allows the 3D visualization of blood vessels from the body surface, and helps understand the 3D anatomic variation of the blood vessels to be applied as long as the blood vessels can be visualized.

[Application of mixed reality in oromaxillofacial head and neck oncology surgery: a preliminary study]

Mixed reality (MR), characterized by the ability to integrate digital data into human real feeling, is a new technique in medical imaging and surgical navigation. MR has tremendous value in surgery, but its application in oromaxillofacial head and neck oncology surgery is not yet reported. This paper reports the application of MR in oromaxillofacial head and neck oncology surgery. The merits, demerits, and present research situations and prospects of MR are further discussed.

Surgical Algorithm for Obstructive Sleep Apnea: An Update

Sleep surgery is part of a continuum of care for obstructive sleep apnea (OSA) that involves medical, pharmacologic, and behavioral therapy. Upper airway surgery for OSA can significantly improve stability by way of modulating the critical negative closing pressure. This is the same mechanism of action as positive airway pressure or oral appliance therapy. The updated surgical algorithm in this review adds precision in three areas: patient selection, identification of previously unaddressed anatomic phenotypes with associated treatment modality, and improved techniques of previously established procedures. While the original Riley and Powell phase 1 and 2 approach to sleep surgery has focused on individual surgical success rate, this algorithm strives for an overall treatment success with multi-modal and patient-centric treatments.