Head-Mounted Mixed-Reality Technology During Robotic-Assisted Transanal Total Mesorectal Excision

The use of mixed reality in the preoperative planning of colorectal surgery: Preliminary experience with a narrative review

New advanced technologies have recently been developed and preliminarily applied to surgery, including virtual reality (VR), augmented reality (AR) and mixed reality (MR). We retrospectively review all colorectal cases in which we used holographic 3D reconstruction from February 2020 to December 2022. This innovative approach was used to identify vascular anomalies, pinpoint tumor locations, evaluate infiltration into neighboring organs and devise surgical plans for both training and educating trainee assistants. We have also provided a state-of-the-art analysis, briefly highlighting what has been stated by the scientific literature to date. VR facilitates training and anatomical assessments, while AR enhances training and laparoscopic performance evaluations. MR, powered by HoloLens, enriches anatomic recognition, navigation, and visualization. Successful implementation was observed in 10 colorectal cancer cases, showcasing the effectiveness of MR in improving preoperative planning and its intraoperative application. This technology holds significant promise for advancing colorectal surgery by elevating safety and reliability standards.

Artificial Intelligence-Based Hazard Detection in Robotic-Assisted Single-Incision Oncologic Surgery

THE PROBLEM

Single-incision surgery is a complex procedure in which any additional information automatically collected from the operating field can be of significance. While the use of robotic devices has greatly improved surgical outcomes, there are still many unresolved issues. One of the major surgical complications, with higher occurrence in cancer patients, is intraoperative hemorrhages, which if detected early, can be more efficiently controlled.

AIM

This paper proposes a hazard detection system which incorporates the advantages of both Artificial Intelligence (AI) and Augmented Reality (AR) agents, capable of identifying, in real-time, intraoperative bleedings, which are subsequently displayed on a Hololens 2 device.

METHODS

The authors explored the different techniques for real-time processing and determined, based on a critical analysis, that YOLOv5 is one of the most promising solutions. An innovative, real-time, bleeding detection system, developed using the YOLOv5 algorithm and the Hololens 2 device, was evaluated on different surgical procedures and tested in multiple configurations to obtain the optimal prediction time and accuracy.

RESULTS

The detection system was able to identify the bleeding occurrence in multiple surgical procedures with a high rate of accuracy. Once detected, the area of interest was marked with a bounding box and displayed on the Hololens 2 device. During the tests, the system was able to differentiate between bleeding occurrence and intraoperative irrigation; thus, reducing the risk of false-negative and false-positive results.

CONCLUSION

The current level of AI and AR technologies enables the development of real-time hazard detection systems as efficient assistance tools for surgeons, especially in high-risk interventions.

Mixed Reality in Modern Surgical and Interventional Practice: Narrative Review of the Literature

BACKGROUND

Mixed reality (MR) and its potential applications have gained increasing interest within the medical community over the recent years. The ability to integrate virtual objects into a real-world environment within a single video-see-through display is a topic that sparks imagination. Given these characteristics, MR could facilitate preoperative and preinterventional planning, provide intraoperative and intrainterventional guidance, and aid in education and training, thereby improving the skills and merits of surgeons and residents alike.

OBJECTIVE

In this narrative review, we provide a broad overview of the different applications of MR within the entire spectrum of surgical and interventional practice and elucidate on potential future directions.

METHODS

A targeted literature search within the PubMed, Embase, and Cochrane databases was performed regarding the application of MR within surgical and interventional practice. Studies were included if they met the criteria for technological readiness level 5, and as such, had to be validated in a relevant environment.

RESULTS

A total of 57 studies were included and divided into studies regarding preoperative and interventional planning, intraoperative and interventional guidance, as well as training and education.

CONCLUSIONS

The overall experience with MR is positive. The main benefits of MR seem to be related to improved efficiency. Limitations primarily seem to be related to constraints associated with head-mounted display. Future directions should be aimed at improving head-mounted display technology as well as incorporation of MR within surgical microscopes, robots, and design of trials to prove superiority.

Augmented reality hologram combined with pre-bent distractor enhanced the accuracy of distraction vector transfer in maxillary distraction osteogenesis, a study based on 3D printed phantoms

BACKGROUND

Vector control is a significant concern in maxillary distraction osteogenesis (DO). Distraction vector planning on the patient's 3D-printed skull phantom is more intuitive for surgeons and cost-efficient than virtual surgical planning. However, the accuracy of transferring the planned vector to intraoperative (vector transfer) according to the shape of the pre-bent footplate alone is relatively limited. The application of augmented reality (AR) in surgical navigation has been studied for years. However, few studies have focused on its role in maxillary DO vector transfer. This study aimed to evaluate the accuracy of AR surgical navigation combined with the pre-bent distractor in vector transfer by comparing it with the pre-bent distractor alone.

METHODS

Ten patients with maxillary hypoplasia were enrolled with consent, and three identical 3D-printed skull phantoms were manufactured based on per patient's corresponding pre-operative CT data. Among these, one phantom was for pre-operative planning (n = 10), while and the other two were for the AR+Pre-bending group (n = 10) and the Pre-bending group (n = 10) for the experimental surgery, respectively. In the Pre-bending group, the distraction vector was solely determined by matching the shape of footplates and maxillary surface. In the AR+Pre-bending group, the distractors were first confirmed to have no deformation. Then AR surgical navigation was applied to check and adjust the vector in addition to the steps as in the Pre-bending Group.

RESULTS

For the angular deviation of the distraction vector, the AR+Pre-bending group was significantly smaller than the Pre-bending group in spatial (p < 0.001), x-y plane (p = 0.002), and y-z plane (p < 0.001), and there were no significant differences in the x-z plane (p = 0.221). The AR+Pre-bending group was more accurate in deviations of the Euclidean distance (p = 0.004) and the y-axis (p = 0.011). In addition, the AR+Pre-bending group was more accurate for the distraction result.

CONCLUSIONS

In this study based on 3D printed skull phantoms, the AR surgical navigation combined with the pre-bent distractor enhanced the accuracy of vector transfer in maxillary DO, compared with the pre-bending technique alone.

Practical Application of Augmented/Mixed Reality Technologies in Surgery of Abdominal Cancer Patients

The technology of augmented and mixed reality (AR/MR) is useful in various areas of modern surgery. We considered the use of augmented and mixed reality technologies as a method of preoperative planning and intraoperative navigation in abdominal cancer patients. Practical use of AM/MR raises a range questions, which demand suitable solutions. The difficulties and obstacles we encountered in the practical use of AR/MR are presented, along with the ways we chose to overcome them. The most demonstrative case is covered in detail. The three-dimensional anatomical model obtained from the CT scan needed to be rigidly attached to the patient’s body, and therefore an invasive approach was developed, using an orthopedic pin fixed to the pelvic bones. The pin is used both similarly to an X-ray contrast marker and as a marker for augmented reality. This solution made it possible, not only to visualize the anatomical structures of the patient and the border zone of the tumor, but also to change the position of the patient during the operation. In addition, a noninvasive (skin-based) marking method was developed that allows the application of mixed and augmented reality during operation. Both techniques were used (8 clinical cases) for preoperative planning and intraoperative navigation, which allowed surgeons to verify the radicality of the operation, to have visual control of all anatomical structures near the zone of interest, and to reduce the time of surgical intervention, thereby reducing the complication rate and improving the rehabilitation period.

Mixed Reality: A Step Further for Planning Complex Renal Tumors(RENAL nephrometry score of 7 or higher)

OBJECTIVES

Mixed reality (MR) technology has emerged in recent years and allows three-dimensional visualization, multi-angle observation, remote vision, and virtual-real interaction. This study aims to explore the influence of MR technology on the outcomes and strategy planning of robotic surgery for complex renal tumors.

PATIENTS AND METHODS

A total of 92 patients with complex renal tumors were enrolled in this study from June 2018 to June 2020. All patients were diagnosed in our department by magnetic resonance imaging（MRI）. This trial follows CONSORT guidelines and adopts a parallel single blind design and randomizes patients with a random number table. The study was approved by the institutional review board, and written informed consent was obtained from each participant. All surgeries were performed by 3 experienced and high-volume surgeons. The demographic indicators, intraoperative and postoperative complications, renal function outcomes, pathological results, and surgical strategies were recorded. Student's t-test and Wilcoxon rank-sum were used to compare continuous variables, and Pearson's chi-squared and Fisher's exact tests were used to compare categorical variables.

RESULTS

WIT is mainly composed of tumor resection time and reconstruction time, and the reconstruction time accounts for a larger proportion. For urologists treating complex renal tumors, MR technology can help them reduce warm ischemia time (21.3±4.0 vs 23.6±5.9min, P=0.031), reconstruction time (15.4±3.8 vs 17.2±4.2min, P=0.034), estimated blood loss (P=0.044), operation time (125.7±26.3 vs 144.6±27.9min, P=0.001) and intraoperative complications (P=0.030).

CONCLUSIONS

MR assisted surgery can reduce the incidence of intraoperative complications, improve perioperative outcomes, which may be a good preoperative tool for planning complex renal tumors.

[Perioperative augmented reality technology in surgical treatment of locally advanced recurrent rectal cancer]

Rectal cancer occupies the leading position among cancers, and incidence of locally advanced recurrences is still high despite comprehensive treatment. Combined resections are usually associated with high perioperative risks. These procedures are technically complex interventions requiring further improvement. Virtual reality technology in surgical treatment of locally advanced rectal cancer recurrence has not been widely discussed. The authors present multidisciplinary construction of the matched topographic-anatomical virtual model and virtual planning of the combined surgical intervention. Intraoperative use of augmented reality allowed specifying topographic and anatomical features of surgical area, level of vascular ligation, localization of tumor fixation points and resection borders. These data ensured safety and quality of resection. Further research of augmented reality technology and improvement of its technical aspects will improve the results of surgical treatment of patients with locally advanced pelvic tumors and recurrences.

Imaging to assist fertility-sparing surgery

Cytological screening and human papilloma virus testing has led to diagnosis of cervical cancer in young women at an earlier stage. Defining the full extent of the disease within the cervix with imaging aids the decision on feasibility of fertility-sparing surgical options, such as extended cone biopsy or trachelectomy. High spatial resolution images with maximal contrast between tumour and surrounding background are achieved with T2-weighted and diffusion-weighted (DW) magnetic resonance imaging (MRI) obtained using an endovaginal receiver coil. Tumour size and volume demonstrated in this way correlates between observers and with histology and differences between MRI and histology estimates of normal endocervical canal length are not significant. For planning fertility-sparing surgery, this imaging technique facilitates the best oncological outcome while minimising subsequent obstetric risks. Parametrial invasion may be assessed on large field of view T2-weighted MRI. The fat content of the parametrium limits the utility of DW imaging in this context, because fat typically shows diffusion restriction. The use of contrast-enhanced MRI for assessing the parametrium does not provide additional benefits to the T2-weighted images and the need for an extrinsic contrast agent merely adds additional complexity and cost. For nodal assessment, 18fluorodeoxyglucose positron emission tomography-computerised tomography (18FDG PET-CT) remains the gold standard.

Supporting laparoscopic general surgery training with digital technology: The United Kingdom and Ireland paradigm

Surgical training in the UK and Ireland has faced challenges following the implementation of the European Working Time Directive and postgraduate training reform. The health services are undergoing a digital transformation; digital technology is remodelling the delivery of surgical care and surgical training. This review aims to critically evaluate key issues in laparoscopic general surgical training and the digital technology such as virtual and augmented reality, telementoring and automated workflow analysis and surgical skills assessment. We include pre-clinical, proof of concept research and commercial systems that are being developed to provide solutions. Digital surgical technology is evolving through interdisciplinary collaboration to provide widespread access to high-quality laparoscopic general surgery training and assessment. In the future this could lead to integrated, context-aware systems that support surgical teams in providing safer surgical care.

HoloPointer: a virtual augmented reality pointer for laparoscopic surgery training

PURPOSE

In laparoscopic surgery training, experts guide novice physicians to desired instrument positions or indicate relevant areas of interest. These instructions are usually given via verbal communication or using physical pointing devices. To facilitate a sterile work flow and to improve training, new guiding methods are needed. This work proposes to use optical see-through augmented reality to visualize an interactive virtual pointer on the laparoscopic.

METHODS

After an interdisciplinary development, the pointer's applicability and feasibility for training was evaluated and it was compared to a standard condition based on verbal and gestural communication only. In this study, ten surgical trainees were guided by an experienced trainer during cholecystectomies on a laparoscopic training simulator. All trainees completed a virtual cholecystectomy with and without the interactive virtual pointer in alternating order. Measures included procedure time, economy of movement and error rates.

RESULTS

Results of standardized variables revealed significantly improved economy of movement (p = 0.047) and error rates (p = 0.047), as well as an overall improved user performance (Total z-score; p = 0.031) in conditions using the proposed method.

CONCLUSION

The proposed HoloPointer is a feasible and applicable tool for laparoscopic surgery training. It improved objective performance metrics without prolongation of the task completion time in this pre-clinical setup.

Mixed-reality hologram for diagnosis and surgical planning of double outlet of the right ventricle: a pilot study

AIM

To evaluate the mixed-reality (MR) hologram, a novel technology based on two-dimensional images, which simulates three-dimensional (3D) images and provides a dynamic and interactive alternative, for its usefulness in the diagnosis and surgical planning of double outlet of the right ventricle (DORV).

MATERIALS AND METHODS

Thirty-four patients who were suspected of DORV based on ultrasound findings underwent cardiac computed tomography angiography (CTA). The patients were assigned randomly to the MR holographic guidance (MRHG) group or the control group. For the patients in the MRHG group, the CTA images were converted into Standard Template Library (STL) files after segmentation, 3D reconstruction, colourisation, and transparentisation, and then exported for MR holographic visualisation. The CTA images of the patients in the control group were analysed using routine 3D reconstruction only. Diagnostic accuracy and surgical planning were compared between the two groups based on visualisation at surgery.

RESULTS

In the MRHG group, the 3D hologram observation was in concordance with the actual anatomical findings, and the DORV type was classified accurately in all patients. The diagnostic accuracy for the malformation was 95.5% in the MRHG group and 89.7% in the control group, but the difference was not significant (p=0.3). All the procedures were exactly the same as planned based on the 3D MR holographic model. The surgical planning time was shorter for the MRHG group (51.65 ± 11.11 min) than that for the control group (65.71 ± 18.07 min, p<0.05).

CONCLUSION

MR 3D holograms may provide a clear and deeper anatomical perception of DORV and improve surgical planning.

Intraoperative 3D Hologram Support With Mixed Reality Techniques in Liver Surgery

OBJECTIVE

The aim of this study was to investigate the potential of an intraoperative 3D hologram, which was a computer graphics model liver, with mixed reality techniques in liver surgery.

SUMMARY BACKGROUND DATA

The merits for the application of a hologram for surgical support are: 1) no sterilized display monitor; 2) better spatial awareness; and 3) 3D images shared by all the surgeons.

METHODS

3D polygon data using preoperative computed tomography data was installed into head mount displays, HoloLens (Microsoft Corporation, Redmond, WA).

RESULTS

In a Wi-Fi-enabled operative room, several surgeons wearing HoloLens succeeded in sharing the same hologram and moving that hologram from respective operators' angles by means of easy gesture-handling without any monitors. The intraoperative hologram contributed to better imagination of tumor locations, and for determining the parenchymal dissection line in the hepatectomy for the patients with more than 20 multiple colo-rectal liver metastases. In another case, the hologram enabled a safe Gliisonean pedicle approach for hepato-cellular carcinoma with a hilar anatomical anomaly. Surgeons could easily compare the real patient's anatomy and that of the hologram just before the hepatic hilar procedure.

CONCLUSIONS

This initial experience suggested that an intraoperative hologram with mixed reality techniques contributed to "last-minute simulation," not for "navigation." The intraoperative hologram might be a new next-generation operation-supportive tool in terms of spatial awareness, sharing, and simplicity.

Augmented reality for image guidance in transoral robotic surgery

With the advent of precision surgery, there have been attempts to integrate imaging with robotic systems to guide sound oncologic surgical resections while preserving critical structures. In the confined space of transoral robotic surgery (TORS), this offers great potential given the proximity of structures. In this cadaveric experiment, we describe the use of a 3D virtual model displayed in the surgeon's console with the surgical field in view, to facilitate image-guided surgery at the oropharynx where there is significant soft tissue deformation. We also utilized the 3D model that was registered to the maxillary dentition, allowing for real-time image overlay of the internal carotid artery system. This allowed for real-time visualization of the internal carotid artery system that was qualitatively accurate on cadaveric dissection. Overall, this shows that virtual models and image overlays can be useful in image-guided surgery while approaching different sites in head and neck surgery with TORS.