Augmented Reality during Open Liver Surgery Using a Markerless Non-rigid Registration System

Augmented Reality Versus Freehand Spinopelvic Fixation in Spinal Deformity: A Case-Control Study

PURPOSE

This study sought to compare screw placement accuracy and outcomes between freehand (FH) and AR-guided pelvic fixation. While pelvic fixation is a critical technique in spinal deformity surgery, S2-alar iliac (S2AI) screw placement poses challenges.

METHODS

We conducted a case-control study of 50 consecutive patients who underwent spinopelvic fixation at a single institution. AR guidance was performed using a head-mounted display (Xvision, Augmedics). Patient demographics, surgical characteristics, spinopelvic parameters, and screw breach grade were compared using univariate and multivariate statistics.

RESULTS

Pelvic fixation was performed FH in 21 patients (median age, 64; female, 38.1%; median BMI 32.3 kg/m2) and AR-guided in 29 patients (median age, 66; female, 51.7%; median BMI 28.4 kg/m2). Mean follow-up was longer in the FH group (28 mos vs 11 mos, P < 0.001). Pelvic fixation in the FH group was performed using either S2AI (90.5%) or dual S2AI (9.5%) screws. There were no significant differences in length of surgery (FH, 439 minutes; AR, 490 minutes; P = 0.1) or estimated blood loss (FH, 2.1L; AR, 1.9L; P = 0.7). Accuracy of FH pelvic fixation was 95.6% (43/45 screws) and accuracy of AR pelvic fixation was 96.5% (55/57 screws). Multivariable logistic regression for screw breach revealed no significant association with AR guidance when controlling for age, BMI, osteoporosis, and smoking.

CONCLUSIONS

We present the first case-control study of AR-guided spinopelvic fixation, with findings suggesting parity between FH and AR-guidance, serving as foundation for prospective controlled studies with longitudinal follow-up to interrogate the benefits of AR-guidance in spinal deformity surgery.

A review on organ deformation modeling approaches for reliable surgical navigation using augmented reality

Augmented Reality (AR) holds the potential to revolutionize surgical procedures by allowing surgeons to visualize critical structures within the patient's body. This is achieved through superimposing preoperative organ models onto the actual anatomy. Challenges arise from dynamic deformations of organs during surgery, making preoperative models inadequate for faithfully representing intraoperative anatomy. To enable reliable navigation in augmented surgery, modeling of intraoperative deformation to obtain an accurate alignment of the preoperative organ model with the intraoperative anatomy is indispensable. Despite the existence of various methods proposed to model intraoperative organ deformation, there are still few literature reviews that systematically categorize and summarize these approaches. This review aims to fill this gap by providing a comprehensive and technical-oriented overview of modeling methods for intraoperative organ deformation in augmented reality in surgery. Through a systematic search and screening process, 112 closely relevant papers were included in this review. By presenting the current status of organ deformation modeling methods and their clinical applications, this review seeks to enhance the understanding of organ deformation modeling in AR-guided surgery, and discuss the potential topics for future advancements.

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

Neural fields for 3D tracking of anatomy and surgical instruments in monocular laparoscopic video clips

Laparoscopic video tracking primarily focuses on two target types: surgical instruments and anatomy. The former could be used for skill assessment, while the latter is necessary for the projection of virtual overlays. Where instrument and anatomy tracking have often been considered two separate problems, in this article, a method is proposed for joint tracking of all structures simultaneously. Based on a single 2D monocular video clip, a neural field is trained to represent a continuous spatiotemporal scene, used to create 3D tracks of all surfaces visible in at least one frame. Due to the small size of instruments, they generally cover a small part of the image only, resulting in decreased tracking accuracy. Therefore, enhanced class weighting is proposed to improve the instrument tracks. The authors evaluate tracking on video clips from laparoscopic cholecystectomies, where they find mean tracking accuracies of 92.4% for anatomical structures and 87.4% for instruments. Additionally, the quality of depth maps obtained from the method's scene reconstructions is assessed. It is shown that these pseudo-depths have comparable quality to a state-of-the-art pre-trained depth estimator. On laparoscopic videos in the SCARED dataset, the method predicts depth with an MAE of 2.9 mm and a relative error of 9.2%. These results show the feasibility of using neural fields for monocular 3D reconstruction of laparoscopic scenes. Code is available via GitHub: https://github.com/Beerend/Surgical-OmniMotion.

Navigating the Future of 3D Laparoscopic Liver Surgeries: Visualization of Internal Anatomy on Laparoscopic Images With Augmented Reality

INTRODUCTION

Liver tumor resection requires precise localization of tumors and blood vessels. Despite advancements in 3-dimensional (3D) visualization for laparoscopic surgeries, challenges persist. We developed and evaluated an augmented reality (AR) system that overlays preoperative 3D models onto laparoscopic images, offering crucial support for 3D visualization during laparoscopic liver surgeries.

METHODS

Anatomic liver structures from preoperative computed tomography scans were segmented using open-source software including 3D Slicer and Maya 2022 for 3D model editing. A registration system was created with 3D visualization software utilizing a stereo registration input system to overlay the virtual liver onto laparoscopic images during surgical procedures. A controller was customized using a modified keyboard to facilitate manual alignment of the virtual liver with the laparoscopic image. The AR system was evaluated by 3 experienced surgeons who performed manual registration for a total of 27 images from 7 clinical cases. The evaluation criteria included registration time; measured in minutes, and accuracy; measured using the Dice similarity coefficient.

RESULTS

The overall mean registration time was 2.4±1.7 minutes (range: 0.3 to 9.5 min), and the overall mean registration accuracy was 93.8%±4.9% (range: 80.9% to 99.7%).

CONCLUSION

Our validated AR system has the potential to effectively enable the prediction of internal hepatic anatomic structures during 3D laparoscopic liver resection, and may enhance 3D visualization for select laparoscopic liver surgeries.

Selective Clamping for Robot-Assisted Surgical Procedures

Partial nephrectomy, the gold standard treatment for renal tumors, is performed with clamping of the renal arteries, in order to interrupt the blood flowing towards the tumor. However, the temporary interruption of arterial flow may lead to ischemia of the renal parenchyma. Thus, the interruption of the flow should be as short as possible and the clamping should be localized to the arteries flowing towards the tumor only, by implementing the so-called selective clamping. In this paper, we propose a system to automatically provide to the surgeon the optimal clamping points, according to our methods, which allow to minimize the ischemia percentage, thus preserving the health of the remaining renal parenchyma. Moreover, we exploit the algorithm as a planner for a robotic system that, starting from the clamping points automatically computed, emulates the clamping procedure. The overall architecture is validated on different patient's anatomies and using a robotic setup.

Augmented Reality Guided Laparoscopic Liver Resection: A Phantom Study With Intraparenchymal Tumors

INTRODUCTION

Augmented reality (AR) in laparoscopic liver resection (LLR) can improve intrahepatic navigation by creating a virtual liver transparency. Our team has recently developed Hepataug, an AR software that projects the invisible intrahepatic tumors onto the laparoscopic images and allows the surgeon to localize them precisely. However, the accuracy of registration according to the location and size of the tumors, as well as the influence of the projection axis, have never been measured. The aim of this work was to measure the three-dimensional (3D) tumor prediction error of Hepataug.

METHODS

Eight 3D virtual livers were created from the computed tomography scan of a healthy human liver. Reference markers with known coordinates were virtually placed on the anterior surface. The virtual livers were then deformed and 3D printed, forming 3D liver phantoms. After placing each 3D phantom inside a pelvitrainer, registration allowed Hepataug to project virtual tumors along two axes: the laparoscope axis and the operator port axis. The surgeons had to point the center of eight virtual tumors per liver with a pointing tool whose coordinates were precisely calculated.

RESULTS

We obtained 128 pointing experiments. The average pointing error was 29.4 ± 17.1 mm and 9.2 ± 5.1 mm for the laparoscope and operator port axes respectively (P = 0.001). The pointing errors tended to increase with tumor depth (correlation coefficients greater than 0.5 with P < 0.001). There was no significant dependency of the pointing error on the tumor size for both projection axes.

CONCLUSIONS

Tumor visualization by projection toward the operating port improves the accuracy of AR guidance and partially solves the problem of the two-dimensional visual interface of monocular laparoscopy. Despite a lower precision of AR for tumors located in the posterior part of the liver, it could allow the surgeons to access these lesions without completely mobilizing the liver, hence decreasing the surgical trauma.

Intraoperative Angle Measurement of Anatomical Structures: A Systematic Review

Ensuring precise angle measurement during surgical correction of orientation-related deformities is crucial for optimal postoperative outcomes, yet there is a lack of an ideal commercial solution. Current measurement sensors and instrumentation have limitations that make their use context-specific, demanding a methodical evaluation of the field. A systematic review was carried out in March 2023. Studies reporting technologies and validation methods for intraoperative angular measurement of anatomical structures were analyzed. A total of 32 studies were included, 17 focused on image-based technologies (6 fluoroscopy, 4 camera-based tracking, and 7 CT-based), while 15 explored non-image-based technologies (6 manual instruments and 9 inertial sensor-based instruments). Image-based technologies offer better accuracy and 3D capabilities but pose challenges like additional equipment, increased radiation exposure, time, and cost. Non-image-based technologies are cost-effective but may be influenced by the surgeon's perception and require careful calibration. Nevertheless, the choice of the proper technology should take into consideration the influence of the expected error in the surgery, surgery type, and radiation dose limit. This comprehensive review serves as a valuable guide for surgeons seeking precise angle measurements intraoperatively. It not only explores the performance and application of existing technologies but also aids in the future development of innovative solutions.

Ultrasound guidance in navigated liver surgery: toward deep-learning enhanced compensation of deformation and organ motion

PURPOSE

Accuracy of image-guided liver surgery is challenged by deformation of the liver during the procedure. This study aims at improving navigation accuracy by using intraoperative deep learning segmentation and nonrigid registration of hepatic vasculature from ultrasound (US) images to compensate for changes in liver position and deformation.

METHODS

This was a single-center prospective study of patients with liver metastases from any origin. Electromagnetic tracking was used to follow US and liver movement. A preoperative 3D model of the liver, including liver lesions, and hepatic and portal vasculature, was registered with the intraoperative organ position. Hepatic vasculature was segmented using a reduced 3D U-Net and registered to preoperative imaging after initial alignment followed by nonrigid registration. Accuracy was assessed as Euclidean distance between the tumor center imaged in the intraoperative US and the registered preoperative image.

RESULTS

Median target registration error (TRE) after initial alignment was 11.6 mm in 25 procedures and improved to 6.9 mm after nonrigid registration (p = 0.0076). The number of TREs above 10 mm halved from 16 to 8 after nonrigid registration. In 9 cases, registration was performed twice after failure of the first attempt. The first registration cycle was completed in median 11 min (8:00-18:45 min) and a second in 5 min (2:30-10:20 min).

CONCLUSION

This novel registration workflow using automatic vascular detection and nonrigid registration allows to accurately localize liver lesions. Further automation in the workflow is required in initial alignment and classification accuracy.

State of the Art in Immersive Interactive Technologies for Surgery Simulation: A Review and Prospective

Immersive technologies have thrived on a strong foundation of software and hardware, injecting vitality into medical training. This surge has witnessed numerous endeavors incorporating immersive technologies into surgery simulation for surgical skills training, with a growing number of researchers delving into this domain. Relevant experiences and patterns need to be summarized urgently to enable researchers to establish a comprehensive understanding of this field, thus promoting its continuous growth. This study provides a forward-looking perspective by reviewing the latest development of immersive interactive technologies for surgery simulation. The investigation commences from a technological standpoint, delving into the core aspects of virtual reality (VR), augmented reality (AR) and mixed reality (MR) technologies, namely, haptic rendering and tracking. Subsequently, we summarize recent work based on the categorization of minimally invasive surgery (MIS) and open surgery simulations. Finally, the study showcases the impressive performance and expansive potential of immersive technologies in surgical simulation while also discussing the current limitations. We find that the design of interaction and the choice of immersive technology in virtual surgery development should be closely related to the corresponding interactive operations in the real surgical speciality. This alignment facilitates targeted technological adaptations in the direction of greater applicability and fidelity of simulation.

Registration Sanity Check for AR-guided Surgical Interventions: Experience From Head and Face Surgery

Achieving and maintaining proper image registration accuracy is an open challenge of image-guided surgery. This work explores and assesses the efficacy of a registration sanity check method for augmented reality-guided navigation (AR-RSC), based on the visual inspection of virtual 3D models of landmarks. We analyze the AR-RSC sensitivity and specificity by recruiting 36 subjects to assess the registration accuracy of a set of 114 AR images generated from camera images acquired during an AR-guided orthognathic intervention. Translational or rotational errors of known magnitude up to ±1.5 mm/±15.5°, were artificially added to the image set in order to simulate different registration errors. This study analyses the performance of AR-RSC when varying (1) the virtual models selected for misalignment evaluation (e. g., the model of brackets, incisor teeth, and gingival margins in our experiment), (2) the type (translation/rotation) of registration error, and (3) the level of user experience in using AR technologies. Results show that: 1) the sensitivity and specificity of the AR-RSC depends on the virtual models (globally, a median true positive rate of up to 79.2% was reached with brackets, and a median true negative rate of up to 64.3% with incisor teeth), 2) there are error components that are more difficult to identify visually, 3) the level of user experience does not affect the method. In conclusion, the proposed AR-RSC, tested also in the operating room, could represent an efficient method to monitor and optimize the registration accuracy during the intervention, but special attention should be paid to the selection of the AR data chosen for the visual inspection of the registration accuracy.

Soft tissue surgical robot for minimally invasive surgery: a review

Purpose

The current state of soft tissue surgery robots is surveyed, and the key technologies underlying their success are analyzed. State-of-the-art technologies are introduced, and future directions are discussed.

Methods

Relevant literature is explored, analyzed, and summarized.

Results

Soft tissue surgical robots had rapidly spread in the field of laparoscopic surgery based on the multi-degree-of-freedom movement of intra-abdominal surgical tools and stereoscopic imaging that are not possible in conventional surgery. The three key technologies that have made surgical robots successful are wire-driven mechanisms for multi-degree-of-freedom movement, master devices for intuitive remote control, and stereoscopic imaging technology. Recently, human-robot interaction technologies have been applied to develop user interfaces such as vision assistance and haptic feedback, and research on autonomous surgery has begun.

Conclusion

Robotic surgery not only replaces conventional laparoscopic surgery but also allows for complex surgeries that are not possible with laparoscopic surgery. On the other hand, it is also criticized for its high cost and lack of clinical superiority or patient benefit compared to conventional laparoscopic surgery. As various robots compete in the market, the cost of surgical robots is expected to decrease. Surgical robots are expected to continue to evolve in the future due to the need to reduce the workload of medical staff and improve the level of care demanded by patients.

Can 3D imaging modeling recognize functional tissue and predict liver failure? A retrospective study based on 3D modelling of the major hepatectomies after hepatic modulation

BACKGROUND

Thanks to the introduction of radiomics, 3d reconstruction can be able to analyse tissues and recognise true hypertrophy from non-functioning tissue in patients treated with major hepatectomies with hepatic modulation.The aim of this study is to evaluate the performance of 3D Imaging Modelling in predict liver failure.

METHODS

Patients submitted to major hepatectomies after hepatic modulation at Sanchinarro University Hospital from May 2015 to October 2019 were analysed. Three-dimensional reconstruction was realised before and after surgical treatment. The volumetry of Future Liver Remnant was calculated, distinguishing in Functional Future Liver Remnant (FRFx) i.e. true hypertrophy tissue and Anatomic Future Liver Remnant (FRL) i.e. hypertrophy plus no functional tissue (oedema/congestion) These volumes were analysed in patients with and without post hepatic liver failure.

RESULTS

Twenty-four procedures were realised (11 ALPPS and 13 PVE followed by major hepatectomy). Post hepatic liver failure grade B and C occurred in 6 patients. The ROC curve showed a better AUC for FRFxV (74%) with respect to FRLV (54%) in prediction PHLF > B. The increase of anatomical FRL (iFRL) was superior in the ALPPS group (120%) with respect to the PVE group (73%) (p = 0,041), while the increase of functional FRFX (iFRFx) was 35% in the ALLPS group and 46% in the PVE group (p > 0,05), showing no difference in the two groups.

CONCLUSION

The 3D reconstruction model can allow optimal surgical planning, and through the use of specific algorithms, can contribute to differential functioning liver parenchyma of the FLR.

Image-to-Patient Registration in Computer-Assisted Surgery of Head and Neck: State-of-the-Art, Perspectives, and Challenges

Today, image-guided systems play a significant role in improving the outcome of diagnostic and therapeutic interventions. They provide crucial anatomical information during the procedure to decrease the size and the extent of the approach, to reduce intraoperative complications, and to increase accuracy, repeatability, and safety. Image-to-patient registration is the first step in image-guided procedures. It establishes a correspondence between the patient's preoperative imaging and the intraoperative data. When it comes to the head-and-neck region, the presence of many sensitive structures such as the central nervous system or the neurosensory organs requires a millimetric precision. This review allows evaluating the characteristics and the performances of different registration methods in the head-and-neck region used in the operation room from the perspectives of accuracy, invasiveness, and processing times. Our work led to the conclusion that invasive marker-based methods are still considered as the gold standard of image-to-patient registration. The surface-based methods are recommended for faster procedures and applied on the surface tissues especially around the eyes. In the near future, computer vision technology is expected to enhance these systems by reducing human errors and cognitive load in the operating room.

Augmented reality in liver surgery

INTRODUCTION

During an operation, augmented reality (AR) enables surgeons to enrich their vision of the operating field by means of digital imagery, particularly as regards tumors and anatomical structures. While in some specialties, this type of technology is routinely ustilized, in liver surgery due to the complexity of modeling organ deformities in real time, its applications remain limited. At present, numerous teams are attempting to find a solution applicable to current practice, the objective being to overcome difficulties of intraoperative navigation in an opaque organ.

OBJECTIVE

To identify, itemize and analyze series reporting AR techniques tested in liver surgery, the objectives being to establish a state of the art and to provide indications of perspectives for the future.

METHODS

In compliance with the PRISMA guidelines and availing ourselves of the PubMed, Embase and Cochrane databases, we identified English-language articles published between January 2020 and January 2022 corresponding to the following keywords: augmented reality, hepatic surgery, liver and hepatectomy.

RESULTS

Initially, 102 titles, studies and summaries were preselected. Twenty-eight corresponding to the inclusion criteria were included, reporting on 183patients operated with the help of AR by laparotomy (n=31) or laparoscopy (n=152). Several techniques of acquisition and visualization were reported. Anatomical precision was the main assessment criterion in 19 articles, with values ranging from 3mm to 14mm, followed by time of acquisition and clinical feasibility.

CONCLUSION

While several AR technologies are presently being developed, due to insufficient anatomical precision their clinical applications have remained limited. That much said, numerous teams are currently working toward their optimization, and it is highly likely that in the short term, the application of AR in liver surgery will have become more frequent and effective. As for its clinical impact, notably in oncology, it remains to be assessed.

Interpretation and Use of Applied/Operational Machine Learning and Artificial Intelligence in Surgery

Applications for artificial intelligence (AI) and machine learning in surgery include image interpretation, data summarization, automated narrative construction, trajectory and risk prediction, and operative navigation and robotics. The pace of development has been exponential, and some AI applications are working well. However, demonstrations of clinical utility, validity, and equity have lagged algorithm development and limited widespread adoption of AI into clinical practice. Outdated computing infrastructure and regulatory challenges which promote data silos are key barriers. Multidisciplinary teams will be needed to address these challenges and to build AI systems that are relevant, equitable, and dynamic.

Enhanced Recognition Confidence of Millimeter-Sized Intrahepatic Targets by Real-Time Virtual Sonography

OBJECTIVES

Real-time virtual sonography (RVS) is an artificial-intelligence-assisted ultrasonographic navigation system that displays synchronized preoperative computed tomography (CT) images corresponding to real-time intraoperative ultrasonograms (IOUS). This study aimed to investigate whether RVS can enhance IOUS identification of small intrahepatic targets found in preoperative CT.

METHODS

Patients with small intrahepatic targets detected by preoperative thin-slice dynamic CT before liver resection were included. The targets included millimeter-sized liver tumors or a third-order or more distal portal branch and were marked on CT images using 3D simulation software. After laparotomy, the targets were searched using fundamental IOUS, and participating liver surgeons subjectively scored the target identifying confidence on a scale of 1-5 (5 points for detection with the highest confidence and one point for undetectable). Then, the search procedure was repeated using the RVS, and the scores were compared.

RESULTS

Totally, 55 patients with 117 small targets were investigated. The median target size was 6.0 mm, and the median registration time was 3.6 seconds. The target identification confidence score significantly increased from 2.78 to 4.52 points after using RVS. Seventeen targets (14.5%) were undetectable in fundamental IOUS, and 14 of them were identified by RVS. The detectability of small liver tumors (2-5 points of identification confidence) by IOUS was 81.1 and 96.7% by RVS.

CONCLUSION

RVS enhanced surgeons' confidence in identifying millimeter-sized intrahepatic targets found in preoperative CT.

Mixed reality navigation training system for liver surgery based on a high-definition human cross-sectional anatomy data set

OBJECTIVES

This study aims to use the three-dimensional (3D) mixed-reality model of liver, entailing complex intrahepatic systems and to deeply study the anatomical structures and to promote the training, diagnosis and treatment of liver diseases.

METHODS

Vascular perfusion human specimens were used for thin-layer frozen milling to obtain liver cross-sections. The 104-megapixel-high-definition cross sectional data set was established and registered to achieve structure identification and manual segmentation. The digital model was reconstructed and data was used to print a 3D hepatic model. The model was combined with HoloLens mixed reality technology to reflect the complex relationships of intrahepatic systems. We simulated 3D patient specific anatomy for identification and preoperative planning, conducted a questionnaire survey, and evaluated the results.

RESULTS

The 3D digital model and 1:1 transparent and colored model of liver established truly reflected intrahepatic vessels and their complex relationships. The reconstructed model imported into HoloLens could be accurately matched with the 3D model. Only 7.7% participants could identify accessory hepatic veins. The depth and spatial-relationship of intrahepatic structures were better understandable for 92%. The 100%, 84.6%, 69% and 84% believed the 3D models were useful in planning, safer surgical paths, reducing intraoperative complications and training of young surgeons respectively.

CONCLUSIONS

A detailed 3D model can be reconstructed using the higher quality cross-sectional anatomical data set. When combined with 3D printing and HoloLens technology, a novel hybrid-reality navigation-training system for liver surgery is created. Mixed Reality training is a worthy alternative to provide 3D information to clinicians and its possible application in surgery. This conclusion was obtained based on a questionnaire and evaluation. Surgeons with extensive experience in surgical operations perceived in the questionnaire that this technology might be useful in liver surgery, would help in precise preoperative planning, accurate intraoperative identification, and reduction of hepatic injury.

Basic Experiments Toward Mixed Reality Dynamic Navigation for Laparoscopic Surgery

Laparoscopic surgery is a minimally invasive procedure that is performed by viewing endoscopic camera images. However, the limited field of view of endoscopic cameras makes laparoscopic surgery difficult. To provide more visual information during laparoscopic surgeries, augmented reality (AR) surgical navigation systems have been developed to visualize the positional relationship between the surgical field and organs based on preoperative medical images of a patient. However, since earlier studies used preoperative medical images, the navigation became inaccurate as the surgery progressed because the organs were displaced and deformed during surgery. To solve this problem, we propose a mixed reality (MR) surgery navigation system in which surgical instruments are tracked by a motion capture (Mocap) system; we also evaluated the contact between the instruments and organs and simulated and visualized the deformation of the organ caused by the contact. This paper describes a method for the numerical calculation of the deformation of a soft body. Then, the basic technology of MR and projection mapping is presented for MR surgical navigation. The accuracy of the simulated and visualized deformations is evaluated through basic experiments using a soft rectangular cuboid object.

Leberchirurgie 4.0 - OP-Planung, Volumetrie, Navigation und Virtuelle Realität

Durch die Optimierung der konservativen Behandlung, die Verbesserung der bildgebenden Verfahren und die Weiterentwicklung der Operationstechniken haben sich das operative Spektrum sowie der Maßstab für die Resektabilität in Bezug auf die Leberchirurgie in den letzten Jahrzehnten deutlich verändert.

Dank zahlreicher technischer Entwicklungen, insbesondere der 3-dimensionalen Segmentierung, kann heutzutage die präoperative Planung und die Orientierung während der Operation selbst, vor allem bei komplexen Eingriffen, unter Berücksichtigung der patientenspezifischen Anatomie erleichtert werden.

Neue Technologien wie 3-D-Druck, virtuelle und augmentierte Realität bieten zusätzliche Darstellungsmöglichkeiten für die individuelle Anatomie. Verschiedene intraoperative Navigationsmöglichkeiten sollen die präoperative Planung im Operationssaal verfügbar machen, um so die Patientensicherheit zu erhöhen.

Dieser Übersichtsartikel soll einen Überblick über den gegenwärtigen Stand der verfügbaren Technologien sowie einen Ausblick in den Operationssaal der Zukunft geben.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Navigated liver surgery: State of the art and future perspectives

BACKGROUND

In recent years, the development of digital imaging technology has had a significant influence in liver surgery. The ability to obtain a 3-dimensional (3D) visualization of the liver anatomy has provided surgery with virtual reality of simulation 3D computer models, 3D printing models and more recently holograms and augmented reality (when virtual reality knowledge is superimposed onto reality). In addition, the utilization of real-time fluorescent imaging techniques based on indocyanine green (ICG) uptake allows clinicians to precisely delineate the liver anatomy and/or tumors within the parenchyma, applying the knowledge obtained preoperatively through digital imaging. The combination of both has transformed the abstract thinking until now based on 2D imaging into a 3D preoperative conception (virtual reality), enhanced with real-time visualization of the fluorescent liver structures, effectively facilitating intraoperative navigated liver surgery (augmented reality).

DATA SOURCES

A literature search was performed from inception until January 2021 in MEDLINE (PubMed), Embase, Cochrane library and database for systematic reviews (CDSR), Google Scholar, and National Institute for Health and Clinical Excellence (NICE) databases.

RESULTS

Fifty-one pertinent articles were retrieved and included. The different types of digital imaging technologies and the real-time navigated liver surgery were estimated and compared.

CONCLUSIONS

ICG fluorescent imaging techniques can contribute essentially to the real-time definition of liver segments; as a result, precise hepatic resection can be guided by the presence of fluorescence. Furthermore, 3D models can help essentially to further advancing of precision in hepatic surgery by permitting estimation of liver volume and functional liver remnant, delineation of resection lines along the liver segments and evaluation of tumor margins. In liver transplantation and especially in living donor liver transplantation (LDLT), 3D printed models of the donor's liver and models of the recipient's hilar anatomy can contribute further to improving the results. In particular, pediatric LDLT abdominal cavity models can help to manage the largest challenge of this procedure, namely large-for-size syndrome.

Ultrasound-based navigation for open liver surgery using active liver tracking

PURPOSE

Despite extensive preoperative imaging, intraoperative localization of liver lesions after systemic treatment can be challenging. Therefore, an image-guided navigation setup is explored that links preoperative diagnostic scans and 3D models to intraoperative ultrasound (US), enabling overlay of detailed diagnostic images on intraoperative US. Aim of this study is to assess the workflow and accuracy of such a navigation system which compensates for liver motion.

METHODS

Electromagnetic (EM) tracking was used for organ tracking and movement of the transducer. After laparotomy, a sensor was attached to the liver surface while the EM-tracked US transducer enabled image acquisition and landmark digitization. Landmarks surrounding the lesion were selected during patient-specific preoperative 3D planning and identified for registration during surgery. Endpoints were accuracy and additional times of the investigative steps. Accuracy was computed at the center of the target lesion.

RESULTS

In total, 22 navigated procedures were performed. Navigation provided useful visualization of preoperative 3D models and their overlay on US imaging. Landmark-based registration resulted in a mean fiducial registration error of 10.3 ± 4.3 mm, and a mean target registration error of 8.5 ± 4.2 mm. Navigation was available after an average of 12.7 min.

CONCLUSION

We developed a navigation method combining ultrasound with active liver tracking for organ motion compensation, with an accuracy below 10 mm. Fixation of the liver sensor near the target lesion compensates for local movement and contributes to improved reliability during navigation. This represents an important step forward in providing surgical navigation throughout the procedure.

TRIAL REGISTRATION

This study is registered in the Netherlands Trial Register (number NL7951).

Essential updates 2020/2021: Current topics of simulation and navigation in hepatectomy

With the development of three-dimensional (3D) simulation software, preoperative simulation technology is almost completely established. The remaining issue is how to recognize anatomy three-dimensionally. Extended reality is a newly developed technology with several merits for surgical application: no requirement for a sterilized display monitor, better spatial awareness, and the ability to share 3D images among all surgeons. Various technology or devices for intraoperative navigation have also been developed to support the safety and certainty of liver surgery. Consensus recommendations regarding indocyanine green fluorescence were determined in 2021. Extended reality has also been applied to intraoperative navigation, and artificial intelligence (AI) is one of the topics of real-time navigation. AI might overcome the problem of liver deformity with automatic registration. Including the issues described above, this article focuses on recent advances in simulation and navigation in liver surgery from 2020 to 2021.

Architecture of a Hybrid Video/Optical See-through Head-Mounted Display-Based Augmented Reality Surgical Navigation Platform

In the context of image-guided surgery, augmented reality (AR) represents a ground-breaking enticing improvement, mostly when paired with wearability in the case of open surgery. Commercially available AR head-mounted displays (HMDs), designed for general purposes, are increasingly used outside their indications to develop surgical guidance applications with the ambition to demonstrate the potential of AR in surgery. The applications proposed in the literature underline the hunger for AR-guidance in the surgical room together with the limitations that hinder commercial HMDs from being the answer to such a need. The medical domain demands specifically developed devices that address, together with ergonomics, the achievement of surgical accuracy objectives and compliance with medical device regulations. In the framework of an EU Horizon2020 project, a hybrid video and optical see-through augmented reality headset paired with a software architecture, both specifically designed to be seamlessly integrated into the surgical workflow, has been developed. In this paper, the overall architecture of the system is described. The developed AR HMD surgical navigation platform was positively tested on seven patients to aid the surgeon while performing Le Fort 1 osteotomy in cranio-maxillofacial surgery, demonstrating the value of the hybrid approach and the safety and usability of the navigation platform.

New Approach to Accelerated Image Annotation by Leveraging Virtual Reality and Cloud Computing

Three-dimensional imaging is at the core of medical imaging and is becoming a standard in biological research. As a result, there is an increasing need to visualize, analyze and interact with data in a natural three-dimensional context. By combining stereoscopy and motion tracking, commercial virtual reality (VR) headsets provide a solution to this critical visualization challenge by allowing users to view volumetric image stacks in a highly intuitive fashion. While optimizing the visualization and interaction process in VR remains an active topic, one of the most pressing issue is how to utilize VR for annotation and analysis of data. Annotating data is often a required step for training machine learning algorithms. For example, enhancing the ability to annotate complex three-dimensional data in biological research as newly acquired data may come in limited quantities. Similarly, medical data annotation is often time-consuming and requires expert knowledge to identify structures of interest correctly. Moreover, simultaneous data analysis and visualization in VR is computationally demanding. Here, we introduce a new procedure to visualize, interact, annotate and analyze data by combining VR with cloud computing. VR is leveraged to provide natural interactions with volumetric representations of experimental imaging data. In parallel, cloud computing performs costly computations to accelerate the data annotation with minimal input required from the user. We demonstrate multiple proof-of-concept applications of our approach on volumetric fluorescent microscopy images of mouse neurons and tumor or organ annotations in medical images.

A Percutaneous Portal Vein Puncture Under Artificial Ascites for Intraoperative Hepatic Segmentation Using Indocyanine Green Fluorescence: A Technical Report of Laparoscopic Anatomic Liver Resection

BACKGROUND

Laparoscopic liver resection have developed and is widely spread as standard procedure in these days, however, laparoscopic anatomic liver resection is still challenging, especially for posterosuperior lesions because of difficulties in segmental mapping and surgical techniques. Recently, the positive staining and negative staining method using fluorescent imaging techniques have been reported from experienced Asian centers, allowing to identify the tumor-bearing portal territory to be resected including the posterosuperior segment in laparoscopy. Those techniques are applicable in some cases; hence, it remains the room for improvement to establish as a feasible approach. Herein, we describe a percutaneous tumor-bearing portal vein puncture method under artificial ascites after the pneumoperitoneum for laparoscopic segmentectomy for segment 8.

CASE PRESENTATION AND SURGICAL PROCEDURE

A male patient in his 60s was admitted for an incidentally diagnosed hepatic mass in segment 8. Findings of the computed tomography scan showed a 2.5-cm-sized hepatocellular carcinoma lesion. Then, laparoscopic anatomic liver resection for segment 8 was planned. The segmentation of the segment 8 was performed through a percutaneous tumor-bearing portal vein puncture using indocyanine green injection with extracorporeal ultrasound guidance under artificial ascites. According to indocyanine green fluorescence navigation, anatomic liver resection was completed. Operative time was recorded as 375 minutes. The estimated intraoperative blood loss was 50 mL without the requirement for an intraoperative transfusion. The planned resections were successful with histologically negative surgical margins. The patient was discharged on the 19th postoperative day with normal liver function test results. There was no operation-related complication during hospitalization.

CONCLUSION

The intraoperative percutaneous portal vein puncture method under artificial ascites was useful for the identification of posterosuperior segment in laparoscopic anatomic segmentectomy.

Augmented reality in laparoscopic liver resection evaluated on an ex-vivo animal model with pseudo-tumours

BACKGROUND

The aim of this study was to assess the performance of our augmented reality (AR) software (Hepataug) during laparoscopic resection of liver tumours and compare it to standard ultrasonography (US).

MATERIALS AND METHODS

Ninety pseudo-tumours ranging from 10 to 20 mm were created in sheep cadaveric livers by injection of alginate. CT-scans were then performed and 3D models reconstructed using a medical image segmentation software (MITK). The livers were placed in a pelvi-trainer on an inclined plane, approximately perpendicular to the laparoscope. The aim was to obtain free resection margins, as close as possible to 1 cm. Laparoscopic resection was performed using US alone (n = 30, US group), AR alone (n = 30, AR group) and both US and AR (n = 30, ARUS group). R0 resection, maximal margins, minimal margins and mean margins were assessed after histopathologic examination, adjusted to the tumour depth and to a liver zone-wise difficulty level.

RESULTS

The minimal margins were not different between the three groups (8.8, 8.0 and 6.9 mm in the US, AR and ARUS groups, respectively). The maximal margins were larger in the US group compared to the AR and ARUS groups after adjustment on depth and zone difficulty (21 vs. 18 mm, p = 0.001 and 21 vs. 19.5 mm, p = 0.037, respectively). The mean margins, which reflect the variability of the measurements, were larger in the US group than in the ARUS group after adjustment on depth and zone difficulty (15.2 vs. 12.8 mm, p < 0.001). When considering only the most difficult zone (difficulty 3), there were more R1/R2 resections in the US group than in the AR + ARUS group (50% vs. 21%, p = 0.019).

CONCLUSION

Laparoscopic liver resection using AR seems to provide more accurate resection margins with less variability than the gold standard US navigation, particularly in difficult to access liver zones with deep tumours.

Joint scene and object tracking for cost-Effective augmented reality guided patient positioning in radiation therapy

BACKGROUND AND OBJECTIVE

The research is done in the field of Augmented Reality (AR) for patient positioning in radiation therapy is scarce. We propose an efficient and cost-effective algorithm for tracking the scene and the patient to interactively assist the patient's positioning process by providing visual feedback to the operator. Up to our knowledge, this is the first framework that can be employed for mobile interactive AR to guide patient positioning.

METHODS

We propose a pointcloud processing method that, combined with a fiducial marker-mapper algorithm and the generalized ICP algorithm, tracks the patient and the camera precisely and efficiently only using the CPU unit. The 3D reference model and body marker map alignment is calculated employing an efficient body reconstruction algorithm.

RESULTS

Our quantitative evaluation shows that the proposed method achieves a translational and rotational error of 4.17 mm/0.82^∘ at 9 fps. Furthermore, the qualitative results demonstrate the usefulness of our algorithm in patient positioning on different human subjects.

CONCLUSION

Since our algorithm achieves a relatively high frame rate and accuracy employing a regular laptop (without a dedicated GPU), it is a very cost-effective AR-based patient positioning method. It also opens the way for other researchers by introducing a framework that could be improved upon for better mobile interactive AR patient positioning solutions in the future.

Clinical use of augmented reality, mixed reality, three-dimensional-navigation and artificial intelligence in liver surgery

A precise knowledge of intra-parenchymal vascular and biliary architecture and the location of lesions in relation to the complex anatomy is indispensable to perform liver surgery. Therefore, virtual three-dimensional (3D)-reconstruction models from computed tomography/magnetic resonance imaging scans of the liver might be helpful for visualization. Augmented reality, mixed reality and 3D-navigation could transfer such 3D-image data directly into the operation theater to support the surgeon. This review examines the literature about the clinical and intraoperative use of these image guidance techniques in liver surgery and provides the reader with the opportunity to learn about these techniques. Augmented reality and mixed reality have been shown to be feasible for the use in open and minimally invasive liver surgery. 3D-navigation facilitated targeting of intraparenchymal lesions. The existing data is limited to small cohorts and description about technical details e.g., accordance between the virtual 3D-model and the real liver anatomy. Randomized controlled trials regarding clinical data or oncological outcome are not available. Up to now there is no intraoperative application of artificial intelligence in liver surgery. The usability of all these sophisticated image guidance tools has still not reached the grade of immersion which would be necessary for a widespread use in the daily surgical routine. Although there are many challenges, augmented reality, mixed reality, 3D-navigation and artificial intelligence are emerging fields in hepato-biliary surgery.

Development and Accuracy Evaluation of Augmented Reality-based Patient Positioning System in Radiotherapy: A Phantom Study

BACKGROUND/AIM

To develop and evaluate the accuracy of augmented reality (AR)-based patient positioning systems in radiotherapy.

MATERIALS AND METHODS

AR head-mounted displays (AR-HMDs), which virtually superimpose a three-dimensional (3D) image generated by the digital imaging and communications in medicine (DICOM) data, have been developed. The AR-based positioning feasibility was evaluated. Then, the setup errors of three translational axes directions and rotation angles between the AR and the conventional laser-based positioning were compared.

RESULTS

The AR-based pelvic phantom positioning was feasible. The setup errors of AR-based positioning were comparable to laser-based positioning in all translational axis directions and rotation angles. The time necessary for AR-based positioning was significantly longer than that for laser-based positioning (171.0 s vs. 47.5 s, p<0.001).

CONCLUSION

AR-based positioning for radiotherapy was feasible, and showed comparable positioning errors to those of conventional line-based positioning; however, a markedly longer setup time was necessary.

A review on the applications of virtual reality, augmented reality and mixed reality in surgical simulation: an extension to different kinds of surgery

Background: Research proves that the apprenticeship model, which is the gold standard for training surgical residents, is obsolete. For that reason, there is a continuing effort toward the development of high-fidelity surgical simulators to replace the apprenticeship model. Applying Virtual Reality Augmented Reality (AR) and Mixed Reality (MR) in surgical simulators increases the fidelity, level of immersion and overall experience of these simulators.Areas covered: The objective of this review is to provide a comprehensive overview of the application of VR, AR and MR for distinct surgical disciplines, including maxillofacial surgery and neurosurgery. The current developments in these areas, as well as potential future directions, are discussed.Expert opinion: The key components for incorporating VR into surgical simulators are visual and haptic rendering. These components ensure that the user is completely immersed in the virtual environment and can interact in the same way as in the physical world. The key components for the application of AR and MR into surgical simulators include the tracking system as well as the visual rendering. The advantages of these surgical simulators are the ability to perform user evaluations and increase the training frequency of surgical residents.

Demarcation Line Assessment in Anatomical Liver Resection: An Overview

Anatomical liver resection (ALR) is the preferred oncological approach for the treatment of primary liver malignancies, such as hepatocellular carcinoma and intrahepatic cholangiocarcinoma. The demarcation line (DL) is formed by means of selective vascular occlusion and is used by surgeons to guide ALR. Emerging intraoperative technologies are playing a major role to enhance the surgeon’s vision and ensure a precise oncologic surgery. In this article, a brief overview of modalities to assess the DL during ALRs is presented, from the established conventional techniques to future perspectives.