Live augmented reality: a new visualization method for laparoscopic surgery using continuous volumetric computed tomography

Beyond Traditional Training: Exploring the Benefits of Virtual Reality Simulator in Lumbar Pedicle Screw Insertion - A Randomized Controlled Trial

INTRODUCTION

This study compares the effectiveness of virtual reality simulators (VRS) and a saw bone model for learning lumbar pedicle screw insertion (LPSI) in neurosurgery.

METHODS

A single-center, cross-sectional, randomized controlled laboratory investigation was conducted involving residents and fellows from a tertiary care referral hospital. Participants were divided into two groups (A and B). Group A performed 3 LPSI tasks: the first on a saw bone model, the second on VRS, and the third on another saw bone model. Group B completed 2 LPSI tasks: the first on a saw bone model and the second on another saw bone model. The accuracy of LPSI was evaluated through noncontrast computed tomography scans for the saw bone models, while the in-built application of VRS was utilized to check for accuracy of screw placement using the simulator.

RESULTS

The study included 38 participants (19 in each group). Group A participants showed reduced mean entry point error (0.11 mm, P 0.024), increased mean purchase length (4.66 cm, P 0.007), and no cortical breaches (P 0.031) when placing the second saw bone model screw. Similar improvements were observed among group A participants in PGY 1-3 while placing the second saw bone model screws.

CONCLUSIONS

Virtual reality simulators (VRS) prove to be an invaluable tool for teaching complex neurosurgical skills, such as LPSI, to trainees. This technology investment can enhance the learning curve while maintaining patient safety.

Smart Operating Room in Digestive Surgery: A Narrative Review

The introduction of new technologies in current digestive surgical practice is progressively reshaping the operating room, defining the fourth surgical revolution. The implementation of black boxes and control towers aims at streamlining workflow and reducing surgical error by early identification and analysis, while augmented reality and artificial intelligence augment surgeons' perceptual and technical skills by superimposing three-dimensional models to real-time surgical images. Moreover, the operating room architecture is transitioning toward an integrated digital environment to improve efficiency and, ultimately, patients' outcomes. This narrative review describes the most recent evidence regarding the role of these technologies in transforming the current digestive surgical practice, underlining their potential benefits and drawbacks in terms of efficiency and patients' outcomes, as an attempt to foresee the digestive surgical practice of tomorrow.

Hybrid operating room applications for precision hepatobiliary surgery: A narrative review

This review summarizes the key applications of a hybrid operating room (HOR) in hepatobiliary surgery and explores the advantages, limitations, and future directions of its utilization. A comprehensive literature search was conducted in PubMed to identify articles reporting on the utilization of HORs in liver surgery. So far, the HOR has been limitedly applied in hepatobiliary surgery. It can offer an optimal environment for combining radiological and surgical interventions and for performing image-guided surgical navigation.

A 3D Image Registration Method for Laparoscopic Liver Surgery Navigation

At present, laparoscopic augmented reality (AR) navigation has been applied to minimally invasive abdominal surgery, which can help doctors to see the location of blood vessels and tumors in organs, so as to perform precise surgery operations. Image registration is the process of optimally mapping one or more images to the target image, and it is also the core of laparoscopic AR navigation. The key is how to shorten the registration time and optimize the registration accuracy. We have studied the three-dimensional (3D) image registration technology in laparoscopic liver surgery navigation and proposed a new registration method combining rough registration and fine registration. First, the adaptive fireworks algorithm (AFWA) is applied to rough registration, and then the optimized iterative closest point (ICP) algorithm is applied to fine registration. We proposed a method that is validated by the computed tomography (CT) dataset 3D-IRCADb-01. Experimental results show that our method is superior to other registration methods based on stochastic optimization algorithms in terms of registration time and accuracy.

How molecular imaging will enable robotic precision surgery : The role of artificial intelligence, augmented reality, and navigation

Molecular imaging is one of the pillars of precision surgery. Its applications range from early diagnostics to therapy planning, execution, and the accurate assessment of outcomes. In particular, molecular imaging solutions are in high demand in minimally invasive surgical strategies, such as the substantially increasing field of robotic surgery. This review aims at connecting the molecular imaging and nuclear medicine community to the rapidly expanding armory of surgical medical devices. Such devices entail technologies ranging from artificial intelligence and computer-aided visualization technologies (software) to innovative molecular imaging modalities and surgical navigation (hardware). We discuss technologies based on their role at different steps of the surgical workflow, i.e., from surgical decision and planning, over to target localization and excision guidance, all the way to (back table) surgical verification. This provides a glimpse of how innovations from the technology fields can realize an exciting future for the molecular imaging and surgery communities.

Augmented and virtual reality in spine surgery, current applications and future potentials

BACKGROUND CONTEXT

The field of artificial intelligence (AI) is rapidly advancing, especially with recent improvements in deep learning (DL) techniques. Augmented (AR) and virtual reality (VR) are finding their place in healthcare, and spine surgery is no exception. The unique capabilities and advantages of AR and VR devices include their low cost, flexible integration with other technologies, user-friendly features and their application in navigation systems, which makes them beneficial across different aspects of spine surgery. Despite the use of AR for pedicle screw placement, targeted cervical foraminotomy, bone biopsy, osteotomy planning, and percutaneous intervention, the current applications of AR and VR in spine surgery remain limited.

PURPOSE

The primary goal of this study was to provide the spine surgeons and clinical researchers with the general information about the current applications, future potentials, and accessibility of AR and VR systems in spine surgery.

STUDY DESIGN/SETTING

We reviewed titles of more than 250 journal papers from google scholar and PubMed with search words: augmented reality, virtual reality, spine surgery, and orthopaedic, out of which 89 related papers were selected for abstract review. Finally, full text of 67 papers were analyzed and reviewed.

METHODS

The papers were divided into four groups: technological papers, applications in surgery, applications in spine education and training, and general application in orthopaedic. A team of two reviewers performed paper reviews and a thorough web search to ensure the most updated state of the art in each of four group is captured in the review.

RESULTS

In this review we discuss the current state of the art in AR and VR hardware, their preoperative applications and surgical applications in spine surgery. Finally, we discuss the future potentials of AR and VR and their integration with AI, robotic surgery, gaming, and wearables.

CONCLUSIONS

AR and VR are promising technologies that will soon become part of standard of care in spine surgery.

Performance of image guided navigation in laparoscopic liver surgery - A systematic review

BACKGROUND

Compared to open surgery, minimally invasive liver resection has improved short term outcomes. It is however technically more challenging. Navigated image guidance systems (IGS) are being developed to overcome these challenges. The aim of this systematic review is to provide an overview of their current capabilities and limitations.

METHODS

Medline, Embase and Cochrane databases were searched using free text terms and corresponding controlled vocabulary. Titles and abstracts of retrieved articles were screened for inclusion criteria. Due to the heterogeneity of the retrieved data it was not possible to conduct a meta-analysis. Therefore results are presented in tabulated and narrative format.

RESULTS

Out of 2015 articles, 17 pre-clinical and 33 clinical papers met inclusion criteria. Data from 24 articles that reported on accuracy indicates that in recent years navigation accuracy has been in the range of 8-15 mm. Due to discrepancies in evaluation methods it is difficult to compare accuracy metrics between different systems. Surgeon feedback suggests that current state of the art IGS may be useful as a supplementary navigation tool, especially in small liver lesions that are difficult to locate. They are however not able to reliably localise all relevant anatomical structures. Only one article investigated IGS impact on clinical outcomes.

CONCLUSIONS

Further improvements in navigation accuracy are needed to enable reliable visualisation of tumour margins with the precision required for oncological resections. To enhance comparability between different IGS it is crucial to find a consensus on the assessment of navigation accuracy as a minimum reporting standard.

Image-guided minimally invasive endopancreatic surgery using a computer-assisted navigation system

BACKGROUND

Minimally invasive endopancreatic surgery (EPS), performing a pancreatic resection from inside the pancreatic duct, has been proposed as an experimental alternative to duodenum-preserving pancreatic head resection in benign diseases such as chronic pancreatitis, but is complicated by difficult spatial orientation when trying to reach structures of interest. This study assessed the feasibility and potential benefits of image-guided EPS using a computer-assisted navigation system in artificial pancreas silicon model.

METHODS

A surgical navigation system displayed a 3D reconstruction of the original computed tomography (CT) scan and the endoscope in relation to a selected target structure. In a first step, different surface landmark (LM)-based and intraparenchymal LM-based approaches for image-to-physical space registration were evaluated. The accuracy of registration was measured as fiducial registration error (FRE). Subsequently, intrapancreatic lesions (n = 8) that were visible on preoperative imaging, but not on the endoscopic view, were targeted with a computer-assisted, image-guided endopancreatic resection technique in pancreas silicon models. After each experiment, a CT scan was obtained for measurement of the shortest distance from the resection cavity to the centre of the lesion.

RESULTS

Intraparenchymal LM registration [FRE 2.24 mm (1.40-2.85)] was more accurate than surface LM registration [FRE 3.46 mm (2.25-4.85); p = 0.035], but not more accurate than combined registration of intraparenchymal and surface LM [FRE 2.46 mm (1.60-3.35); p = 0.052]. Using image-guided EPS, six of seven lesions were successfully targeted. The median distance from the resection cavity to the centre of the lesion on CT was 1.52 mm (0-2.4). In one pancreas, a lesion could not be resected due to the fragility of the pancreas model.

CONCLUSION

Image-guided minimally invasive EPS using a computer-assisted navigation system enabled successful targeting of pancreatic lesions that were invisible on the endoscopic image, but detectable on preoperative imaging. In the clinical setting, this tool could facilitate complex minimally invasive and robotic pancreatic procedures.

Virtual and Augmented Reality in Oncologic Liver Surgery

Virtual reality (VR) and augmented reality (AR) in complex surgery are evolving technologies enabling improved preoperative planning and intraoperative navigation. The basis of these technologies is a computer-based generation of a patient-specific 3-dimensional model from Digital Imaging and Communications in Medicine (DICOM) data. This article provides a state-of-the- art overview on the clinical use of this technology with a specific focus on hepatic surgery. Although VR and AR are still in an evolving stage with only some clinical application today, these technologies have the potential to become a key factor in improving preoperative and intraoperative decision making.

A Novel Augmented Reality-Based Navigation System in Perforator Flap Transplantation - A Feasibility Study

BACKGROUND

In perforator flap transplantation, dissection of the perforator is an important but difficult procedure because of the high variability in vascular anatomy. Preoperative imaging techniques could provide substantial information about vascular anatomy; however, it cannot provide direct guidance for surgeons during the operation. In this study, a navigation system (NS) was established to overlie a vascular map on surgical sites to further provide a direct guide for perforator flap transplantation.

METHODS

The NS was established based on computed tomographic angiography and augmented reality techniques. A virtual vascular map was reconstructed according to computed tomographic angiography data and projected onto real patient images using ARToolKit software. Additionally, a screw-fixation marker holder was created to facilitate registration. With the use of a tracking and display system, we conducted the NS on an animal model and measured the system error on a rapid prototyping model.

RESULTS

The NS assistance allowed for correct identification, as well as a safe and precise dissection of the perforator. The mean value of the system error was determined to be 3.474 ± 1.546 mm.

CONCLUSIONS

Augmented reality-based NS can provide precise navigation information by directly displaying a 3-dimensional individual anatomical virtual model onto the operative field in real time. It will allow rapid identification and safe dissection of a perforator in free flap transplantation surgery.

Mobile, real-time, and point-of-care augmented reality is robust, accurate, and feasible: a prospective pilot study

BACKGROUND

Augmented reality (AR) systems are currently being explored by a broad spectrum of industries, mainly for improving point-of-care access to data and images. Especially in surgery and especially for timely decisions in emergency cases, a fast and comprehensive access to images at the patient bedside is mandatory. Currently, imaging data are accessed at a distance from the patient both in time and space, i.e., at a specific workstation. Mobile technology and 3-dimensional (3D) visualization of radiological imaging data promise to overcome these restrictions by making bedside AR feasible.

METHODS

In this project, AR was realized in a surgical setting by fusing a 3D-representation of structures of interest with live camera images on a tablet computer using marker-based registration. The intent of this study was to focus on a thorough evaluation of AR. Feasibility, robustness, and accuracy were thus evaluated consecutively in a phantom model and a porcine model. Additionally feasibility was evaluated in one male volunteer.

RESULTS

In the phantom model (n = 10), AR visualization was feasible in 84% of the visualization space with high accuracy (mean reprojection error ± standard deviation (SD): 2.8 ± 2.7 mm; 95th percentile = 6.7 mm). In a porcine model (n = 5), AR visualization was feasible in 79% with high accuracy (mean reprojection error ± SD: 3.5 ± 3.0 mm; 95th percentile = 9.5 mm). Furthermore, AR was successfully used and proved feasible within a male volunteer.

CONCLUSIONS

Mobile, real-time, and point-of-care AR for clinical purposes proved feasible, robust, and accurate in the phantom, animal, and single-trial human model shown in this study. Consequently, AR following similar implementation proved robust and accurate enough to be evaluated in clinical trials assessing accuracy, robustness in clinical reality, as well as integration into the clinical workflow. If these further studies prove successful, AR might revolutionize data access at patient bedside.

Augmented reality technology for preoperative planning and intraoperative navigation during hepatobiliary surgery: A review of current methods

BACKGROUND

Augmented reality (AR) technology is used to reconstruct three-dimensional (3D) images of hepatic and biliary structures from computed tomography and magnetic resonance imaging data, and to superimpose the virtual images onto a view of the surgical field. In liver surgery, these superimposed virtual images help the surgeon to visualize intrahepatic structures and therefore, to operate precisely and to improve clinical outcomes.

DATA SOURCES

The keywords "augmented reality", "liver", "laparoscopic" and "hepatectomy" were used for searching publications in the PubMed database. The primary source of literatures was from peer-reviewed journals up to December 2016. Additional articles were identified by manual search of references found in the key articles.

RESULTS

In general, AR technology mainly includes 3D reconstruction, display, registration as well as tracking techniques and has recently been adopted gradually for liver surgeries including laparoscopy and laparotomy with video-based AR assisted laparoscopic resection as the main technical application. By applying AR technology, blood vessels and tumor structures in the liver can be displayed during surgery, which permits precise navigation during complex surgical procedures. Liver transformation and registration errors during surgery were the main factors that limit the application of AR technology.

CONCLUSIONS

With recent advances, AR technologies have the potential to improve hepatobiliary surgical procedures. However, additional clinical studies will be required to evaluate AR as a tool for reducing postoperative morbidity and mortality and for the improvement of long-term clinical outcomes. Future research is needed in the fusion of multiple imaging modalities, improving biomechanical liver modeling, and enhancing image data processing and tracking technologies to increase the accuracy of current AR methods.

Fast elastic registration of soft tissues under large deformations

A fast and accurate fusion of intra-operative images with a pre-operative data is a key component of computer-aided interventions which aim at improving the outcomes of the intervention while reducing the patient's discomfort. In this paper, we focus on the problematic of the intra-operative navigation during abdominal surgery, which requires an accurate registration of tissues undergoing large deformations. Such a scenario occurs in the case of partial hepatectomy: to facilitate the access to the pathology, e.g. a tumor located in the posterior part of the right lobe, the surgery is performed on a patient in lateral position. Due to the change in patient's position, the resection plan based on the pre-operative CT scan acquired in the supine position must be updated to account for the deformations. We suppose that an imaging modality, such as the cone-beam CT, provides the information about the intra-operative shape of an organ, however, due to the reduced radiation dose and contrast, the actual locations of the internal structures necessary to update the planning are not available. To this end, we propose a method allowing for fast registration of the pre-operative data represented by a detailed 3D model of the liver and its internal structure and the actual configuration given by the organ surface extracted from the intra-operative image. The algorithm behind the method combines the iterative closest point technique with a biomechanical model based on a co-rotational formulation of linear elasticity which accounts for large deformations of the tissue. The performance, robustness and accuracy of the method is quantitatively assessed on a control semi-synthetic dataset with known ground truth and a real dataset composed of nine pairs of abdominal CT scans acquired in supine and flank positions. It is shown that the proposed surface-matching method is capable of reducing the target registration error evaluated of the internal structures of the organ from more than 40 mm to less then 10 mm. Moreover, the control data is used to demonstrate the compatibility of the method with intra-operative clinical scenario, while the real datasets are utilized to study the impact of parametrization on the accuracy of the method. The method is also compared to a state-of-the art intensity-based registration technique in terms of accuracy and performance.

Augmented reality navigation in open surgery for hilar cholangiocarcinoma resection with hemihepatectomy using video-based in situ three-dimensional anatomical modeling: A case report

RATIONALE

Patients who undergo hilar cholangiocarcinoma (HCAC) resection with concomitant hepatectomy have a high risk of postoperative morbidity and mortality due to surgical trauma to the hepatic and biliary vasculature.

PATIENT CONCERNS

A 58-year-old Chinese man with yellowing skin and sclera, abdominal distension, pruritus, and anorexia for approximately 3 weeks.

DIAGNOSES

Magnetic resonance cholangiopancreatography and enhanced computed tomography (CT) scanning revealed a mass over the biliary tree at the porta hepatis, which diagnosed to be s a hilar cholangiocarcinoma.

INTERVENTION

Three-dimensional (3D) images of the patient's hepatic and biliary structures were reconstructed preoperatively from CT data, and the 3D images were used for preoperative planning and augmented reality (AR)-assisted intraoperative navigation during open HCAC resection with hemihepatectomy. A 3D-printed model of the patient's biliary structures was also used intraoperatively as a visual reference.

OUTCOMES

No serious postoperative complications occurred, and the patient was tumor-free at the 9-month follow-up examination based on CT results.

LESSONS

AR-assisted preoperative planning and intraoperative navigation might be beneficial in other patients with HCAC patients to reduce postoperative complications and ensure disease-free survival. In our postoperative analysis, we also found that, when the3D images were superimposed 3D-printed model using a see-through integral video graphy display device, our senses of depth perception and motion parallax were improved, compared with that which we had experienced intraoperatively using the videobased AR display system.

Recent Development of Augmented Reality in Surgery: A Review

Introduction

The development augmented reality devices allow physicians to incorporate data visualization into diagnostic and treatment procedures to improve work efficiency, safety, and cost and to enhance surgical training. However, the awareness of possibilities of augmented reality is generally low. This review evaluates whether augmented reality can presently improve the results of surgical procedures.

Methods

We performed a review of available literature dating from 2010 to November 2016 by searching PubMed and Scopus using the terms “augmented reality” and “surgery.” Results. The initial search yielded 808 studies. After removing duplicates and including only journal articles, a total of 417 studies were identified. By reading of abstracts, 91 relevant studies were chosen to be included. 11 references were gathered by cross-referencing. A total of 102 studies were included in this review.

Conclusions

The present literature suggest an increasing interest of surgeons regarding employing augmented reality into surgery leading to improved safety and efficacy of surgical procedures. Many studies showed that the performance of newly devised augmented reality systems is comparable to traditional techniques. However, several problems need to be addressed before augmented reality is implemented into the routine practice.

Fast calibration of electromagnetically tracked oblique-viewing rigid endoscopes

PURPOSE

The oblique-viewing (i.e., angled) rigid endoscope is a commonly used tool in conventional endoscopic surgeries. The relative rotation between its two moveable parts, the telescope and the camera head, creates a rotation offset between the actual and the projection of an object in the camera image. A calibration method tailored to compensate such offset is needed.

METHODS

We developed a fast calibration method for oblique-viewing rigid endoscopes suitable for clinical use. In contrast to prior approaches based on optical tracking, we used electromagnetic (EM) tracking as the external tracking hardware to improve compactness and practicality. Two EM sensors were mounted on the telescope and the camera head, respectively, with considerations to minimize EM tracking errors. Single-image calibration was incorporated into the method, and a sterilizable plate, laser-marked with the calibration pattern, was also developed. Furthermore, we proposed a general algorithm to estimate the rotation center in the camera image. Formulas for updating the camera matrix in terms of clockwise and counterclockwise rotations were also developed.

RESULTS

The proposed calibration method was validated using a conventional [Formula: see text], 5-mm laparoscope. Freehand calibrations were performed using the proposed method, and the calibration time averaged 2 min and 8 s. The calibration accuracy was evaluated in a simulated clinical setting with several surgical tools present in the magnetic field of EM tracking. The root-mean-square re-projection error averaged 4.9 pixel (range 2.4-8.5 pixel, with image resolution of [Formula: see text] for rotation angles ranged from [Formula: see text] to [Formula: see text].

CONCLUSIONS

We developed a method for fast and accurate calibration of oblique-viewing rigid endoscopes. The method was also designed to be performed in the operating room and will therefore support clinical translation of many emerging endoscopic computer-assisted surgical systems.

Robust augmented reality registration method for localization of solid organs' tumors using CT-derived virtual biomechanical model and fluorescent fiducials

BACKGROUND

Augmented reality (AR) is the fusion of computer-generated and real-time images. AR can be used in surgery as a navigation tool, by creating a patient-specific virtual model through 3D software manipulation of DICOM imaging (e.g., CT scan). The virtual model can be superimposed to real-time images enabling transparency visualization of internal anatomy and accurate localization of tumors. However, the 3D model is rigid and does not take into account inner structures' deformations. We present a concept of automated AR registration, while the organs undergo deformation during surgical manipulation, based on finite element modeling (FEM) coupled with optical imaging of fluorescent surface fiducials.

METHODS

Two 10 × 1 mm wires (pseudo-tumors) and six 10 × 0.9 mm fluorescent fiducials were placed in ex vivo porcine kidneys (n = 10). Biomechanical FEM-based models were generated from CT scan. Kidneys were deformed and the shape changes were identified by tracking the fiducials, using a near-infrared optical system. The changes were registered automatically with the virtual model, which was deformed accordingly. Accuracy of prediction of pseudo-tumors' location was evaluated with a CT scan in the deformed status (ground truth). In vivo: fluorescent fiducials were inserted under ultrasound guidance in the kidney of one pig, followed by a CT scan. The FEM-based virtual model was superimposed on laparoscopic images by automatic registration of the fiducials.

RESULTS

Biomechanical models were successfully generated and accurately superimposed on optical images. The mean measured distance between the estimated tumor by biomechanical propagation and the scanned tumor (ground truth) was 0.84 ± 0.42 mm. All fiducials were successfully placed in in vivo kidney and well visualized in near-infrared mode enabling accurate automatic registration of the virtual model on the laparoscopic images.

CONCLUSIONS

Our preliminary experiments showed the potential of a biomechanical model with fluorescent fiducials to propagate the deformation of solid organs' surface to their inner structures including tumors with good accuracy and automatized robust tracking.

Laparoscopic stereoscopic augmented reality: toward a clinically viable electromagnetic tracking solution

The purpose of this work was to develop a clinically viable laparoscopic augmented reality (AR) system employing stereoscopic (3-D) vision, laparoscopic ultrasound (LUS), and electromagnetic (EM) tracking to achieve image registration. We investigated clinically feasible solutions to mount the EM sensors on the 3-D laparoscope and the LUS probe. This led to a solution of integrating an externally attached EM sensor near the imaging tip of the LUS probe, only slightly increasing the overall diameter of the probe. Likewise, a solution for mounting an EM sensor on the handle of the 3-D laparoscope was proposed. The spatial image-to-video registration accuracy of the AR system was measured to be [Formula: see text] and [Formula: see text] for the left- and right-eye channels, respectively. The AR system contributed 58-ms latency to stereoscopic visualization. We further performed an animal experiment to demonstrate the use of the system as a visualization approach for laparoscopic procedures. In conclusion, we have developed an integrated, compact, and EM tracking-based stereoscopic AR visualization system, which has the potential for clinical use. The system has been demonstrated to achieve clinically acceptable accuracy and latency. This work is a critical step toward clinical translation of AR visualization for laparoscopic procedures.

Intraoperative Computed Tomography Imaging for Navigated Laparoscopic Renal Surgery: First Clinical Experience

INTRODUCTION

Laparoscopic partial nephrectomy (LPN) remains challenging in endophytic and complex kidney tumors as the clear understanding of tumor location and spreading depends on a precise analysis of available imaging. The purpose of this study was to investigate navigated kidney surgery using intraoperative cone-beam computed tomography (CBCT) images in conjunction with a previously proposed method for augmented reality (AR) guidance for safe LPN.

MATERIALS AND METHODS

The concept proposed is based on using an intraoperative CBCT scan for (1) marker-based AR guidance for fast and reliable tumor access and (2) enhancement of real-time fluoroscopy images for accurate tumor resection. Workflow and accuracy of the system were assessed using a porcine kidney model. Ten patients with complex or endophytic tumor localization and R.E.N.A.L. Nephrometry Score of at least nine scheduled for LPN were included in this study. Patients received an intraoperative CBCT after marker placement. Defining the resection line was assisted by AR. In addition, fluoroscopy imaging for depth perception was used for assistance during dissection. Feasibility and performance were assessed by histopathological results, peri- and postoperative data.

RESULTS

Surgery was performed successfully and negative margins were found in all cases. Segmental branches of the renal artery shifted as much as 10 mm in the vertical and 11 mm in the sagittal axis intraoperatively compared to preoperative imaging. Fluoroscopy to intraoperative computed tomography image fusion enabled enhanced depth perception during dissection in all cases. Radiation dose area product was 4.8 mGym².

CONCLUSIONS

The application of the navigation system is feasible and allows for safe and direct access to complex or endophytic renal masses. Radiation limits the application to selected indications.

Future of Minimally Invasive Colorectal Surgery

Minimally invasive surgery is slowly taking over as the preferred operative approach for colorectal diseases. However, many of the procedures remain technically difficult. This article will give an overview of the state of minimally invasive surgery and the many advances that have been made over the last two decades. Specifically, we discuss the introduction of the robotic platform and some of its benefits and limitations. We also describe some newer techniques related to robotics.

Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience

BACKGROUND

Modern chemotherapy achieves the shrinking of colorectal cancer liver metastases (CRLM) to such extent that they may disappear from radiological imaging. Disappearing CRLM rarely represents a complete pathological remission and have an important risk of recurrence. Augmented reality (AR) consists in the fusion of real-time patient images with a computer-generated 3D virtual patient model created from pre-operative medical imaging. The aim of this prospective pilot study is to investigate the potential of AR navigation as a tool to help locate and surgically resect missing CRLM.

METHODS

A 3D virtual anatomical model was created from thoracoabdominal CT-scans using customary software (VR RENDER(®), IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM(®), Karl Storz, Tüttlingen, Germany). Virtual and real images were manually registered in real-time using a video mixer, based on external anatomical landmarks with an estimated accuracy of 5 mm. This modality was tested in three patients, with four missing CRLM that had sizes from 12 to 24 mm, undergoing laparotomy after receiving pre-operative oxaliplatin-based chemotherapy.

RESULTS

AR display and fine registration was performed within 6 min. AR helped detect all four missing CRLM, and guided their resection. In all cases the planned security margin of 1 cm was clear and resections were confirmed to be R0 by pathology. There was no postoperative major morbidity or mortality. No local recurrence occurred in the follow-up period of 6-22 months.

CONCLUSIONS

This initial experience suggests that AR may be a helpful navigation tool for the resection of missing CRLM.

Augmented Reality Guidance for the Resection of Missing Colorectal Liver Metastases: An Initial Experience

BACKGROUND

Modern chemotherapy achieves the shrinking of colorectal cancer liver metastases (CRLM) to such extent that they may disappear from radiological imaging. Disappearing CRLM rarely represents a complete pathological remission and have an important risk of recurrence. Augmented reality (AR) consists in the fusion of real-time patient images with a computer-generated 3D virtual patient model created from pre-operative medical imaging. The aim of this prospective pilot study is to investigate the potential of AR navigation as a tool to help locate and surgically resect missing CRLM.

METHODS

A 3D virtual anatomical model was created from thoracoabdominal CT-scans using customary software (VR RENDER(®), IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM(®), Karl Storz, Tüttlingen, Germany). Virtual and real images were manually registered in real-time using a video mixer, based on external anatomical landmarks with an estimated accuracy of 5 mm. This modality was tested in three patients, with four missing CRLM that had sizes from 12 to 24 mm, undergoing laparotomy after receiving pre-operative oxaliplatin-based chemotherapy.

RESULTS

AR display and fine registration was performed within 6 min. AR helped detect all four missing CRLM, and guided their resection. In all cases the planned security margin of 1 cm was clear and resections were confirmed to be R0 by pathology. There was no postoperative major morbidity or mortality. No local recurrence occurred in the follow-up period of 6-22 months.

CONCLUSIONS

This initial experience suggests that AR may be a helpful navigation tool for the resection of missing CRLM.

On-demand calibration and evaluation for electromagnetically tracked laparoscope in augmented reality visualization

PURPOSE

Common camera calibration methods employed in current laparoscopic augmented reality systems require the acquisition of multiple images of an entire checkerboard pattern from various poses. This lengthy procedure prevents performing laparoscope calibration in the operating room (OR). The purpose of this work was to develop a fast calibration method for electromagnetically (EM) tracked laparoscopes, such that the calibration can be performed in the OR on demand.

METHODS

We designed a mechanical tracking mount to uniquely and snugly position an EM sensor to an appropriate location on a conventional laparoscope. A tool named fCalib was developed to calibrate intrinsic camera parameters, distortion coefficients, and extrinsic parameters (transformation between the scope lens coordinate system and the EM sensor coordinate system) using a single image that shows an arbitrary portion of a special target pattern. For quick evaluation of calibration results in the OR, we integrated a tube phantom with fCalib prototype and overlaid a virtual representation of the tube on the live video scene.

RESULTS

We compared spatial target registration error between the common OpenCV method and the fCalib method in a laboratory setting. In addition, we compared the calibration re-projection error between the EM tracking-based fCalib and the optical tracking-based fCalib in a clinical setting. Our results suggest that the proposed method is comparable to the OpenCV method. However, changing the environment, e.g., inserting or removing surgical tools, might affect re-projection accuracy for the EM tracking-based approach. Computational time of the fCalib method averaged 14.0 s (range 3.5 s-22.7 s).

CONCLUSIONS

We developed and validated a prototype for fast calibration and evaluation of EM tracked conventional (forward viewing) laparoscopes. The calibration method achieved acceptable accuracy and was relatively fast and easy to be performed in the OR on demand.

Three-dimensional virtual navigation versus conventional image guidance: A randomized controlled trial

OBJECTIVES/HYPOTHESIS

Providing image guidance in a 3-dimensional (3D) format, visually more in keeping with the operative field, could potentially reduce workload and lead to faster and more accurate navigation. We wished to assess a 3D virtual-view surgical navigation prototype in comparison to a traditional 2D system.

METHODS

Thirty-seven otolaryngology surgeons and trainees completed a randomized crossover navigation exercise on a cadaver model. Each subject identified three sinonasal landmarks with 3D virtual (3DV) image guidance and three landmarks with conventional cross-sectional computed tomography (CT) image guidance. Subjects were randomized with regard to which side and display type was tested initially. Accuracy, task completion time, and task workload were recorded.

RESULTS

Display type did not influence accuracy (P > 0.2) or efficiency (P > 0.3) for any of the six landmarks investigated. Pooled landmark data revealed a trend of improved accuracy in the 3DV group by 0.44 millimeters (95% confidence interval [0.00-0.88]). High-volume surgeons were significantly faster (P < 0.01) and had reduced workload scores in all domains (P < 0.01), but they were no more accurate (P > 0.28).

CONCLUSION

Real-time 3D image guidance did not influence accuracy, efficiency, or task workload when compared to conventional triplanar image guidance. The subtle pooled accuracy advantage for the 3DV view is unlikely to be of clinical significance. Experience level was strongly correlated to task completion time and workload but did not influence accuracy.

LEVEL OF EVIDENCE

N/A. Laryngoscope, 126:1510-1515, 2016.

A Novel Ultrasound-Based Registration for Image-Guided Laparoscopic Liver Ablation

Background Patient-to-image registration is a core process of image-guided surgery (IGS) systems. We present a novel registration approach for application in laparoscopic liver surgery, which reconstructs in real time an intraoperative volume of the underlying intrahepatic vessels through an ultrasound (US) sweep process. Methods An existing IGS system for an open liver procedure was adapted, with suitable instrument tracking for laparoscopic equipment. Registration accuracy was evaluated on a realistic phantom by computing the target registration error (TRE) for 5 intrahepatic tumors. The registration work flow was evaluated by computing the time required for performing the registration. Additionally, a scheme for intraoperative accuracy assessment by visual overlay of the US image with preoperative image data was evaluated. Results The proposed registration method achieved an average TRE of 7.2 mm in the left lobe and 9.7 mm in the right lobe. The average time required for performing the registration was 12 minutes. A positive correlation was found between the intraoperative accuracy assessment and the obtained TREs. Conclusions The registration accuracy of the proposed method is adequate for laparoscopic intrahepatic tumor targeting. The presented approach is feasible and fast and may, therefore, not be disruptive to the current surgical work flow.

Robotic surgery

BACKGROUND

Proficiency in minimally invasive surgery requires intensive and continuous training, as it is technically challenging for unnatural visual and haptic perceptions. Robotic and computer sciences are producing innovations to augment the surgeon's skills to achieve accuracy and high precision during complex surgery. This article reviews the current use of robotically assisted surgery, focusing on technology as well as main applications in digestive surgery, and future perspectives.

METHODS

The PubMed database was interrogated to retrieve evidence-based data on surgical applications. Internal and external consulting with key opinion leaders, renowned robotics laboratories and robotic platform manufacturers was used to produce state-of-the art business intelligence around robotically assisted surgery.

RESULTS

Selected digestive procedures (oesophagectomy, gastric bypass, pancreatic and liver resections, rectal resection for cancer) might benefit from robotic assistance, although the current level of evidence is insufficient to support widespread adoption. The surgical robotic market is growing, and a variety of projects have recently been launched at both academic and corporate levels to develop lightweight, miniaturized surgical robotic prototypes.

CONCLUSION

The magnified view, and improved ergonomics and dexterity offered by robotic platforms, might facilitate the uptake of minimally invasive procedures. Image guidance to complement robotically assisted procedures, through the concepts of augmented reality, could well represent a major revolution to increase safety and deal with difficulties associated with the new minimally invasive approaches.

Navigation surgery using an augmented reality for pancreatectomy

AIM

The aim of this study was to evaluate the utility of navigation surgery using augmented reality technology (AR-based NS) for pancreatectomy.

METHODS

The 3D reconstructed images from CT were created by segmentation. The initial registration was performed by using the optical location sensor. The reconstructed images were superimposed onto the real organs in the monitor display. Of the 19 patients who had undergone hepatobiliary and pancreatic surgery using AR-based NS, the accuracy, visualization ability, and utility of our system were assessed in five cases with pancreatectomy.

RESULTS

The position of each organ in the surface-rendering image corresponded almost to that of the actual organ. Reference to the display image allowed for safe dissection while preserving the adjacent vessels or organs. The locations of the lesions and resection line on the targeted organ were overlaid on the operating field. The initial mean registration error was improved to approximately 5 mm by our refinements. However, several problems such as registration accuracy, portability and cost still remain.

CONCLUSION

AR-based NS contributed to accurate and effective surgical resection in pancreatectomy. The pancreas appears to be a suitable organ for further investigations. This technology is promising to improve surgical quality, training, and education.

Next step in minimally invasive surgery: hybrid image-guided surgery

Surgery, interventional radiology, and advanced endoscopy have all developed minimally invasive techniques to effectively treat a variety of diseases with positive impact on patients' postoperative outcomes. However, those techniques are challenging and require extensive training. Robotics and computer sciences can help facilitate minimally invasive approaches. Furthermore, surgery, advanced endoscopy, and interventional radiology could converge towards a new hybrid specialty, hybrid image-guided minimally invasive therapies, in which the three fundamental disciplines could complement one another to maximize the positive effects and reduce the iatrogenic footprint on patients. The present manuscript describes the fundamental steps of this new paradigm shift in surgical therapies that, in our opinion, will be the next revolutionary step in minimally invasive approaches.

Real-time computer-generated integral imaging and 3D image calibration for augmented reality surgical navigation

Autostereoscopic 3D image overlay for augmented reality (AR) based surgical navigation has been studied and reported many times. For the purpose of surgical overlay, the 3D image is expected to have the same geometric shape as the original organ, and can be transformed to a specified location for image overlay. However, how to generate a 3D image with high geometric fidelity and quantitative evaluation of 3D image's geometric accuracy have not been addressed. This paper proposes a graphics processing unit (GPU) based computer-generated integral imaging pipeline for real-time autostereoscopic 3D display, and an automatic closed-loop 3D image calibration paradigm for displaying undistorted 3D images. Based on the proposed methods, a novel AR device for 3D image surgical overlay is presented, which mainly consists of a 3D display, an AR window, a stereo camera for 3D measurement, and a workstation for information processing. The evaluation on the 3D image rendering performance with 2560×1600 elemental image resolution shows the rendering speeds of 50-60 frames per second (fps) for surface models, and 5-8 fps for large medical volumes. The evaluation of the undistorted 3D image after the calibration yields sub-millimeter geometric accuracy. A phantom experiment simulating oral and maxillofacial surgery was also performed to evaluate the proposed AR overlay device in terms of the image registration accuracy, 3D image overlay accuracy, and the visual effects of the overlay. The experimental results show satisfactory image registration and image overlay accuracy, and confirm the system usability.

Towards cybernetic surgery: robotic and augmented reality-assisted liver segmentectomy

BACKGROUND

Augmented reality (AR) in surgery consists in the fusion of synthetic computer-generated images (3D virtual model) obtained from medical imaging preoperative workup and real-time patient images in order to visualize unapparent anatomical details. The 3D model could be used for a preoperative planning of the procedure. The potential of AR navigation as a tool to improve safety of the surgical dissection is outlined for robotic hepatectomy.

MATERIALS AND METHODS

Three patients underwent a fully robotic and AR-assisted hepatic segmentectomy. The 3D virtual anatomical model was obtained using a thoracoabdominal CT scan with a customary software (VR-RENDER®, IRCAD). The model was then processed using a VR-RENDER® plug-in application, the Virtual Surgical Planning (VSP®, IRCAD), to delineate surgical resection planes including the elective ligature of vascular structures. Deformations associated with pneumoperitoneum were also simulated. The virtual model was superimposed to the operative field. A computer scientist manually registered virtual and real images using a video mixer (MX 70; Panasonic, Secaucus, NJ) in real time.

RESULTS

Two totally robotic AR segmentectomy V and one segmentectomy VI were performed. AR allowed for the precise and safe recognition of all major vascular structures during the procedure. Total time required to obtain AR was 8 min (range 6-10 min). Each registration (alignment of the vascular anatomy) required a few seconds. Hepatic pedicle clamping was never performed. At the end of the procedure, the remnant liver was correctly vascularized. Resection margins were negative in all cases. The postoperative period was uneventful without perioperative transfusion.

CONCLUSIONS

AR is a valuable navigation tool which may enhance the ability to achieve safe surgical resection during robotic hepatectomy.

Clinical application of navigation surgery using augmented reality in the abdominal field

This article presents general principles and recent advancements in the clinical application of augmented reality-based navigation surgery (AR based NS) for abdominal procedures and includes a description of our clinical trial and subsequent outcomes. Moreover, current problems and future aspects are discussed. The development of AR-based NS in the abdomen is delayed compared with another field because of the problem of intraoperative organ deformations or the existence of established modalities. Although there are a few reports on the clinical use of AR-based NS for digestive surgery, sophisticated technologies in urology have often been reported. However, the rapid widespread use of video- or robot assisted surgeries requires this technology. We have worked to develop a system of AR-based NS for hepatobiliary and pancreatic surgery. Then we developed a short rigid scope that enables surgeons to obtain 3D view. We recently focused on pancreatic surgery, because intraoperative organ shifting is minimal. The position of each organ in overlaid image almost corresponded with that of the actual organ with about 5 mm of mean registration errors. Intraoperative information generated from this system provided us with useful navigation. However, AR-based NS has several problems to overcome such as organ deformity, evaluation of utility, portability or cost.

Real-time 3D image reconstruction guidance in liver resection surgery

BACKGROUND

Minimally invasive surgery represents one of the main evolutions of surgical techniques. However, minimally invasive surgery adds difficulty that can be reduced through computer technology.

METHODS

From a patient's medical image [US, computed tomography (CT) or MRI], we have developed an Augmented Reality (AR) system that increases the surgeon's intraoperative vision by providing a virtual transparency of the patient. AR is based on two major processes: 3D modeling and visualization of anatomical or pathological structures appearing in the medical image, and the registration of this visualization onto the real patient. We have thus developed a new online service, named Visible Patient, providing efficient 3D modeling of patients. We have then developed several 3D visualization and surgical planning software tools to combine direct volume rendering and surface rendering. Finally, we have developed two registration techniques, one interactive and one automatic providing intraoperative augmented reality view.

RESULTS

From January 2009 to June 2013, 769 clinical cases have been modeled by the Visible Patient service. Moreover, three clinical validations have been realized demonstrating the accuracy of 3D models and their great benefit, potentially increasing surgical eligibility in liver surgery (20% of cases). From these 3D models, more than 50 interactive AR-assisted surgical procedures have been realized illustrating the potential clinical benefit of such assistance to gain safety, but also current limits that automatic augmented reality will overcome.

CONCLUSIONS

Virtual patient modeling should be mandatory for certain interventions that have now to be defined, such as liver surgery. Augmented reality is clearly the next step of the new surgical instrumentation but remains currently limited due to the complexity of organ deformations during surgery. Intraoperative medical imaging used in new generation of automated augmented reality should solve this issue thanks to the development of Hybrid OR.

Robotic duodenopancreatectomy assisted with augmented reality and real-time fluorescence guidance

BACKGROUND

The minimally invasive surgeon cannot use 'sense of touch' to orientate surgical resection, identifying important structures (vessels, tumors, etc.) by manual palpation. Robotic research has provided technology to facilitate laparoscopic surgery; however, robotics has yet to solve the lack of tactile feedback inherent to keyhole surgery. Misinterpretation of the vascular supply and tumor location may increase the risk of intraoperative bleeding and worsen dissection with positive resection margins.

METHODS

Augmented reality (AR) consists of the fusion of synthetic computer-generated images (three-dimensional virtual model) obtained from medical imaging preoperative work-up and real-time patient images with the aim of visualizing unapparent anatomical details.

RESULTS

In this article, we review the most common modalities used to achieve surgical navigation through AR, along with a report of a case of robotic duodenopancreatectomy using AR guidance complemented with the use of fluorescence guidance.

CONCLUSIONS

The presentation of this complex and high-technology case of robotic duodenopancreatectomy, and the overview of current technology that has made it possible to use AR in the operating room, highlights the needs for further evolution and the windows of opportunity to create a new paradigm in surgical practice.

Stereoscopic augmented reality for laparoscopic surgery

BACKGROUND

Conventional laparoscopes provide a flat representation of the three-dimensional (3D) operating field and are incapable of visualizing internal structures located beneath visible organ surfaces. Computed tomography (CT) and magnetic resonance (MR) images are difficult to fuse in real time with laparoscopic views due to the deformable nature of soft-tissue organs. Utilizing emerging camera technology, we have developed a real-time stereoscopic augmented-reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound (LUS) with stereoscopic video. The system creates two new visual cues: (1) perception of true depth with improved understanding of 3D spatial relationships among anatomical structures, and (2) visualization of critical internal structures along with a more comprehensive visualization of the operating field.

METHODS

The stereoscopic AR system has been designed for near-term clinical translation with seamless integration into the existing surgical workflow. It is composed of a stereoscopic vision system, a LUS system, and an optical tracker. Specialized software processes streams of imaging data from the tracked devices and registers those in real time. The resulting two ultrasound-augmented video streams (one for the left and one for the right eye) give a live stereoscopic AR view of the operating field. The team conducted a series of stereoscopic AR interrogations of the liver, gallbladder, biliary tree, and kidneys in two swine.

RESULTS

The preclinical studies demonstrated the feasibility of the stereoscopic AR system during in vivo procedures. Major internal structures could be easily identified. The system exhibited unobservable latency with acceptable image-to-video registration accuracy.

CONCLUSIONS

We presented the first in vivo use of a complete system with stereoscopic AR visualization capability. This new capability introduces new visual cues and enhances visualization of the surgical anatomy. The system shows promise to improve the precision and expand the capacity of minimally invasive laparoscopic surgeries.

Real-time image guidance in laparoscopic liver surgery: first clinical experience with a guidance system based on intraoperative CT imaging

BACKGROUND

Laparoscopic liver surgery is particularly challenging owing to restricted access, risk of bleeding, and lack of haptic feedback. Navigation systems have the potential to improve information on the exact position of intrahepatic tumors, and thus facilitate oncological resection. This study aims to evaluate the feasibility of a commercially available augmented reality (AR) guidance system employing intraoperative robotic C-arm cone-beam computed tomography (CBCT) for laparoscopic liver surgery.

METHODS

A human liver-like phantom with 16 target fiducials was used to evaluate the Syngo iPilot(®) AR system. Subsequently, the system was used for the laparoscopic resection of a hepatocellular carcinoma in segment 7 of a 50-year-old male patient.

RESULTS

In the phantom experiment, the AR system showed a mean target registration error of 0.96 ± 0.52 mm, with a maximum error of 2.49 mm. The patient successfully underwent the operation and showed no postoperative complications.

CONCLUSION

The use of intraoperative CBCT and AR for laparoscopic liver resection is feasible and could be considered an option for future liver surgery in complex cases.

Augmented reality-guided artery-first pancreatico-duodenectomy

BACKGROUND

Augmented Reality (AR) in surgery consists in the fusion of synthetic computer-generated images (3D virtual model) obtained from medical imaging preoperative work-up and real-time patient images with the aim to visualize unapparent anatomical details. The potential of AR navigation as a tool to improve safety of the surgical dissection is presented in a case of pancreatico-duodenectomy (PD).

METHODS

A 77-year-old male patient underwent an AR-assisted PD. The 3D virtual anatomical model was obtained from thoraco-abdominal CT scan using customary software (VR-RENDER®, IRCAD). The virtual model was superimposed to the operative field using an Exoscope (VITOM®, Karl Storz, Tüttlingen, Germany) as well as different visible landmarks (inferior vena cava, left renal vein, aorta, superior mesenteric vein, inferior margin of the pancreas). A computer scientist manually registered virtual and real images using a video mixer (MX 70; Panasonic, Secaucus, NJ) in real time. Dissection of the superior mesenteric artery and the hanging maneuver were performed under AR guidance along the hanging plane.

RESULTS

AR allowed for precise and safe recognition of all the important vascular structures. Operative time was 360 min. AR display and fine registration was performed within 6 min. The postoperative course was uneventful. The pathology was positive for ampullary adenocarcinoma; the final stage was pT1N0 (0/43 retrieved lymph nodes) with clear surgical margins.

CONCLUSIONS

AR is a valuable navigation tool that can enhance the ability to achieve a safe surgical resection during PD.

Utility of augmented reality system in hepatobiliary surgery

BACKGROUND/PURPOSE

The aim of this study was to evaluate the utility of an image display system for augmented reality in hepatobiliary surgery under laparotomy.

METHODS

An overlay display of organs, vessels, or tumor was obtained using a video see-through system as a display system developed at our institute. Registration between visceral organs and the surface-rendering image reconstructed by preoperative computed tomography (CT) was carried out with an optical location sensor. Using this system, we performed laparotomy for a patient with benign biliary stricture, a patient with gallbladder carcinoma, and a patient with hepatocellular carcinoma.

RESULTS

The operative procedures performed consisted of choledochojejunostomy, right hepatectomy, and microwave coagulation therapy. All the operations were carried out safely using images of the site of tumor, preserved organs, and resection aspect overlaid onto the operation field images observed on the monitors. The position of each organ in the overlaid image closely corresponded with that of the actual organ. Intraoperative information generated from this system provided us with useful navigation. However, several problems such as registration error and lack of depth knowledge were noted.

CONCLUSION

The image display system appeared to be useful in performing hepatobiliary surgery under laparotomy. Further improvement of the system with individualized function for each operation will be essential, with feedback from clinical trials in the future.

The state of the art of visualization in mixed reality image guided surgery

This paper presents a review of the state of the art of visualization in mixed reality image guided surgery (IGS). We used the DVV (data, visualization processing, view) taxonomy to classify a large unbiased selection of publications in the field. The goal of this work was not only to give an overview of current visualization methods and techniques in IGS but more importantly to analyze the current trends and solutions used in the domain. In surveying the current landscape of mixed reality IGS systems, we identified a strong need to assess which of the many possible data sets should be visualized at particular surgical steps, to focus on novel visualization processing techniques and interface solutions, and to evaluate new systems.

Surgeons blinded by enhanced navigation: the effect of augmented reality on attention

BACKGROUND

Advanced image-guidance systems allowing presentation of three-dimensional navigational data in real time are being developed enthusiastically for many medical procedures. Other industries, including aviation and the military, have noted that shifting attention toward such compelling assistance has detrimental effects. Using the detection rate of unexpected findings, we assess whether inattentional blindness is significant in a surgical context and evaluate the impact of on-screen navigational cuing with augmented reality.

METHODS

Surgeons and trainees performed an endoscopic navigation exercise on a cadaveric specimen. The subjects were randomized to either a standard endoscopic view (control) or an AR view consisting of an endoscopic video fused with anatomic contours. Two unexpected findings were presented in close proximity to the target point: one critical complication and one foreign body (screw). Task completion time, accuracy, and recognition of findings were recorded.

RESULTS

Detection of the complication was 0/15 in the AR group versus 7/17 in the control group (p = 0.008). Detection of the screw was 1/15 (AR) and 7/17 (control) (p = 0.041). Recognition of either finding was 12/17 for the control group and 1/15 for the AR group (p < 0.001). Accuracy was greater for the AR group than for the control group, with the median distance from the target point measuring respectively 2.10 mm (interquartile range [IQR], 1.29-2.37) and 4.13 (IQR, 3.11-7.39) (p < 0.001).

CONCLUSION

Inattentional blindness was evident in both groups. Although more accurate, the AR group was less likely to identify significant unexpected findings clearly within view. Advanced navigational displays may increase precision, but strategies to mitigate attentional costs need further investigation to allow safe implementation.

[Intraoperative augmented reality visualization. Current state of development and initial experiences with the CamC]

The intraoperative application of augmented reality (AR) has so far mainly taken place in the field of endoscopy. Here, the camera image of the endoscope was augmented by computer graphics derived mostly from preoperative imaging. Due to the complex setup and operation of the devices, they have not yet become part of routine clinical practice. The Camera Augmented Mobile C-arm (CamC) that extends a classic C-arm by a video camera and mirror construction is characterized by its uncomplicated handling. It combines its video live stream geometrically correct with the acquired X-ray. The clinical application of the device in 43 cases showed the strengths of the device in positioning for X-ray acquisition, incision placement, K-wire placement, and instrument guidance. With its new function and the easy integration into the OR workflow of any procedure that requires X-ray imaging, the CamC has the potential to become the first widely used AR technology for orthopedic and trauma surgery.

Augmented reality and image overlay navigation with OsiriX in laparoscopic and robotic surgery: not only a matter of fashion

BACKGROUND

New technologies can considerably improve preoperative planning, enhance the surgeon's skill and simplify the approach to complex procedures. Augmented reality techniques, robot assisted operations and computer assisted navigation tools will become increasingly important in surgery and in residents' education.

METHODS

We obtained 3D reconstructions from simple spiral computed tomography (CT) slides using OsiriX, an open source processing software package dedicated to DICOM images. These images were then projected on the patient's body with a beamer fixed to the operating table to enhance spatial perception during surgical intervention (augmented reality).

RESULTS

Changing a window's deepness level allowed the surgeon to navigate through the patient's anatomy, highlighting regions of interest and marked pathologies. We used image overlay navigation for laparoscopic operations such cholecystectomy, abdominal exploration, distal pancreas resection and robotic liver resection.

CONCLUSIONS

Augmented reality techniques will transform the behaviour of surgeons, making surgical interventions easier, faster and probably safer. These new techniques will also renew methods of surgical teaching, facilitating transmission of knowledge and skill to young surgeons.

Augmented reality in laparoscopic surgical oncology

Minimally invasive surgery represents one of the main evolutions of surgical techniques aimed at providing a greater benefit to the patient. However, minimally invasive surgery increases the operative difficulty since the depth perception is usually dramatically reduced, the field of view is limited and the sense of touch is transmitted by an instrument. However, these drawbacks can currently be reduced by computer technology guiding the surgical gesture. Indeed, from a patient's medical image (US, CT or MRI), Augmented Reality (AR) can increase the surgeon's intra-operative vision by providing a virtual transparency of the patient. AR is based on two main processes: the 3D visualization of the anatomical or pathological structures appearing in the medical image, and the registration of this visualization on the real patient. 3D visualization can be performed directly from the medical image without the need for a pre-processing step thanks to volume rendering. But better results are obtained with surface rendering after organ and pathology delineations and 3D modelling. Registration can be performed interactively or automatically. Several interactive systems have been developed and applied to humans, demonstrating the benefit of AR in surgical oncology. It also shows the current limited interactivity due to soft organ movements and interaction between surgeon instruments and organs. If the current automatic AR systems show the feasibility of such system, it is still relying on specific and expensive equipment which is not available in clinical routine. Moreover, they are not robust enough due to the high complexity of developing a real-time registration taking organ deformation and human movement into account. However, the latest results of automatic AR systems are extremely encouraging and show that it will become a standard requirement for future computer-assisted surgical oncology. In this article, we will explain the concept of AR and its principles. Then, we will review the existing interactive and automatic AR systems in digestive surgical oncology, highlighting their benefits and limitations. Finally, we will discuss the future evolutions and the issues that still have to be tackled so that this technology can be seamlessly integrated in the operating room.