Augmented reality in liver surgery

Real-time deformable SLAM with geometrically adapted template for dynamic monocular laparoscopic scenes

PURPOSE

Intraoperative reconstruction of endoscopic scenes is a key technology for surgical navigation systems. The accuracy and efficiency of 3D reconstruction directly determine the effectiveness of navigation systems in a variety of clinical applications. While current deformable SLAM algorithms can meet real-time requirements, their underlying reliance on regular templates still makes it challenging to efficiently capture abrupt geometric features within scenes, such as organ contours and surgical margins.

METHODS

We propose a novel real-time monocular deformable SLAM algorithm with geometrically adapted template. To ensure real-time performance, the proposed algorithm consists of two threads: a deformation mapping thread updates the template at keyframe rate and a deformation tracking thread estimates the camera pose and the deformation at frame rate. To capture geometric features more efficiently, the algorithm first detects salient edge features using a pre-trained contour detection network and then constructs the template through a triangulation method with guidance of the salient features.

RESULTS

We thoroughly evaluated this method on Mandala and Hamlyn datasets in terms of accuracy and performance. The results demonstrated that the proposed method achieves better accuracy with 0.75-7.95% improvement and achieves consistent effectiveness in data association compared with the closest method.

CONCLUSION

This study verified an adaptive template does improve the performance of reconstruction of dynamic laparoscopic Scenes with abrupt geometric features. However, further exploration is needed for applications in laparoscopic surgery with incisal margins caused by surgical instruments. This research serves as a crucial step toward enhanced automatic computer-assisted navigation in laparoscopic surgery. Code is available at https://github.com/Tang257/SLAM-with-geometrically-adapted-template .

Development of real-time navigation system for laparoscopic hepatectomy using magnetic micro sensor

BACKGROUND

We report a new real-time navigation system for laparoscopic hepatectomy (LH), which resembles a car navigation system.

MATERIAL AND METHODS

Virtual three-dimensional liver and body images were reconstructed using the "New-VES" system, which worked as roadmap during surgery. Several points of the patient's body were registered in virtual images using a magnetic position sensor (MPS). A magnetic transmitter, corresponding to an artificial satellite, was placed about 40 cm above the patient's body. Another MPS, corresponding to a GPS antenna, was fixed on the handling part of the laparoscope. Fiducial registration error (FRE, an error between real and virtual lengths) was utilized to evaluate the accuracy of this system.

RESULTS

Twenty-one patients underwent LH with this system. Mean FRE of the initial five patients was 17.7 mm. Mean FRE of eight patients in whom MDCT was taken using radiological markers for registration of body parts as first improvement, was reduced to 10.2 mm (p = .014). As second improvement, a new MPS as an intraoperative body position sensor was fixed on the right-sided chest wall for automatic correction of postural gap. The preoperative and postoperative mean FREs of 8 patients with both improvements were 11.1 mm and 10.1 mm (p = .250).

CONCLUSIONS

Our system may provide a promising option that virtually guides LH.

Hybrid operating room applications for precision hepatobiliary surgery: A narrative review

This review summarizes the key applications of a hybrid operating room (HOR) in hepatobiliary surgery and explores the advantages, limitations, and future directions of its utilization. A comprehensive literature search was conducted in PubMed to identify articles reporting on the utilization of HORs in liver surgery. So far, the HOR has been limitedly applied in hepatobiliary surgery. It can offer an optimal environment for combining radiological and surgical interventions and for performing image-guided surgical navigation.

Tracking and mapping in medical computer vision: A review

As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.

A clinical assessment of three-dimensional-printed liver model navigation for thrice or more repeated hepatectomy based on a conversation analysis

PURPOSES

We performed a conversation analysis of the speech conducted among the surgical team during three-dimensional (3D)-printed liver model navigation for thrice or more repeated hepatectomy (TMRH).

METHODS

Seventeen patients underwent 3D-printed liver navigation surgery for TMRH. After transcription of the utterances recorded during surgery, the transcribed utterances were coded by the utterer, utterance object, utterance content, sensor, and surgical process during conversation. We then analyzed the utterances and clarified the association between the surgical process and conversation through the intraoperative reference of the 3D-printed liver.

RESULTS

In total, 130 conversations including 1648 segments were recorded. Utterance coding showed that the operator/assistant, 3D-printed liver/real liver, fact check (F)/plan check (Pc), visual check/tactile check, and confirmation of planned resection or preservation target (T)/confirmation of planned or ongoing resection line (L) accounted for 791/857, 885/763, 1148/500, 1208/440, and 1304/344 segments, respectively. The utterance's proportions of assistants, F, F of T on 3D-printed liver, F of T on real liver, and Pc of L on 3D-printed liver were significantly higher during non-expert surgeries than during expert surgeries. Confirming the surgical process with both 3D-printed liver and real liver and performing planning using a 3D-printed liver facilitates the safe implementation of TMRH, regardless of the surgeon's experience.

CONCLUSIONS

The present study, using a unique conversation analysis, provided the first evidence for the clinical value of 3D-printed liver for TMRH for anatomical guidance of non-expert surgeons.

First-in-human real-time AI-assisted instrument deocclusion during augmented reality robotic surgery

The integration of Augmented Reality (AR) into daily surgical practice is withheld by the correct registration of pre-operative data. This includes intelligent 3D model superposition whilst simultaneously handling real and virtual occlusions caused by the AR overlay. Occlusions can negatively impact surgical safety and as such deteriorate rather than improve surgical care. Robotic surgery is particularly suited to tackle these integration challenges in a stepwise approach as the robotic console allows for different inputs to be displayed in parallel to the surgeon. Nevertheless, real-time de-occlusion requires extensive computational resources which further complicates clinical integration. This work tackles the problem of instrument occlusion and presents, to the authors' best knowledge, the first-in-human on edge deployment of a real-time binary segmentation pipeline during three robot-assisted surgeries: partial nephrectomy, migrated endovascular stent removal, and liver metastasectomy. To this end, a state-of-the-art real-time segmentation and 3D model pipeline was implemented and presented to the surgeon during live surgery. The pipeline allows real-time binary segmentation of 37 non-organic surgical items, which are never occluded during AR. The application features real-time manual 3D model manipulation for correct soft tissue alignment. The proposed pipeline can contribute towards surgical safety, ergonomics, and acceptance of AR in minimally invasive surgery.

Virtual and augmented reality in biomedical engineering

BACKGROUND

In the future, extended reality technology will be widely used. People will be led to utilize virtual reality (VR) and augmented reality (AR) technologies in their daily lives, hobbies, numerous types of entertainment, and employment. Medical augmented reality has evolved with applications ranging from medical education to picture-guided surgery. Moreover, a bulk of research is focused on clinical applications, with the majority of research devoted to surgery or intervention, followed by rehabilitation and treatment applications. Numerous studies have also looked into the use of augmented reality in medical education and training.

METHODS

Using the databases Semantic Scholar, Web of Science, Scopus, IEEE Xplore, and ScienceDirect, a scoping review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria. To find other articles, a manual search was also carried out in Google Scholar. This study presents studies carried out over the previous 14 years (from 2009 to 2023) in detail. We classify this area of study into the following categories: (1) AR and VR in surgery, which is presented in the following subsections: subsection A: MR in neurosurgery; subsection B: spine surgery; subsection C: oral and maxillofacial surgery; and subsection D: AR-enhanced human-robot interaction; (2) AR and VR in medical education presented in the following subsections; subsection A: medical training; subsection B: schools and curriculum; subsection C: XR in Biomedicine; (3) AR and VR for rehabilitation presented in the following subsections; subsection A: stroke rehabilitation during COVID-19; subsection B: cancer and VR, and (4) Millimeter-wave and MIMO systems for AR and VR.

RESULTS

In total, 77 publications were selected based on the inclusion criteria. Four distinct AR and/or VR applications groups could be differentiated: AR and VR in surgery (N = 21), VR and AR in Medical Education (N = 30), AR and VR for Rehabilitation (N = 15), and Millimeter-Wave and MIMO Systems for AR and VR (N = 7), where N is number of cited studies. We found that the majority of research is devoted to medical training and education, with surgical or interventional applications coming in second. The research is mostly focused on rehabilitation, therapy, and clinical applications. Moreover, the application of XR in MIMO has been the subject of numerous research.

CONCLUSION

Examples of these diverse fields of applications are displayed in this review as follows: (1) augmented reality and virtual reality in surgery; (2) augmented reality and virtual reality in medical education; (3) augmented reality and virtual reality for rehabilitation; and (4) millimeter-wave and MIMO systems for augmented reality and virtual reality.