Design of interactive augmented reality functions for robotic surgery and evaluation in dry-lab lymphadenectomy

Can Artificial Intelligence Aid Diagnosis by Teleguided Point-of-Care Ultrasound? A Pilot Study for Evaluating a Novel Computer Algorithm for COVID-19 Diagnosis Using Lung Ultrasound

With the 2019 coronavirus disease (COVID-19) pandemic, there is an increasing demand for remote monitoring technologies to reduce patient and provider exposure. One field that has an increasing potential is teleguided ultrasound, where telemedicine and point-of-care ultrasound (POCUS) merge to create this new scope. Teleguided POCUS can minimize staff exposure while preserving patient safety and oversight during bedside procedures. In this paper, we propose the use of teleguided POCUS supported by AI technologies for the remote monitoring of COVID-19 patients by non-experienced personnel including self-monitoring by the patients themselves. Our hypothesis is that AI technologies can facilitate the remote monitoring of COVID-19 patients through the utilization of POCUS devices, even when operated by individuals without formal medical training. In pursuit of this goal, we performed a pilot analysis to evaluate the performance of users with different clinical backgrounds using a computer-based system for COVID-19 detection using lung ultrasound. The purpose of the analysis was to emphasize the potential of the proposed AI technology for improving diagnostic performance, especially for users with less experience.

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

The Role of Augmented Reality in the Advancement of Minimally Invasive Surgery Procedures: A Scoping Review

The purpose of this review was to analyze the evidence on the role of augmented reality (AR) in the improvement of minimally invasive surgical (MIS) procedures. A scoping literature search of the PubMed and ScienceDirect databases was performed to identify articles published in the last five years that addressed the direct impact of AR technology on MIS procedures or that addressed an area of education or clinical care that could potentially be used for MIS development. A total of 359 studies were screened and 31 articles were reviewed in depth and categorized into three main groups: Navigation, education and training, and user-environment interfaces. A comparison of studies within the different application groups showed that AR technology can be useful in various disciplines to advance the development of MIS. Although AR-guided navigation systems do not yet offer a precision advantage, benefits include improved ergonomics and visualization, as well as reduced surgical time and blood loss. Benefits can also be seen in improved education and training conditions and improved user-environment interfaces that can indirectly influence MIS procedures. However, there are still technical challenges that need to be addressed to demonstrate added value to patient care and should be evaluated in clinical trials with sufficient patient numbers or even in systematic reviews or meta-analyses.

Augmented reality for minimally invasive spinal surgery

Background

Augmented reality (AR) is an emerging technology that can overlay computer graphics onto the real world and enhance visual feedback from information systems. Within the past several decades, innovations related to AR have been integrated into our daily lives; however, its application in medicine, specifically in minimally invasive spine surgery (MISS), may be most important to understand. AR navigation provides auditory and haptic feedback, which can further enhance surgeons' capabilities and improve safety.

Purpose

The purpose of this article is to address previous and current applications of AR, AR in MISS, limitations of today's technology, and future areas of innovation.

Methods

A literature review related to applications of AR technology in previous and current generations was conducted.

Results

AR systems have been implemented for treatments related to spinal surgeries in recent years, and AR may be an alternative to current approaches such as traditional navigation, robotically assisted navigation, fluoroscopic guidance, and free hand. As AR is capable of projecting patient anatomy directly on the surgical field, it can eliminate concern for surgeon attention shift from the surgical field to navigated remote screens, line-of-sight interruption, and cumulative radiation exposure as the demand for MISS increases.

Conclusion

AR is a novel technology that can improve spinal surgery, and limitations will likely have a great impact on future technology.

A parallel network utilizing local features and global representations for segmentation of surgical instruments

PURPOSE

Automatic image segmentation of surgical instruments is a fundamental task in robot-assisted minimally invasive surgery, which greatly improves the context awareness of surgeons during the operation. A novel method based on Mask R-CNN is proposed in this paper to realize accurate instance segmentation of surgical instruments.

METHODS

A novel feature extraction backbone is built, which could extract both local features through the convolutional neural network branch and global representations through the Swin-Transformer branch. Moreover, skip fusions are applied in the backbone to fuse both features and improve the generalization ability of the network.

RESULTS

The proposed method is evaluated on the dataset of MICCAI 2017 EndoVis Challenge with three segmentation tasks and shows state-of-the-art performance with an mIoU of 0.5873 in type segmentation and 0.7408 in part segmentation. Furthermore, the results of ablation studies prove that the proposed novel backbone contributes to at least 17% improvement in mIoU.

CONCLUSION

The promising results demonstrate that our method can effectively extract global representations as well as local features in the segmentation of surgical instruments and improve the accuracy of segmentation. With the proposed novel backbone, the network can segment the contours of surgical instruments' end tips more precisely. This method can provide more accurate data for localization and pose estimation of surgical instruments, and make a further contribution to the automation of robot-assisted minimally invasive surgery.

Augmented Reality-Based Visual Cue for Guiding Central Catheter Insertion in Pediatric Oncologic Patients

BACKGROUND

Pediatric hemato-oncologic patients require central catheters for chemotherapy, and the junction of the superior vena cava and right atrium is considered the ideal location for catheter tips. Skin landmarks or fluoroscopic supports have been applied to identify the cavoatrial junction; however, none has been recognized as the gold standard. Therefore, we aim to develop a safe and accurate technique using augmented reality technology for the location of the cavoatrial junction in pediatric hemato-oncologic patients.

METHODS

Fifteen oncology patients who underwent chest computed tomography were enrolled for Hickman catheter or chemoport insertion. With the aid of augmented reality technology, three-dimensional models of the internal jugular veins, external jugular veins, subclavian veins, superior vena cava, and right atrium were constructed. On inserting the central vein catheters, the cavoatrial junction identified using the three-dimensional models were marked on the body surface, the tip was positioned at the corresponding location, and the actual insertion location was confirmed using a portable x-ray machine. The proposed method was evaluated by comparing the distance from the cavoatrial junction to the augmented reality location with that to the conventional location on x-ray.

RESULTS

The mean distance between the cavoatrial junction and augmented reality location on x-ray was 1.2 cm, which was significantly shorter than that between the cavoatrial junction and conventional location (1.9 cm; P = 0.027).

CONCLUSIONS

Central catheter insertion using augmented reality technology is more safe and accurate than that using conventional methods and can be performed at no additional cost in oncology patients.