Vision-based tracking system for augmented reality to localize recurrent laryngeal nerve during robotic thyroid surgery

A decade of progress: bringing mixed reality image-guided surgery systems in the operating room

Advancements in mixed reality (MR) have led to innovative approaches in image-guided surgery (IGS). In this paper, we provide a comprehensive analysis of the current state of MR in image-guided procedures across various surgical domains. Using the Data Visualization View (DVV) Taxonomy, we analyze the progress made since a 2013 literature review paper on MR IGS systems. In addition to examining the current surgical domains using MR systems, we explore trends in types of MR hardware used, type of data visualized, visualizations of virtual elements, and interaction methods in use. Our analysis also covers the metrics used to evaluate these systems in the operating room (OR), both qualitative and quantitative assessments, and clinical studies that have demonstrated the potential of MR technologies to enhance surgical workflows and outcomes. We also address current challenges and future directions that would further establish the use of MR in IGS.

High aggressiveness of papillary thyroid cancer: from clinical evidence to regulatory cellular networks

The global incidence of thyroid cancer has increased over recent decades. Papillary thyroid cancer (PTC) is the most common type of thyroid cancer and accounts for nearly 90% of all cases. Typically, PTC has a good prognosis. However, some PTC variants exhibit more aggressive behaviour, which significantly increases the risk of postoperative recurrence. Over the past decade, the high metastatic potential of PTC has drawn the attention of many researchers and these studies have provided useful molecular markers for improved diagnosis, risk stratification and clinical approaches. The aim of this review is to discuss the progress in epidemiology, metastatic features, risk factors and molecular mechanisms associated with PTC aggressiveness. We present a detailed picture showing that epithelial-to-mesenchymal transition, cancer metabolic reprogramming, alterations in important signalling pathways, epigenetic aberrations and the tumour microenvironment are crucial drivers of PTC metastasis. Further research is needed to more fully elucidate the pathogenesis and biological behaviour underlying the aggressiveness of PTC.

A prostate seed implantation robot system based on human-computer interactions: Augmented reality and voice control

The technology of robot-assisted prostate seed implantation has developed rapidly. However, during the process, there are some problems to be solved, such as non-intuitive visualization effects and complicated robot control. To improve the intelligence and visualization of the operation process, a voice control technology of prostate seed implantation robot in augmented reality environment was proposed. Initially, the MRI image of the prostate was denoised and segmented. The three-dimensional model of prostate and its surrounding tissues was reconstructed by surface rendering technology. Combined with holographic application program, the augmented reality system of prostate seed implantation was built. An improved singular value decomposition three-dimensional registration algorithm based on iterative closest point was proposed, and the results of three-dimensional registration experiments verified that the algorithm could effectively improve the three-dimensional registration accuracy. A fusion algorithm based on spectral subtraction and BP neural network was proposed. The experimental results showed that the average delay of the fusion algorithm was 1.314 s, and the overall response time of the integrated system was 1.5 s. The fusion algorithm could effectively improve the reliability of the voice control system, and the integrated system could meet the responsiveness requirements of prostate seed implantation.

Comparing a virtual reality head-mounted display to on-screen three-dimensional visualization and two-dimensional computed tomography data for training in decision making in hepatic surgery: a randomized controlled study

OBJECTIVE

Evaluation of the benefits of a virtual reality (VR) environment with a head-mounted display (HMD) for decision-making in liver surgery.

BACKGROUND

Training in liver surgery involves appraising radiologic images and considering the patient's clinical information. Accurate assessment of 2D-tomography images is complex and requires considerable experience, and often the images are divorced from the clinical information. We present a comprehensive and interactive tool for visualizing operation planning data in a VR environment using a head-mounted-display and compare it to 3D visualization and 2D-tomography.

METHODS

Ninety medical students were randomized into three groups (1:1:1 ratio). All participants analyzed three liver surgery patient cases with increasing difficulty. The cases were analyzed using 2D-tomography data (group "2D"), a 3D visualization on a 2D display (group "3D") or within a VR environment (group "VR"). The VR environment was displayed using the "Oculus Rift ™" HMD technology. Participants answered 11 questions on anatomy, tumor involvement and surgical decision-making and 18 evaluative questions (Likert scale).

RESULTS

Sum of correct answers were significantly higher in the 3D (7.1 ± 1.4, p < 0.001) and VR (7.1 ± 1.4, p < 0.001) groups than the 2D group (5.4 ± 1.4) while there was no difference between 3D and VR (p = 0.987). Times to answer in the 3D (6:44 ± 02:22 min, p < 0.001) and VR (6:24 ± 02:43 min, p < 0.001) groups were significantly faster than the 2D group (09:13 ± 03:10 min) while there was no difference between 3D and VR (p = 0.419). The VR environment was evaluated as most useful for identification of anatomic anomalies, risk and target structures and for the transfer of anatomical and pathological information to the intraoperative situation in the questionnaire.

CONCLUSIONS

A VR environment with 3D visualization using a HMD is useful as a surgical training tool to accurately and quickly determine liver anatomy and tumor involvement in surgery.

Future Directions in the Treatment of Thyroid and Parathyroid Disease

The surgical management of thyroid and parathyroid disease has evolved considerably since the era of Theodor Kocher. We review the current trends in thyroid and parathyroid surgery concerning robotic surgery for remote access, the use of parathyroid autofluorescence detection technology to aid in the prevention of hypocalcemia as well as the use of thermal ablation to target thyroid nodules in a minimally invasive way. We also discuss how artificial intelligence is being used to improve the workflow and diagnostics preoperatively as well as for intraoperative decision-making. We also discuss potential areas where future research may enhance outcomes.

Augmented reality navigation-guided intraoperative pulmonary nodule localization: a pilot study

Background

With the increasing number of small pulmonary nodules detected, effective localization of pulmonary nodules has become an issue. The goal of this study is to determine the safety and feasibility of a newly developed augmented reality navigation technology for intraoperative localization of small pulmonary nodules.

Methods

We conducted a prospective single-center feasibility study of a novel augmented reality navigation system and lung localization (LungBrella) marker on ten patients between July and October 2020. For augmented reality navigation-guided localization, a preoperative chest computed tomography scan was performed to generate 3-dimensional (3D) virtual images and individualized localization plan, which were uploaded into Hololens (a head-mounted augmented reality device). Under the guidance of established procedure plan displayed by HoloLens, localization marker was placed in operating room. Segmentectomy or wedge resection was subsequently performed. The primary endpoint was the localization procedure success rate, and the secondary endpoints were localization time, operation time, and complications.

Results

Localization was successful in seven of the ten procedures. Due to different reasons, failures were noted in three cases, after which immediate adjustments were made. In the successful cases, the LungBrella marker was positioned at a median of 5.8 mm (range, 0-10 mm) from the edge of the nodule. Median localization time was 9.4 min (range, 5-19 min), and median operation time was 172.9 min (range, 105-200 min). There were no complications during the entire process.

Conclusions

This exploratory study suggests that augmented reality navigation-guided pulmonary nodule localization is a safe and feasible technique (ClinicalTrials.gov identifier, NCT04211051).

Augmented Reality (AR) for Surgical Robotic and Autonomous Systems: State of the Art, Challenges, and Solutions

Despite the substantial progress achieved in the development and integration of augmented reality (AR) in surgical robotic and autonomous systems (RAS), the center of focus in most devices remains on improving end-effector dexterity and precision, as well as improved access to minimally invasive surgeries. This paper aims to provide a systematic review of different types of state-of-the-art surgical robotic platforms while identifying areas for technological improvement. We associate specific control features, such as haptic feedback, sensory stimuli, and human-robot collaboration, with AR technology to perform complex surgical interventions for increased user perception of the augmented world. Current researchers in the field have, for long, faced innumerable issues with low accuracy in tool placement around complex trajectories, pose estimation, and difficulty in depth perception during two-dimensional medical imaging. A number of robots described in this review, such as Novarad and SpineAssist, are analyzed in terms of their hardware features, computer vision systems (such as deep learning algorithms), and the clinical relevance of the literature. We attempt to outline the shortcomings in current optimization algorithms for surgical robots (such as YOLO and LTSM) whilst providing mitigating solutions to internal tool-to-organ collision detection and image reconstruction. The accuracy of results in robot end-effector collisions and reduced occlusion remain promising within the scope of our research, validating the propositions made for the surgical clearance of ever-expanding AR technology in the future.

Acquisition and usage of robotic surgical data for machine learning analysis

BACKGROUND

The increasing use of robot-assisted surgery (RAS) has led to the need for new methods of assessing whether new surgeons are qualified to perform RAS, without the resource-demanding process of having expert surgeons do the assessment. Computer-based automation and artificial intelligence (AI) are seen as promising alternatives to expert-based surgical assessment. However, no standard protocols or methods for preparing data and implementing AI are available for clinicians. This may be among the reasons for the impediment to the use of AI in the clinical setting.

METHOD

We tested our method on porcine models with both the da Vinci Si and the da Vinci Xi. We sought to capture raw video data from the surgical robots and 3D movement data from the surgeons and prepared the data for the use in AI by a structured guide to acquire and prepare video data using the following steps: 'Capturing image data from the surgical robot', 'Extracting event data', 'Capturing movement data of the surgeon', 'Annotation of image data'.

RESULTS

15 participant (11 novices and 4 experienced) performed 10 different intraabdominal RAS procedures. Using this method we captured 188 videos (94 from the surgical robot, and 94 corresponding movement videos of the surgeons' arms and hands). Event data, movement data, and labels were extracted from the raw material and prepared for use in AI.

CONCLUSION

With our described methods, we could collect, prepare, and annotate images, events, and motion data from surgical robotic systems in preparation for its use in AI.

Modern Surgical Techniques of Thyroidectomy and Advances in the Prevention and Treatment of Perioperative Complications

Thyroid cancer is the most common cancer of the endocrine system, and, in recent years, there has been a phenomenon of overdiagnosis followed by subsequent overtreatment. This results in an increasing number of thyroidectomy complications being faced in clinical practice. In this paper, we present the current state of knowledge and the latest findings in the fields of modern surgical techniques, thermal ablation, the identification and assessment of parathyroid function, recurrent laryngeal nerve monitoring and treatment and perioperative bleeding. We reviewed 485 papers, from which we selected 125 papers that are the most relevant. The main merit of this article is its comprehensive view of the subject under discussion-both general, concerning the selection of the appropriate method of surgery, and particular, concerning the selection of the appropriate method of prevention or treatment of selected perioperative complications.

Shifted-windows transformers for the detection of cerebral aneurysms in microsurgery

Purpose

Microsurgical Aneurysm Clipping Surgery (MACS) carries a high risk for intraoperative aneurysm rupture. Automated recognition of instances when the aneurysm is exposed in the surgical video would be a valuable reference point for neuronavigation, indicating phase transitioning and more importantly designating moments of high risk for rupture. This article introduces the MACS dataset containing 16 surgical videos with frame-level expert annotations and proposes a learning methodology for surgical scene understanding identifying video frames with the aneurysm present in the operating microscope’s field-of-view.

Methods

Despite the dataset imbalance (80% no presence, 20% presence) and developed without explicit annotations, we demonstrate the applicability of Transformer-based deep learning architectures (MACSSwin-T, vidMACSSwin-T) to detect the aneurysm and classify MACS frames accordingly. We evaluate the proposed models in multiple-fold cross-validation experiments with independent sets and in an unseen set of 15 images against 10 human experts (neurosurgeons).

Results

Average (across folds) accuracy of 80.8% (range 78.5–82.4%) and 87.1% (range 85.1–91.3%) is obtained for the image- and video-level approach, respectively, demonstrating that the models effectively learn the classification task. Qualitative evaluation of the models’ class activation maps shows these to be localized on the aneurysm’s actual location. Depending on the decision threshold, MACSWin-T achieves 66.7–86.7% accuracy in the unseen images, compared to 82% of human raters, with moderate to strong correlation.

Conclusions

Proposed architectures show robust performance and with an adjusted threshold promoting detection of the underrepresented (aneurysm presence) class, comparable to human expert accuracy. Our work represents the first step towards landmark detection in MACS with the aim to inform surgical teams to attend to high-risk moments, taking precautionary measures to avoid rupturing.

Augmented Reality in Surgery: A Scoping Review

Augmented reality (AR) is an innovative system that enhances the real world by superimposing virtual objects on reality. The aim of this study was to analyze the application of AR in medicine and which of its technical solutions are the most used. We carried out a scoping review of the articles published between 2019 and February 2022. The initial search yielded a total of 2649 articles. After applying filters, removing duplicates and screening, we included 34 articles in our analysis. The analysis of the articles highlighted that AR has been traditionally and mainly used in orthopedics in addition to maxillofacial surgery and oncology. Regarding the display application in AR, the Microsoft HoloLens Optical Viewer is the most used method. Moreover, for the tracking and registration phases, the marker-based method with a rigid registration remains the most used system. Overall, the results of this study suggested that AR is an innovative technology with numerous advantages, finding applications in several new surgery domains. Considering the available data, it is not possible to clearly identify all the fields of application and the best technologies regarding AR.

Augmented Reality-Based Visual Cue for Guiding Central Catheter Insertion in Pediatric Oncologic Patients

BACKGROUND

Pediatric hemato-oncologic patients require central catheters for chemotherapy, and the junction of the superior vena cava and right atrium is considered the ideal location for catheter tips. Skin landmarks or fluoroscopic supports have been applied to identify the cavoatrial junction; however, none has been recognized as the gold standard. Therefore, we aim to develop a safe and accurate technique using augmented reality technology for the location of the cavoatrial junction in pediatric hemato-oncologic patients.

METHODS

Fifteen oncology patients who underwent chest computed tomography were enrolled for Hickman catheter or chemoport insertion. With the aid of augmented reality technology, three-dimensional models of the internal jugular veins, external jugular veins, subclavian veins, superior vena cava, and right atrium were constructed. On inserting the central vein catheters, the cavoatrial junction identified using the three-dimensional models were marked on the body surface, the tip was positioned at the corresponding location, and the actual insertion location was confirmed using a portable x-ray machine. The proposed method was evaluated by comparing the distance from the cavoatrial junction to the augmented reality location with that to the conventional location on x-ray.

RESULTS

The mean distance between the cavoatrial junction and augmented reality location on x-ray was 1.2 cm, which was significantly shorter than that between the cavoatrial junction and conventional location (1.9 cm; P = 0.027).

CONCLUSIONS

Central catheter insertion using augmented reality technology is more safe and accurate than that using conventional methods and can be performed at no additional cost in oncology patients.

Effect of an anti-adhesion agent on vision-based assessment of cervical adhesions after thyroid surgery: randomized, placebo-controlled trial

Many patients experience cervical adhesions after thyroid surgery. To date, however, no studies have objectively measured the effects of anti-adhesion agents on cervical adhesion symptoms. This study evaluated the effects of an anti-adhesion agent on cervical adhesions after thyroid surgery, as determined using a system that measures the extent of marker movement objectively. One hundred patients were randomized in a 1:1 ratio to undergo thyroid surgery with or without the anti-adhesion agent Collabarrier. Using specially manufactured recording equipment, the position of the marker on neck skin was measured before surgery, and 2 weeks, 3 months, and 9 months after surgery. Relative change in marker distance, calculated by subtracting the marker position before surgery from the marker positions 2 weeks, 3 months, and 9 months after surgery, differed significantly in the groups of patients who underwent thyroid surgery with and without the anti-adhesion agent (P < 0.05). A novel measuring system can objectively evaluate the effectiveness of a thyroid anti-adhesion agent. The anti-adhesion agent used significantly reduced adhesions compared with the control group. The trial is registered at www.cris.nih.go.kr (KCT0005745; date of registration, 08/01/2021).

Using deep learning to identify the recurrent laryngeal nerve during thyroidectomy

Surgeons must visually distinguish soft-tissues, such as nerves, from surrounding anatomy to prevent complications and optimize patient outcomes. An accurate nerve segmentation and analysis tool could provide useful insight for surgical decision-making. Here, we present an end-to-end, automatic deep learning computer vision algorithm to segment and measure nerves. Unlike traditional medical imaging, our unconstrained setup with accessible handheld digital cameras, along with the unstructured open surgery scene, makes this task uniquely challenging. We investigate one common procedure, thyroidectomy, during which surgeons must avoid damaging the recurrent laryngeal nerve (RLN), which is responsible for human speech. We evaluate our segmentation algorithm on a diverse dataset across varied and challenging settings of operating room image capture, and show strong segmentation performance in the optimal image capture condition. This work lays the foundation for future research in real-time tissue discrimination and integration of accessible, intelligent tools into open surgery to provide actionable insights.

Feasibility of the application of mixed reality in mandible reconstruction with fibula flap: A cadaveric specimen study

BACKGROUND

In recent years, a new technology, mixed reality (MR), has emerged and surpassed the limitations of augmented reality (AR) with its inability to interact with hologram. This study aimed to investigate the feasibility of the application of MR in mandible reconstruction with fibula flap.

METHODS

Computed tomography (CT) examination was performed for one cadaveric mandible and ten fibula bones. Using professional software Proplan CMF 3.0 (Materialize, Leuven, Belgium), we created a defected mandibular model and simulated the reconstruction design with these 10 fibula bones. The surgical plans were transferred to the HoloLens. We used HoloLens to guide the osteotomy and shaping of the fibular bone. After fixing the fibular segments using the Ti template, all segments underwent a CT examination. Before and after objects were compared for measurements of the location of fibular osteotomies, angular deviation of fibular segments, and intergonial angle distances.

RESULTS

The mean location of the fibular osteotomies, angular deviation of the fibular segments, and intergonial angle distances were 2.11 ± 1.31 mm, 2.85°± 1.97°, and 7.24 ± 3.42 mm, respectively.

CONCLUSION

The experimental results revealed that slight deviations remained in the accuracy of fibular osteotomy. With the further development of technology, it has the potential to improve the efficiency and precision of the reconstructive surgery.

Evaluation of Surgical Skills during Robotic Surgery by Deep Learning-Based Multiple Surgical Instrument Tracking in Training and Actual Operations

As the number of robotic surgery procedures has increased, so has the importance of evaluating surgical skills in these techniques. It is difficult, however, to automatically and quantitatively evaluate surgical skills during robotic surgery, as these skills are primarily associated with the movement of surgical instruments. This study proposes a deep learning-based surgical instrument tracking algorithm to evaluate surgeons’ skills in performing procedures by robotic surgery. This method overcame two main drawbacks: occlusion and maintenance of the identity of the surgical instruments. In addition, surgical skill prediction models were developed using motion metrics calculated from the motion of the instruments. The tracking method was applied to 54 video segments and evaluated by root mean squared error (RMSE), area under the curve (AUC), and Pearson correlation analysis. The RMSE was 3.52 mm, the AUC of 1 mm, 2 mm, and 5 mm were 0.7, 0.78, and 0.86, respectively, and Pearson’s correlation coefficients were 0.9 on the x-axis and 0.87 on the y-axis. The surgical skill prediction models showed an accuracy of 83% with Objective Structured Assessment of Technical Skill (OSATS) and Global Evaluative Assessment of Robotic Surgery (GEARS). The proposed method was able to track instruments during robotic surgery, suggesting that the current method of surgical skill assessment by surgeons can be replaced by the proposed automatic and quantitative evaluation method.