Machine Vision for Real-Time Intraoperative Anatomic Guidance: A Proof-of-Concept Study in Endoscopic Pituitary Surgery

Artificial intelligence for automatic detection and segmentation of nasal polyposis: a pilot study

PURPOSE

Accurate diagnosis and quantification of polyps and symptoms are pivotal for planning the therapeutic strategy of Chronic rhinosinusitis with nasal polyposis (CRSwNP). This pilot study aimed to develop an artificial intelligence (AI)-based image analysis system capable of segmenting nasal polyps from nasal endoscopy videos.

METHODS

Recorded nasal videoendoscopies from 52 patients diagnosed with CRSwNP between 2019 and 2022 were retrospectively analyzed. Images extracted were manually segmented on the web application Roboflow. A dataset of 342 images was generated and divided into training (80%), validation (10%), and testing (10%) sets. The Ultralytics YOLOv8.0.28 model was employed for automated segmentation.

RESULTS

The YOLOv8s-seg model consisted of 195 layers and required 42.4 GFLOPs for operation. When tested against the validation set, the algorithm achieved a precision of 0.91, recall of 0.839, and mean average precision at 50% IoU (mAP50) of 0.949. For the segmentation task, similar metrics were observed, including a mAP ranging from 0.675 to 0.679 for IoUs between 50% and 95%.

CONCLUSIONS

The study shows that a carefully trained AI algorithm can effectively identify and delineate nasal polyps in patients with CRSwNP. Despite certain limitations like the focus on CRSwNP-specific samples, the algorithm presents a promising complementary tool to existing diagnostic methods.

PitSurgRT: real-time localization of critical anatomical structures in endoscopic pituitary surgery

PURPOSE

Endoscopic pituitary surgery entails navigating through the nasal cavity and sphenoid sinus to access the sella using an endoscope. This procedure is intricate due to the proximity of crucial anatomical structures (e.g. carotid arteries and optic nerves) to pituitary tumours, and any unintended damage can lead to severe complications including blindness and death. Intraoperative guidance during this surgery could support improved localization of the critical structures leading to reducing the risk of complications.

METHODS

A deep learning network PitSurgRT is proposed for real-time localization of critical structures in endoscopic pituitary surgery. The network uses high-resolution net (HRNet) as a backbone with a multi-head for jointly localizing critical anatomical structures while segmenting larger structures simultaneously. Moreover, the trained model is optimized and accelerated by using TensorRT. Finally, the model predictions are shown to neurosurgeons, to test their guidance capabilities.

RESULTS

Compared with the state-of-the-art method, our model significantly reduces the mean error in landmark detection of the critical structures from 138.76 to 54.40 pixels in a 1280 × 720-pixel image. Furthermore, the semantic segmentation of the most critical structure, sella, is improved by 4.39% IoU. The inference speed of the accelerated model achieves 298 frames per second with floating-point-16 precision. In the study of 15 neurosurgeons, 88.67% of predictions are considered accurate enough for real-time guidance.

CONCLUSION

The results from the quantitative evaluation, real-time acceleration, and neurosurgeon study demonstrate the proposed method is highly promising in providing real-time intraoperative guidance of the critical anatomical structures in endoscopic pituitary surgery.

Current status of artificial intelligence technologies in pituitary adenoma surgery: a scoping review

PURPOSE

Pituitary adenoma surgery is a complex procedure due to critical adjacent neurovascular structures, variations in size and extensions of the lesions, and potential hormonal imbalances. The integration of artificial intelligence (AI) and machine learning (ML) has demonstrated considerable potential in assisting neurosurgeons in decision-making, optimizing surgical outcomes, and providing real-time feedback. This scoping review comprehensively summarizes the current status of AI/ML technologies in pituitary adenoma surgery, highlighting their strengths and limitations.

METHODS

PubMed, Embase, Web of Science, and Scopus were searched following the PRISMA-ScR guidelines. Studies discussing the use of AI/ML in pituitary adenoma surgery were included. Eligible studies were grouped to analyze the different outcomes of interest of current AI/ML technologies.

RESULTS

Among the 2438 identified articles, 44 studies met the inclusion criteria, with a total of seventeen different algorithms utilized across all studies. Studies were divided into two groups based on their input type: clinicopathological and imaging input. The four main outcome variables evaluated in the studies included: outcome (remission, recurrence or progression, gross-total resection, vision improvement, and hormonal recovery), complications (CSF leak, readmission, hyponatremia, and hypopituitarism), cost, and adenoma-related factors (aggressiveness, consistency, and Ki-67 labeling) prediction. Three studies focusing on workflow analysis and real-time navigation were discussed separately.

CONCLUSION

AI/ML modeling holds promise for improving pituitary adenoma surgery by enhancing preoperative planning and optimizing surgical strategies. However, addressing challenges such as algorithm selection, performance evaluation, data heterogeneity, and ethics is essential to establish robust and reliable ML models that can revolutionize neurosurgical practice and benefit patients.

Simulated outcomes for durotomy repair in minimally invasive spine surgery

Minimally invasive spine surgery (MISS) is increasingly performed using endoscopic and microscopic visualization, and the captured video can be used for surgical education and development of predictive artificial intelligence (AI) models. Video datasets depicting adverse event management are also valuable, as predictive models not exposed to adverse events may exhibit poor performance when these occur. Given that no dedicated spine surgery video datasets for AI model development are publicly available, we introduce Simulated Outcomes for Durotomy Repair in Minimally Invasive Spine Surgery (SOSpine). A validated MISS cadaveric dural repair simulator was used to educate neurosurgery residents, and surgical microscope video recordings were paired with outcome data. Objects including durotomy, needle, grasper, needle driver, and nerve hook were then annotated. Altogether, SOSpine contains 15,698 frames with 53,238 annotations and associated durotomy repair outcomes. For validation, an AI model was fine-tuned on SOSpine video and detected surgical instruments with a mean average precision of 0.77. In summary, SOSpine depicts spine surgeons managing a common complication, providing opportunities to develop surgical AI models.

Computer-vision based analysis of the neurosurgical scene - A systematic review

Introduction

With increasing use of robotic surgical adjuncts, artificial intelligence and augmented reality in neurosurgery, the automated analysis of digital images and videos acquired over various procedures becomes a subject of increased interest. While several computer vision (CV) methods have been developed and implemented for analyzing surgical scenes, few studies have been dedicated to neurosurgery.

Research question

In this work, we present a systematic literature review focusing on CV methodologies specifically applied to the analysis of neurosurgical procedures based on intra-operative images and videos. Additionally, we provide recommendations for the future developments of CV models in neurosurgery.

Material and methods

We conducted a systematic literature search in multiple databases until January 17, 2023, including Web of Science, PubMed, IEEE Xplore, Embase, and SpringerLink.

Results

We identified 17 studies employing CV algorithms on neurosurgical videos/images. The most common applications of CV were tool and neuroanatomical structure detection or characterization, and to a lesser extent, surgical workflow analysis. Convolutional neural networks (CNN) were the most frequently utilized architecture for CV models (65%), demonstrating superior performances in tool detection and segmentation. In particular, mask recurrent-CNN manifested most robust performance outcomes across different modalities.

Discussion and conclusion

Our systematic review demonstrates that CV models have been reported that can effectively detect and differentiate tools, surgical phases, neuroanatomical structures, as well as critical events in complex neurosurgical scenes with accuracies above 95%. Automated tool recognition contributes to objective characterization and assessment of surgical performance, with potential applications in neurosurgical training and intra-operative safety management.

Deep learning based identification of pituitary adenoma on surgical endoscopic images: a pilot study

Accurate tumor identification during surgical excision is necessary for neurosurgeons to determine the extent of resection without damaging the surrounding tissues. No conventional technologies have achieved reliable performance for pituitary adenomas. This study proposes a deep learning approach using intraoperative endoscopic images to discriminate pituitary adenomas from non-tumorous tissue inside the sella turcica. Static images were extracted from 50 intraoperative videos of patients with pituitary adenomas. All patients underwent endoscopic transsphenoidal surgery with a 4 K ultrahigh-definition endoscope. The tumor and non-tumorous tissue within the sella turcica were delineated on static images. Using intraoperative images, we developed and validated deep learning models to identify tumorous tissue. Model performance was evaluated using a fivefold per-patient methodology. As a proof-of-concept, the model's predictions were pathologically cross-referenced with a medical professional's diagnosis using the intraoperative images of a prospectively enrolled patient. In total, 605 static images were obtained. Among the cropped 117,223 patches, 58,088 were labeled as tumors, while the remaining 59,135 were labeled as non-tumorous tissues. The evaluation of the image dataset revealed that the wide-ResNet model had the highest accuracy of 0.768, with an F1 score of 0.766. A preliminary evaluation on one patient indicated alignment between the ground truth set by neurosurgeons, the model's predictions, and histopathological findings. Our deep learning algorithm has a positive tumor discrimination performance in intraoperative 4-K endoscopic images in patients with pituitary adenomas.

Deep-learning-based semantic segmentation of autonomic nerves from laparoscopic images of colorectal surgery: an experimental pilot study

BACKGROUND

The preservation of autonomic nerves is the most important factor in maintaining genitourinary function in colorectal surgery; however, these nerves are not clearly recognisable, and their identification is strongly affected by the surgical ability. Therefore, this study aimed to develop a deep learning model for the semantic segmentation of autonomic nerves during laparoscopic colorectal surgery and to experimentally verify the model through intraoperative use and pathological examination.

MATERIALS AND METHODS

The annotation data set comprised videos of laparoscopic colorectal surgery. The images of the hypogastric nerve (HGN) and superior hypogastric plexus (SHP) were manually annotated under a surgeon's supervision. The Dice coefficient was used to quantify the model performance after five-fold cross-validation. The model was used in actual surgeries to compare the recognition timing of the model with that of surgeons, and pathological examination was performed to confirm whether the samples labelled by the model from the colorectal branches of the HGN and SHP were nerves.

RESULTS

The data set comprised 12 978 video frames of the HGN from 245 videos and 5198 frames of the SHP from 44 videos. The mean (±SD) Dice coefficients of the HGN and SHP were 0.56 (±0.03) and 0.49 (±0.07), respectively. The proposed model was used in 12 surgeries, and it recognised the right HGN earlier than the surgeons did in 50.0% of the cases, the left HGN earlier in 41.7% of the cases and the SHP earlier in 50.0% of the cases. Pathological examination confirmed that all 11 samples were nerve tissue.

CONCLUSION

An approach for the deep-learning-based semantic segmentation of autonomic nerves was developed and experimentally validated. This model may facilitate intraoperative recognition during laparoscopic colorectal surgery.

Physicians’ attitudes and knowledge toward artificial intelligence in medicine: Benefits and drawbacks

The use of artificial intelligence (AI) in the medical field is increasing and is expected to shape future clinical practice and job security. Therefore, this study aimed to assess the opinions and attitudes of practicing physicians in Bahrain regarding the benefits and drawbacks of AI for their future daily practice. A cross-sectional survey of practicing physicians with a minimum of five years' experience across the main secondary and tertiary care hospitals in Bahrain was conducted. An online questionnaire was used to collect data on demographics, knowledge of AI, attitudes towards the use of AI in 10 tasks of daily clinical practice, and opinions on the benefits and drawbacks of AI. A total of 114 physicians participated in the survey. Among them, 43 (37.7%) were registered psychiatrists, 15 (13.2%) were pathologists, 17 (14.9%) were radiologists, and 39 (34.2%) were surgical specialists. The participants' attitudes were overall positive towards AI. Pathologists were particularly in favor of using AI to "Formulate personalized medication and/or treatment plans for patients" and to "Interview patients in a range of settings to obtain medical history." Most participants agreed that AI would reduce the time needed to establish a diagnosis and negatively affect employment rates. There were no correlations between the responses and the participants' age, gender, years of experience, or AI knowledge. This study demonstrates that the attitudes towards the use of AI in medicine among practicing physicians in Bahrain are similar to those of physicians in developed countries in that they are positive and welcoming of AI implementation in practice. However, the potential effects of AI on job security are a major concern.

Expert surgeons and deep learning models can predict the outcome of surgical hemorrhage from 1 min of video

Major vascular injury resulting in uncontrolled bleeding is a catastrophic and often fatal complication of minimally invasive surgery. At the outset of these events, surgeons do not know how much blood will be lost or whether they will successfully control the hemorrhage (achieve hemostasis). We evaluate the ability of a deep learning neural network (DNN) to predict hemostasis control ability using the first minute of surgical video and compare model performance with human experts viewing the same video. The publicly available SOCAL dataset contains 147 videos of attending and resident surgeons managing hemorrhage in a validated, high-fidelity cadaveric simulator. Videos are labeled with outcome and blood loss (mL). The first minute of 20 videos was shown to four, blinded, fellowship trained skull-base neurosurgery instructors, and to SOCALNet (a DNN trained on SOCAL videos). SOCALNet architecture included a convolutional network (ResNet) identifying spatial features and a recurrent network identifying temporal features (LSTM). Experts independently assessed surgeon skill, predicted outcome and blood loss (mL). Outcome and blood loss predictions were compared with SOCALNet. Expert inter-rater reliability was 0.95. Experts correctly predicted 14/20 trials (Sensitivity: 82%, Specificity: 55%, Positive Predictive Value (PPV): 69%, Negative Predictive Value (NPV): 71%). SOCALNet correctly predicted 17/20 trials (Sensitivity 100%, Specificity 66%, PPV 79%, NPV 100%) and correctly identified all successful attempts. Expert predictions of the highest and lowest skill surgeons and expert predictions reported with maximum confidence were more accurate. Experts systematically underestimated blood loss (mean error - 131 mL, RMSE 350 mL, R² 0.70) and fewer than half of expert predictions identified blood loss > 500 mL (47.5%, 19/40). SOCALNet had superior performance (mean error - 57 mL, RMSE 295 mL, R² 0.74) and detected most episodes of blood loss > 500 mL (80%, 8/10). In validation experiments, SOCALNet evaluation of a critical on-screen surgical maneuver and high/low-skill composite videos were concordant with expert evaluation. Using only the first minute of video, experts and SOCALNet can predict outcome and blood loss during surgical hemorrhage. Experts systematically underestimated blood loss, and SOCALNet had no false negatives. DNNs can provide accurate, meaningful assessments of surgical video. We call for the creation of datasets of surgical adverse events for quality improvement research.

Utility of the Simulated Outcomes Following Carotid Artery Laceration Video Data Set for Machine Learning Applications

IMPORTANCE

Surgical data scientists lack video data sets that depict adverse events, which may affect model generalizability and introduce bias. Hemorrhage may be particularly challenging for computer vision-based models because blood obscures the scene.

OBJECTIVE

To assess the utility of the Simulated Outcomes Following Carotid Artery Laceration (SOCAL)-a publicly available surgical video data set of hemorrhage complication management with instrument annotations and task outcomes-to provide benchmarks for surgical data science techniques, including computer vision instrument detection, instrument use metrics and outcome associations, and validation of a SOCAL-trained neural network using real operative video.

DESIGN, SETTING, AND PARTICIPANTS

For this quailty improvement study, a total of 75 surgeons with 1 to 30 years' experience (mean, 7 years) were filmed from January 1, 2017, to December 31, 2020, managing catastrophic surgical hemorrhage in a high-fidelity cadaveric training exercise at nationwide training courses. Videos were annotated from January 1 to June 30, 2021.

INTERVENTIONS

Surgeons received expert coaching between 2 trials.

MAIN OUTCOMES AND MEASURES

Hemostasis within 5 minutes (task success, dichotomous), time to hemostasis (in seconds), and blood loss (in milliliters) were recorded. Deep neural networks (DNNs) were trained to detect surgical instruments in view. Model performance was measured using mean average precision (mAP), sensitivity, and positive predictive value.

RESULTS

SOCAL contains 31 443 frames with 65 071 surgical instrument annotations from 147 trials with associated surgeon demographic characteristics, time to hemostasis, and recorded blood loss for each trial. Computer vision-based instrument detection methods using DNNs trained on SOCAL achieved a mAP of 0.67 overall and 0.91 for the most common surgical instrument (suction). Hemorrhage control challenges standard object detectors: detection of some surgical instruments remained poor (mAP, 0.25). On real intraoperative video, the model achieved a sensitivity of 0.77 and a positive predictive value of 0.96. Instrument use metrics derived from the SOCAL video were significantly associated with performance (blood loss).

CONCLUSIONS AND RELEVANCE

Hemorrhage control is a high-stakes adverse event that poses unique challenges for video analysis, but no data sets of hemorrhage control exist. The use of SOCAL, the first data set to depict hemorrhage control, allows the benchmarking of data science applications, including object detection, performance metric development, and identification of metrics associated with outcomes. In the future, SOCAL may be used to build and validate surgical data science models.

Robotic and robot-assisted skull base neurosurgery: systematic review of current applications and future directions

OBJECTIVE

The utility of robotic instrumentation is expanding in neurosurgery. Despite this, successful examples of robotic implementation for endoscopic endonasal or skull base neurosurgery remain limited. Therefore, the authors performed a systematic review of the literature to identify all articles that used robotic systems to access the sella or anterior, middle, or posterior cranial fossae.

METHODS

A systematic review of MEDLINE and PubMed in accordance with PRISMA guidelines performed for articles published between January 1, 1990, and August 1, 2021, was conducted to identify all robotic systems (autonomous, semiautonomous, or surgeon-controlled) used for skull base neurosurgical procedures. Cadaveric and human clinical studies were included. Studies with exclusively otorhinolaryngological applications or using robotic microscopes were excluded.

RESULTS

A total of 561 studies were identified from the initial search, of which 22 were included following full-text review. Transoral robotic surgery (TORS) using the da Vinci Surgical System was the most widely reported system (4 studies) utilized for skull base and pituitary fossa procedures; additionally, it has been reported for resection of sellar masses in 4 patients. Seven cadaveric studies used the da Vinci Surgical System to access the skull base using alternative, non-TORS approaches (e.g., transnasal, transmaxillary, and supraorbital). Five cadaveric studies investigated alternative systems to access the skull base. Six studies investigated the use of robotic endoscope holders. Advantages to robotic applications in skull base neurosurgery included improved lighting and 3D visualization, replication of more traditional gesture-based movements, and the ability for dexterous movements ordinarily constrained by small operative corridors. Limitations included the size and angulation capacity of the robot, lack of drilling components preventing fully robotic procedures, and cost. Robotic endoscope holders may have been particularly advantageous when the use of a surgical assistant or second surgeon was limited.

CONCLUSIONS

Robotic skull base neurosurgery has been growing in popularity and feasibility, but significant limitations remain. While robotic systems seem to have allowed for greater maneuverability and 3D visualization, their size and lack of neurosurgery-specific tools have continued to prevent widespread adoption into current practice. The next generation of robotic technologies should prioritize overcoming these limitations.