Intraoperative stent segmentation in X-ray fluoroscopy for endovascular aortic repair

Pioneering artificial intelligence-based real time assistance for intracranial liquid embolization in humans: an initial experience

BACKGROUND

Liquid embolization in neuroendovascular procedures carries the risk of embolizing an inappropriate vessel. Operators must pay close attention to multiple vessels during the procedure to avoid ischemic complications. We report our experience with real time artificial intelligence (AI) assisted liquid embolization and evaluate its performance.

METHODS

An AI-based system (Neuro-Vascular Assist, iMed technologies, Tokyo, Japan) was used in eight endovascular liquid embolization procedures in two institutions. The software automatically detects liquid embolic agent on biplane fluoroscopy images in real time and notifies operators when the agent reaches a predefined area. Safety, efficacy, and accuracy of the notifications were evaluated using recorded videos.

RESULTS

Onyx or n-butyl-2-cyanoacrylate (NBCA) was used in the treatment of arteriovenous malformation, dural arteriovenous fistula, meningioma, and chronic subdural hematoma. The mean number of true positive and false negative notifications per case was 31.8 and 2.8, respectively. No false positive notifications occurred. The precision and recall of the notifications were 100% and 92.0%, respectively. In 28.3% of the true positive notifications, the operator immediately paused agent injection after receiving the notification, which demonstrates the potential effectiveness of the AI-based system. No adverse events were associated with the notifications.

CONCLUSIONS

To the best of our knowledge, this is the first report of real time AI assistance with liquid embolization procedures in humans. The system demonstrated high notification accuracy, safety, and potential clinical usefulness in liquid embolization procedures. Further research is warranted to validate its impact on clinical outcomes. AI-based real time surgical support has the potential to advance neuroendovascular treatment.

First-in-human, real-time artificial intelligence assisted cerebral aneurysm coiling: a preliminary experience

BACKGROUND

Neuroendovascular procedures require careful and simultaneous attention to multiple devices on multiple screens. Overlooking unintended device movements can result in complications. Advancements in artificial intelligence (AI) have enabled real-time notifications of device movements during procedures. We report our preliminary experience with real-time AI-assisted cerebral aneurysm coiling in humans.

METHODS

A real-time AI-assistance software (Neuro-Vascular Assist, iMed technologies, Tokyo, Japan) was used during coil embolization procedures in nine patients with an unruptured aneurysm. The AI system provided real-time notifications for 'coil marker approaching', 'guidewire movement', and 'device entry' on biplane fluoroscopic images. The efficacy, accuracy, and safety of the notifications were evaluated using video recordings.

RESULTS

The AI system functioned properly in all cases. The mean number of notifications for coil marker approaching, guidewire movement, and device entry per procedure was 20.0, 3.0, and 18.3, respectively. The overall precision and recall were 92.7% and 97.2%, respectively. Five of 26 true positive guidewire notifications (19%) resulted in adjustment of the guidewire back toward its original position, indicating the potential effectiveness of the AI system. No adverse events occurred.

CONCLUSIONS

The software was sufficiently accurate and safe in this preliminary study, suggesting its potential usefulness. To the best of our knowledge, this is the first reported use of a real-time AI system for assisting cerebral aneurysm coiling in humans. Large scale studies are warranted to validate its effectiveness. Real-time AI assistance has significant potential for future neuroendovascular therapy.

Deep Learning-Based Intraoperative Stent Graft Segmentation on Completion Digital Subtraction Angiography During Endovascular Aneurysm Repair

PURPOSE

Modern endovascular hybrid operating rooms generate large amounts of medical images during a procedure, which are currently mostly assessed by eye. In this paper, we present fully automatic segmentation of the stent graft on the completion digital subtraction angiography during endovascular aneurysm repair, utilizing a deep learning network.

TECHNIQUE

Completion digital subtraction angiographies (cDSAs) of 47 patients treated for an infrarenal aortic aneurysm using EVAR were collected retrospectively. A two-dimensional convolutional neural network (CNN) with a U-Net architecture was trained for segmentation of the stent graft from the completion angiographies. The cross-validation resulted in an average Dice similarity score of 0.957 ± 0.041 and median of 0.968 (IQR: 0.950 - 0.976). The mean and median of the average surface distance are 1.266 ± 1.506 mm and 0.870 mm (IQR: 0.490 - 1.430), respectively.

CONCLUSION

We developed a fully automatic stent graft segmentation method based on the completion digital subtraction angiography during EVAR, utilizing a deep learning network. This can provide the platform for the development of intraoperative analytical applications in the endovascular hybrid operating room such as stent graft deployment accuracy, endoleak visualization, and image fusion correction.

Prediction of guidewire-induced aortic deformations during EVAR: a finite element and in vitro study

Introduction and aims: During an Endovascular Aneurysm Repair (EVAR) procedure a stiff guidewire is inserted from the iliac arteries. This induces significant deformations on the vasculature, thus, affecting the pre-operative planning, and the accuracy of image fusion. The aim of the present work is to predict the guidewire induced deformations using a finite element approach validated through experiments with patient-specific additive manufactured models. The numerical approach herein developed could improve the pre-operative planning and the intra-operative navigation. Material and methods: The physical models used for the experiments in the hybrid operating room, were manufactured from the segmentations of pre-operative Computed Tomography (CT) angiographies. The finite element analyses (FEA) were performed with LS-DYNA Explicit. The material properties used in finite element analyses were obtained by uniaxial tensile tests. The experimental deformed configurations of the aorta were compared to those obtained from FEA. Three models, obtained from Computed Tomography acquisitions, were investigated in the present work: A) without intraluminal thrombus (ILT), B) with ILT, C) with ILT and calcifications. Results and discussion: A good agreement was found between the experimental and the computational studies. The average error between the final in vitro vs. in silico aortic configurations, i.e., when the guidewire is fully inserted, are equal to 1.17, 1.22 and 1.40 mm, respectively, for Models A, B and C. The increasing trend in values of deformations from Model A to Model C was noticed both experimentally and numerically. The presented validated computational approach in combination with a tracking technology of the endovascular devices may be used to obtain the intra-operative configuration of the vessels and devices prior to the procedure, thus limiting the radiation exposure and the contrast agent dose.

Comparing Apparent Diffusion Coefficient and FNCLCC Grading to Improve Pretreatment Grading of Soft Tissue Sarcoma-A Translational Feasibility Study on Fusion Imaging

Simple Summary

Histological subtype and grading are essential for the planning of soft tissue sarcoma. Pretherapeutic grading based on core needle biopsies is frequently not reliable due to intratumoral heterogeneity. This pilot study assessed the ability of functional radiological imaging to improve histopathological grading. Multiple biopsies were taken from the sarcoma specimens during tumor resection and radiopaque markers were placed. Subsequently, fusion of preoperative magnetic resonance imaging and postoperative computed tomography of the specimen allowed for comparison of histopathological grading and diffusion-weighted imaging. The apparent diffusion coefficient appears to correlate with FNCLCC criteria and may supplement pretreatment assessment and multimodal treatment allocation in soft tissue sarcoma.

Abstract

Histological subtype and grading are cornerstones of treatment decisions in soft tissue sarcoma (STS). Due to intratumoral heterogeneity, pretreatment grading assessment is frequently unreliable and may be improved through functional imaging. In this pilot study, 12 patients with histologically confirmed STS were included. Preoperative functional magnetic resonance imaging was fused with a computed tomography scan of the resected specimen after collecting core needle biopsies and placing radiopaque markers at distinct tumor sites. The Fédération Nationale des Centres de Lutte Contre le Cancer (FNCLCC) grading criteria of the biopsies and apparent diffusion coefficients (ADCs) of the biopsy sites were correlated. Concordance in grading between the specimen and at least one biopsy was achieved in 9 of 11 cases (81.8%). In 7 of 12 cases, fusion imaging was feasible without relevant contour deviation. Functional analysis revealed a tendency for high-grade regions (Grade 2/3 (G2/G3)) (median (range) ± standard deviation: 1.13 (0.78–1.70) ± 0.23 × 10⁻³ mm²/s) to have lower ADC values than low-grade regions (G1; 1.43 (0.64–2.03) ± 0.46 × 10⁻³ mm²/s). In addition, FNCLCC scoring of multiple tumor biopsies proved intratumoral heterogeneity as expected. The ADC appears to correlate with the FNCLCC grading criteria. Further studies are needed to determine whether functional imaging may supplement histopathological grading.

Recursive Centerline- and Direction-Aware Joint Learning Network with Ensemble Strategy for Vessel Segmentation in X-ray Angiography Images

BACKGROUND AND OBJECTIVE

Automatic vessel segmentation from X-ray angiography images is an important research topic for the diagnosis and treatment of cardiovascular disease. The main challenge is how to extract continuous and completed vessel structures from XRA images with poor quality and high complexity. Most existing methods predominantly focus on pixel-wise segmentation and overlook the geometric features, resulting in breaking and absence in segmentation results. To improve the completeness and accuracy of vessel segmentation, we propose a recursive joint learning network embedded with geometric features.

METHODS

The network joins the centerline- and direction-aware auxiliary tasks with the primary task of segmentation, which guides the network to explore the geometric features of vessel connectivity. Moreover, the recursive learning strategy is designed by passing the previous segmentation result into the same network iteratively to improve segmentation. To further enhance connectivity, we present a complementary-task ensemble strategy by fusing the outputs of the three tasks for the final segmentation result with majority voting.

RESULTS

To validate the effectiveness of our method, we conduct qualitative and quantitative experiments on the XRA images of the coronary artery and aorta including aortic arch, thoracic aorta, and abdominal aorta. Our method achieves F₁ scores of 85.61±3.48% for the coronary artery, 89.02±2.89% for the aortic arch, 88.22±3.33% for the thoracic aorta, and 83.12±4.61% for the abdominal aorta.

CONCLUSIONS

Compared with six state-of-the-art methods, our method shows the most complete and accurate vessel segmentation results.

Synthesize and Segment: Towards Improved Catheter Segmentation via Adversarial Augmentation

Automatic catheter and guidewire segmentation plays an important role in robot-assisted interventions that are guided by fluoroscopy. Existing learning based methods addressing the task of segmentation or tracking are often limited by the scarcity of annotated samples and difficulty in data collection. In the case of deep learning based methods, the demand for large amounts of labeled data further impedes successful application. We propose a synthesize and segment approach with plug in possibilities for segmentation to address this. We show that an adversarially learned image-to-image translation network can synthesize catheters in X-ray fluoroscopy enabling data augmentation in order to alleviate a low data regime. To make realistic synthesized images, we train the translation network via a perceptual loss coupled with similarity constraints. Then existing segmentation networks are used to learn accurate localization of catheters in a semi-supervised setting with the generated images. The empirical results on collected medical datasets show the value of our approach with significant improvements over existing translation baseline methods.

Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration

PURPOSE

Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. In this paper, we propose a method for fully automatic registration using anatomical annotations produced by a neural network.

METHODS

Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data are obtained using a computationally intensive, intraoperatively incompatible, 2D/3D registration of the pelvis and each femur. Ground truth 2D segmentation labels and anatomical landmark locations are established using projected 3D annotations. Intraoperative registration couples a traditional intensity-based strategy with annotations inferred by the network and requires no human assistance.

RESULTS

Ground truth segmentation labels and anatomical landmarks were obtained in 366 fluoroscopic images across 6 cadaveric specimens. In a leave-one-subject-out experiment, networks trained on these data obtained mean dice coefficients for left and right hemipelves, left and right femurs of 0.86, 0.87, 0.90, and 0.84, respectively. The mean 2D landmark localization error was 5.0 mm. The pelvis was registered within [Formula: see text] for 86% of the images when using the proposed intraoperative approach with an average runtime of 7 s. In comparison, an intensity-only approach without manual initialization registered the pelvis to [Formula: see text] in 18% of images.

CONCLUSIONS

We have created the first accurately annotated, non-synthetic, dataset of hip fluoroscopy. By using these annotations as training data for neural networks, state-of-the-art performance in fluoroscopic segmentation and landmark localization was achieved. Integrating these annotations allows for a robust, fully automatic, and efficient intraoperative registration during fluoroscopic navigation of the hip.

Fully automatic estimation of pelvic sagittal inclination from anterior-posterior radiography image using deep learning framework

BACKGROUND AND OBJECTIVE

Malposition of the acetabular component causes dislocation and prosthetic impingement after Total Hip Arthroplasty (THA), which significantly affects the postoperative quality of life and implant longevity. The position of the acetabular component is determined by the Pelvic Sagittal Inclination (PSI), which not only varies among different people but also changes in different positions. It is important to recognize individual dynamic changes of the PSI for patient-specific planning of the THA. Previously PSI was estimated by registering the CT and radiography images. In this study, we introduce a new method for accurate estimation of functional PSI without requiring CT image in order to lower radiation exposure of the patient which opens up the possibility of increasing its application in a larger number of hospitals where CT is not acquired as a routine protocol.

METHODS

The proposed method consists of two main steps: First, the Mask R-CNN framework was employed to segment the pelvic shape from the background in the radiography images. Then, following the segmentation network, another convolutional network regressed the PSI angle. We employed a transfer learning paradigm where the network weights were initialized by non-medical images followed by fine-tuning using radiography images. Furthermore, in the training process, augmented data was generated to improve the performance of both networks. We analyzed the role of segmentation network in our system and investigated the Mask R-CNN performance in comparison with the U-Net, which is commonly used for the medical image segmentation.

RESULTS

In this study, the Mask R-CNN utilizing multi-task learning, transfer learning, and data augmentation techniques achieve 0.960 ± 0.008 DICE coefficient, which significantly outperforms the U-Net. The cascaded system is capable of estimating the PSI with 4.04° ± 3.39° error for the radiography images.

CONCLUSIONS

The proposed framework suggests a fully automatic and robust estimation of the PSI using only an anterior-posterior radiography image.

Simultaneous reconstruction of multiple stiff wires from a single X-ray projection for endovascular aortic repair

PURPOSE

Endovascular repair of aortic aneurysms (EVAR) can be supported by fusing pre- and intraoperative data to allow for improved navigation and to reduce the amount of contrast agent needed during the intervention. However, stiff wires and delivery devices can deform the vasculature severely, which reduces the accuracy of the fusion. Knowledge about the 3D position of the inserted instruments can help to transfer these deformations to the preoperative information.

METHOD

We propose a method to simultaneously reconstruct the stiff wires in both iliac arteries based on only a single monoplane acquisition, thereby avoiding interference with the clinical workflow. In the available X-ray projection, the 2D course of the wire is extracted. Then, a virtual second view of each wire orthogonal to the real projection is estimated using the preoperative vessel anatomy from a computed tomography angiography as prior information. Based on the real and virtual 2D wire courses, the wires can then be reconstructed in 3D using epipolar geometry.

RESULTS

We achieve a mean modified Hausdorff distance of 4.2 mm between the estimated 3D position and the true wire course for the contralateral side and 4.5 mm for the ipsilateral side.

CONCLUSION

The accuracy and speed of the proposed method allow for use in an intraoperative setting of deformation correction for EVAR.