Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation

Exploring semantic consistency in unpaired image translation to generate data for surgical applications

PURPOSE

In surgical computer vision applications, data privacy and expert annotation challenges impede the acquisition of labeled training data. Unpaired image-to-image translation techniques have been explored to automatically generate annotated datasets by translating synthetic images into a realistic domain. The preservation of structure and semantic consistency, i.e., per-class distribution during translation, poses a significant challenge, particularly in cases of semantic distributional mismatch.

METHOD

This study empirically investigates various translation methods for generating data in surgical applications, explicitly focusing on semantic consistency. Through our analysis, we introduce a novel and simple combination of effective approaches, which we call ConStructS. The defined losses within this approach operate on multiple image patches and spatial resolutions during translation.

RESULTS

Various state-of-the-art models were extensively evaluated on two challenging surgical datasets. With two different evaluation schemes, the semantic consistency and the usefulness of the translated images on downstream semantic segmentation tasks were evaluated. The results demonstrate the effectiveness of the ConStructS method in minimizing semantic distortion, with images generated by this model showing superior utility for downstream training.

CONCLUSION

In this study, we tackle semantic inconsistency in unpaired image translation for surgical applications with minimal labeled data. The simple model (ConStructS) enhances consistency during translation and serves as a practical way of generating fully labeled and semantically consistent datasets at minimal cost. Our code is available at https://gitlab.com/nct_tso_public/constructs .

Minimal data requirement for realistic endoscopic image generation with Stable Diffusion

PURPOSE

Computer-assisted surgical systems provide support information to the surgeon, which can improve the execution and overall outcome of the procedure. These systems are based on deep learning models that are trained on complex and challenging-to-annotate data. Generating synthetic data can overcome these limitations, but it is necessary to reduce the domain gap between real and synthetic data.

METHODS

We propose a method for image-to-image translation based on a Stable Diffusion model, which generates realistic images starting from synthetic data. Compared to previous works, the proposed method is better suited for clinical application as it requires a much smaller amount of input data and allows finer control over the generation of details by introducing different variants of supporting control networks.

RESULTS

The proposed method is applied in the context of laparoscopic cholecystectomy, using synthetic and real data from public datasets. It achieves a mean Intersection over Union of 69.76%, significantly improving the baseline results (69.76 vs. 42.21%).

CONCLUSIONS

The proposed method for translating synthetic images into images with realistic characteristics will enable the training of deep learning methods that can generalize optimally to real-world contexts, thereby improving computer-assisted intervention guidance systems.

Surgical training of minimally invasive mitral valve repair on a patient-specific simulator improves surgical skills

OBJECTIVES

Minimally invasive mitral valve repair (MVR) is considered one of the most challenging operations in cardiac surgery and requires much practice and experience. Simulation-based surgical training might be a method to support the learning process and help to flatten the steep learning curve of novices. The purpose of this study was to show the possible effects on learning of surgical training using a high-fidelity simulator with patient-specific mitral valve replicas.

METHODS

Twenty-five participants were recruited to perform MVR on anatomically realistic valve models during different training sessions. After every session their performance was evaluated by a surgical expert regarding accuracy and duration for each step. A second blinded rater similarly assessed the performance after the study. Through repeated documentation of those parameters, their progress in learning was analysed, and gains in proficiency were evaluated.

RESULTS

Participants showed significant performance enhancements in terms of both accuracy and time. Their surgical skills showed sizeable improvements after only 1 session. For example, the time to implant neo-chordae decreased by 24.64% (354 s-264 s, P < 0.001) and the time for annuloplasty by 4.01% (54 s-50 s, P = 0.165), whereas the number of irregular stitches for annuloplasty decreased from 52% to 24%.The significance of simulation-based surgical training as a tool for acquiring and training surgical skills was reviewed positively.

CONCLUSIONS

The results of this study indicate that simulation-based surgical training is a valuable and effective method for learning reconstructive techniques of minimally invasive MVR and overall general dexterity.The novel learning and training options should be implemented in the surgical traineeship for systematic teaching of various surgical skills.

Multi-task learning framework to predict the status of central venous catheter based on radiographs

Hospital patients can have catheters and lines inserted during the course of their admission to give medicines for the treatment of medical issues, especially the central venous catheter (CVC). However, malposition of CVC will lead to many complications, even death. Clinicians always detect the status of the catheter to avoid the above issues via X-ray images. To reduce the workload of clinicians and improve the efficiency of CVC status detection, a multi-task learning framework for catheter status classification based on the convolutional neural network (CNN) is proposed. The proposed framework contains three significant components which are modified HRNet, multi-task supervision including segmentation supervision and heatmap regression supervision as well as classification branch. The modified HRNet maintaining high-resolution features from the start to the end can ensure to generation of high-quality assisted information for classification. The multi-task supervision can assist in alleviating the presence of other line-like structures such as other tubes and anatomical structures shown in the X-ray image. Furthermore, during the inference, this module is also considered as an interpretation interface to show where the framework pays attention to. Eventually, the classification branch is proposed to predict the class of the status of the catheter. A public CVC dataset is utilized to evaluate the performance of the proposed method, which gains 0.823 AUC (Area under the ROC curve) and 82.6% accuracy in the test dataset. Compared with two state-of-the-art methods (ATCM method and EDMC method), the proposed method can perform best.

Cross-Parametric Generative Adversarial Network-Based Magnetic Resonance Image Feature Synthesis for Breast Lesion Classification

Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) contains information on tumor morphology and physiology for breast cancer diagnosis and treatment. However, this technology requires contrast agent injection with more acquisition time than other parametric images, such as T2-weighted imaging (T2WI). Current image synthesis methods attempt to map the image data from one domain to another, whereas it is challenging or even infeasible to map the images with one sequence into images with multiple sequences. Here, we propose a new approach of cross-parametric generative adversarial network (GAN)-based feature synthesis (CPGANFS) to generate discriminative DCE-MRI features from T2WI with applications in breast cancer diagnosis. The proposed approach decodes the T2W images into latent cross-parameter features to reconstruct the DCE-MRI and T2WI features by balancing the information shared between the two. A Wasserstein GAN with a gradient penalty is employed to differentiate the T2WI-generated features from ground-truth features extracted from DCE-MRI. The synthesized DCE-MRI feature-based model achieved significantly (p = 0.036) higher prediction performance (AUC = 0.866) in breast cancer diagnosis than that based on T2WI (AUC = 0.815). Visualization of the model shows that our CPGANFS method enhances the predictive power by levitating attention to the lesion and the surrounding parenchyma areas, which is driven by the interparametric information learned from T2WI and DCE-MRI. Our proposed CPGANFS provides a framework for cross-parametric MR image feature generation from a single-sequence image guided by an information-rich, time-series image with kinetic information. Extensive experimental results demonstrate its effectiveness with high interpretability and improved performance in breast cancer diagnosis.

Semi-Supervised Bladder Tissue Classification in Multi-Domain Endoscopic Images

OBJECTIVE

Accurate visual classification of bladder tissue during Trans-Urethral Resection of Bladder Tumor (TURBT) procedures is essential to improve early cancer diagnosis and treatment. During TURBT interventions, White Light Imaging (WLI) and Narrow Band Imaging (NBI) techniques are used for lesion detection. Each imaging technique provides diverse visual information that allows clinicians to identify and classify cancerous lesions. Computer vision methods that use both imaging techniques could improve endoscopic diagnosis. We address the challenge of tissue classification when annotations are available only in one domain, in our case WLI, and the endoscopic images correspond to an unpaired dataset, i.e. there is no exact equivalent for every image in both NBI and WLI domains.

METHOD

We propose a semi-surprised Generative Adversarial Network (GAN)-based method composed of three main components: a teacher network trained on the labeled WLI data; a cycle-consistency GAN to perform unpaired image-to-image translation, and a multi-input student network. To ensure the quality of the synthetic images generated by the proposed GAN we perform a detailed quantitative, and qualitative analysis with the help of specialists.

CONCLUSION

The overall average classification accuracy, precision, and recall obtained with the proposed method for tissue classification are 0.90, 0.88, and 0.89 respectively, while the same metrics obtained in the unlabeled domain (NBI) are 0.92, 0.64, and 0.94 respectively. The quality of the generated images is reliable enough to deceive specialists.

SIGNIFICANCE

This study shows the potential of using semi-supervised GAN-based bladder tissue classification when annotations are limited in multi-domain data.

CycleGAN-Based Image to Image Translation for Realistic Surgical Training Phantoms

Training in surgery is essential for surgeons to develop skill and dexterity. Physical training phantoms provide excellent haptic feedback and tissue properties for stitching and operating with authentic instruments and are easily available. However, they lack realistic traits and fail to reflect the complex environment of a surgical scene. Generative Adversarial Networks can be used for image-to-image translation, addressing the lack of realism in physical phantoms, by mapping patterns from the intraoperative domain onto the video stream captured during training with these surgical simulators. This work aims to achieve a successful I2I translation, from intra-operatory mitral valve surgery images onto a surgical simulator, using the CycleGAN model. Different experiments are performed - comparing the Mean Square Error Loss with the Binary Cross Entropy Loss; validating the Fréchet Inception Distance as a training and image quality metric; and studying the impact of input variability on the model performance. Differences between MSE and BCE are modest, with MSE being marginally more robust. The FID score proves to be very useful in identifying the best training epochs for the CycleGAN I2I translation architecture. Carefully selecting the input images can have a great impact in the end results. Using less style variability and input images with good feature details and clearly defined characteristics enables the network to achieve better results.Clinical Relevance- This work further contributes for the domain of realistic surgical training, successfully generating fake intra operatory images from a surgical simulator of the cardiac mitral valve.

Point detection through multi-instance deep heatmap regression for sutures in endoscopy

Purpose:

Mitral valve repair is a complex minimally invasive surgery of the heart valve. In this context, suture detection from endoscopic images is a highly relevant task that provides quantitative information to analyse suturing patterns, assess prosthetic configurations and produce augmented reality visualisations. Facial or anatomical landmark detection tasks typically contain a fixed number of landmarks, and use regression or fixed heatmap-based approaches to localize the landmarks. However in endoscopy, there are a varying number of sutures in every image, and the sutures may occur at any location in the annulus, as they are not semantically unique.

Method:

In this work, we formulate the suture detection task as a multi-instance deep heatmap regression problem, to identify entry and exit points of sutures. We extend our previous work, and introduce the novel use of a 2D Gaussian layer followed by a differentiable 2D spatial Soft-Argmax layer to function as a local non-maximum suppression.

Results:

We present extensive experiments with multiple heatmap distribution functions and two variants of the proposed model. In the intra-operative domain, Variant 1 showed a mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1 of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+ 0.0422$$\end{document}+0.0422 over the baseline. Similarly, in the simulator domain, Variant 1 showed a mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1 of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$+ 0.0865$$\end{document}+0.0865 over the baseline.

Conclusion:

The proposed model shows an improvement over the baseline in the intra-operative and the simulator domains. The data is made publicly available within the scope of the MICCAI AdaptOR2021 Challenge https://adaptor2021.github.io/, and the code at https://github.com/Cardio-AI/suture-detection-pytorch/.