Reciprocal assistance of intravascular imaging in three-dimensional stent reconstruction: Using cross-modal translation based on disentanglement representation

BACKGROUND

Accurate and efficient 3-dimension (3D) reconstruction of coronary stents in intravascular imaging of optical coherence tomography (OCT) or intravascular ultrasound (IVUS) is important for optimization of complex percutaneous coronary interventions (PCI). Deep learning has been used to address this technical challenge. However, manual annotation of stent is strenuous, especially for IVUS images. To this end, we aim to explore whether the OCT and IVUS images can assist each other in stent 3D reconstruction when one of them is lack of labeled dataset.

METHODS

We firstly performed cross-modal translation between OCT and IVUS images, where disentangled representation was employed to generate synthetic images with good stent consistency. The reciprocal assistance of OCT and IVUS in stent 3D reconstruction was then conducted by applying unsupervised and semi-supervised learning with the aid of synthetic images. Stent consistency in synthetic images and reciprocal effectiveness in stent 3D reconstruction were quantitatively assessed by F1-Score (FS) on two datasets: OCT-High Definition IVUS (HD IVUS) and OCT-Conventional IVUS (IVUS).

RESULTS

The employment of disentangled representation achieved higher stent consistency in synthetic images (OCT to HD IVUS: FS=0.789 vs 0.684; HD IVUS to OCT: FS=0.766 vs 0.682; OCT to IVUS: FS=0.806 vs 0.664; IVUS to OCT: FS=0.724 vs 0.673). For stent 3D reconstruction, the assistance from synthetic images significantly promoted unsupervised adaptation across modalities (OCT to HD IVUS: FS=0.776 vs 0.109; HD IVUS to OCT: FS=0.826 vs 0.125; OCT to IVUS: FS=0.782 vs 0.068; IVUS to OCT: FS=0.815 vs 0.123), and improved performance in semi-supervised learning, especially when only limited labeled data was available.

CONCLUSION

The intravascular images of OCT and IVUS can provide reciprocal assistance to each other in stent 3D reconstruction by cross-modal translation, where the stent consistency in synthetic images was maintained by disentangled representation.

Learning disentangled representations in the imaging domain

Disentangled representation learning has been proposed as an approach to learning general representations even in the absence of, or with limited, supervision. A good general representation can be fine-tuned for new target tasks using modest amounts of data, or used directly in unseen domains achieving remarkable performance in the corresponding task. This alleviation of the data and annotation requirements offers tantalising prospects for applications in computer vision and healthcare. In this tutorial paper, we motivate the need for disentangled representations, revisit key concepts, and describe practical building blocks and criteria for learning such representations. We survey applications in medical imaging emphasising choices made in exemplar key works, and then discuss links to computer vision applications. We conclude by presenting limitations, challenges, and opportunities.

Disentangled representation and cross-modality image translation based unsupervised domain adaptation method for abdominal organ segmentation

PURPOSE

Existing medical image segmentation models tend to achieve satisfactory performance when the training and test data are drawn from the same distribution, while they often produce significant performance degradation when used for the evaluation of cross-modality data. To facilitate the deployment of deep learning models in real-world medical scenarios and to mitigate the performance degradation caused by domain shift, we propose an unsupervised cross-modality segmentation framework based on representation disentanglement and image-to-image translation.

METHODS

Our approach is based on a multimodal image translation framework, which assumes that the latent space of images can be decomposed into a content space and a style space. First, image representations are decomposed into the content and style codes by the encoders and recombined to generate cross-modality images. Second, we propose content and style reconstruction losses to preserve consistent semantic information from original images and construct content discriminators to match the content distributions between source and target domains. Synthetic images with target domain style and source domain anatomical structures are then utilized for training of the segmentation model.

RESULTS

We applied our framework to the bidirectional adaptation experiments on MRI and CT images of abdominal organs. Compared to the case without adaptation, the Dice similarity coefficient (DSC) increased by almost 30 and 25% and average symmetric surface distance (ASSD) dropped by 13.3 and 12.2, respectively.

CONCLUSION

The proposed unsupervised domain adaptation framework can effectively improve the performance of cross-modality segmentation, and minimize the negative impact of domain shift. Furthermore, the translated image retains semantic information and anatomical structure. Our method significantly outperforms several competing methods.

Unsupervised feature disentanglement for video retrieval in minimally invasive surgery

In this paper, we propose a novel method of Unsupervised Disentanglement of Scene and Motion (UDSM) representations for minimally invasive surgery video retrieval within large databases, which has the potential to advance intelligent and efficient surgical teaching systems. To extract more discriminative video representations, two designed encoders with a triplet ranking loss and an adversarial learning mechanism are established to respectively capture the spatial and temporal information for achieving disentangled features from each frame with promising interpretability. In addition, the long-range temporal dependencies are improved in an integrated video level using a temporal aggregation module and then a set of compact binary codes that carries representative features is yielded to realize fast retrieval. The entire framework is trained in an unsupervised scheme, i.e., purely learning from raw surgical videos without using any annotation. We construct two large-scale minimally invasive surgery video datasets based on the public dataset Cholec80 and our in-house dataset of laparoscopic hysterectomy, to establish the learning process and validate the effectiveness of our proposed method qualitatively and quantitatively on the surgical video retrieval task. Extensive experiments show that our approach significantly outperforms the state-of-the-art video retrieval methods on both datasets, revealing a promising future for injecting intelligence in the next generation of surgical teaching systems.

On learning disentangled representations for individual treatment effect estimation

OBJECTIVE

Estimating the individualized treatment effect (ITE) from observational data is a challenging task due to selection bias, which results from the distributional discrepancy between different treatment groups caused by the dependence between features and assigned treatments. This dependence is induced by the factors related to the treatment assignment. We hypothesize that features consist of three types of latent factors: outcome-specific factors, treatment-specific factors and confounders. Then, we aim to reduce the influence of treatment-related factors, i.e., treatment-specific factors and confounders, on outcome prediction to mitigate the effects of selection bias.

METHOD

We present a novel representation learning model in which both the main task of outcome prediction and the auxiliary task of classifying the treatment assignment are used to learn the outcome-oriented and treatment-oriented latent representations, respectively. However, since the confounders are related to both treatment assignment and outcome, it is still contained in the representations. To further reduce influence of the confounders contained in both representations, individualized orthogonal regularization is incorporated into the proposed model. The orthogonal regularization forces the outcome-oriented and treatment-oriented latent representations of an individual to be vertical in the inner product space, meaning they are orthogonal with each other, and the common information of confounder is reduced. Such that the ITE can be estimated more precisely without the effects of selection bias.

RESULT

We evaluate our proposed model on a semi-simulated dataset and a real-world dataset. The experimental results demonstrate that the proposed model achieves competitive or better performance compared with the performances of the state-of-the-art models.

CONCLUSION

The proposed method is well performed on ITE estimation with the ability to reduce selection bias thoroughly by incorporating an auxiliary task and adopting orthogonal regularization to disentangle the latent factors.

SIGNIFICANCE

This paper offers a novel method of reducing selection bias in estimating the ITE from observational data by disentangled representation learning.