MV-RAN: Multiview recurrent aggregation network for echocardiographic sequences segmentation and full cardiac cycle analysis

MADR-Net: multi-level attention dilated residual neural network for segmentation of medical images

Medical image segmentation has made a significant contribution towards delivering affordable healthcare by facilitating the automatic identification of anatomical structures and other regions of interest. Although convolution neural networks have become prominent in the field of medical image segmentation, they suffer from certain limitations. In this study, we present a reliable framework for producing performant outcomes for the segmentation of pathological structures of 2D medical images. Our framework consists of a novel deep learning architecture, called deep multi-level attention dilated residual neural network (MADR-Net), designed to improve the performance of medical image segmentation. MADR-Net uses a U-Net encoder/decoder backbone in combination with multi-level residual blocks and atrous pyramid scene parsing pooling. To improve the segmentation results, channel-spatial attention blocks were added in the skip connection to capture both the global and local features and superseded the bottleneck layer with an ASPP block. Furthermore, we introduce a hybrid loss function that has an excellent convergence property and enhances the performance of the medical image segmentation task. We extensively validated the proposed MADR-Net on four typical yet challenging medical image segmentation tasks: (1) Left ventricle, left atrium, and myocardial wall segmentation from Echocardiogram images in the CAMUS dataset, (2) Skin cancer segmentation from dermoscopy images in ISIC 2017 dataset, (3) Electron microscopy in FIB-SEM dataset, and (4) Fluid attenuated inversion recovery abnormality from MR images in LGG segmentation dataset. The proposed algorithm yielded significant results when compared to state-of-the-art architectures such as U-Net, Residual U-Net, and Attention U-Net. The proposed MADR-Net consistently outperformed the classical U-Net by 5.43%, 3.43%, and 3.92% relative improvement in terms of dice coefficient, respectively, for electron microscopy, dermoscopy, and MRI. The experimental results demonstrate superior performance on single and multi-class datasets and that the proposed MADR-Net can be utilized as a baseline for the assessment of cross-dataset and segmentation tasks.

Embedding Tasks Into the Latent Space: Cross-Space Consistency for Multi-Dimensional Analysis in Echocardiography

Multi-dimensional analysis in echocardiography has attracted attention due to its potential for clinical indices quantification and computer-aided diagnosis. It can utilize various information to provide the estimation of multiple cardiac indices. However, it still has the challenge of inter-task conflict. This is owing to regional confusion, global abnormalities, and time-accumulated errors. Task mapping methods have the potential to address inter-task conflict. However, they may overlook the inherent differences between tasks, especially for multi-level tasks (e.g., pixel-level, image-level, and sequence-level tasks). This may lead to inappropriate local and spurious task constraints. We propose cross-space consistency (CSC) to overcome the challenge. The CSC embeds multi-level tasks to the same-level to reduce inherent task differences. This allows multi-level task features to be consistent in a unified latent space. The latent space extracts task-common features and constrains the distance in these features. This constrains the task weight region that satisfies multiple task conditions. Extensive experiments compare the CSC with fifteen state-of-the-art echocardiographic analysis methods on five datasets (10,908 patients). The result shows that the CSC can provide left ventricular (LV) segmentation, (DSC = 0.932), keypoint detection (MAE = 3.06mm), and keyframe identification (accuracy = 0.943). These results demonstrate that our method can provide a multi-dimensional analysis of cardiac function and is robust in large-scale datasets.

An Overview of Deep Learning Methods for Left Ventricle Segmentation

Cardiac health diseases are one of the key causes of death around the globe. The number of heart patients has considerably increased during the pandemic. Therefore, it is crucial to assess and analyze the medical and cardiac images. Deep learning architectures, specifically convolutional neural networks have profoundly become the primary choice for the assessment of cardiac medical images. The left ventricle is a vital part of the cardiovascular system where the boundary and size perform a significant role in the evaluation of cardiac function. Due to automatic segmentation and good promising results, the left ventricle segmentation using deep learning has attracted a lot of attention. This article presents a critical review of deep learning methods used for the left ventricle segmentation from frequently used imaging modalities including magnetic resonance images, ultrasound, and computer tomography. This study also demonstrates the details of the network architecture, software, and hardware used for training along with publicly available cardiac image datasets and self-prepared dataset details incorporated. The summary of the evaluation matrices with results used by different researchers is also presented in this study. Finally, all this information is summarized and comprehended in order to assist the readers to understand the motivation and methodology of various deep learning models, as well as exploring potential solutions to future challenges in LV segmentation.

Left ventricle analysis in echocardiographic images using transfer learning

The segmentation of cardiac boundaries, specifically Left Ventricle (LV) segmentation in 2D echocardiographic images, is a critical step in LV segmentation and cardiac function assessment. These images are generally of poor quality and present low contrast, making daily clinical delineation difficult, time-consuming, and often inaccurate. Thus, it is necessary to design an intelligent automatic endocardium segmentation system. The present work aims to examine and assess the performance of some deep learning-based architectures such as U-Net1, U-Net2, LinkNet, Attention U-Net, and TransUNet using the public CAMUS (Cardiac Acquisitions for Multi-structure Ultrasound Segmentation) dataset. The adopted approach emphasizes the advantage of using transfer learning and resorting to pre-trained backbones in the encoder part of a segmentation network for echocardiographic image analysis. The experimental findings indicated that the proposed framework with the [Formula: see text]-[Formula: see text] is quite promising; it outperforms other more recent approaches with a Dice similarity coefficient of 93.30% and a Hausdorff Distance of 4.01 mm. In addition, a good agreement between the clinical indices calculated from the automatic segmentation and those calculated from the ground truth segmentation. For instance, the mean absolute errors for the left ventricular end-diastolic volume, end-systolic volume, and ejection fraction are equal to 7.9 ml, 5.4 ml, and 6.6%, respectively. These results are encouraging and point out additional perspectives for further improvement.

Echocardiography Segmentation With Enforced Temporal Consistency

Convolutional neural networks (CNN) have demonstrated their ability to segment 2D cardiac ultrasound images. However, despite recent successes according to which the intra-observer variability on end-diastole and end-systole images has been reached, CNNs still struggle to leverage temporal information to provide accurate and temporally consistent segmentation maps across the whole cycle. Such consistency is required to accurately describe the cardiac function, a necessary step in diagnosing many cardiovascular diseases. In this paper, we propose a framework to learn the 2D+time apical long-axis cardiac shape such that the segmented sequences can benefit from temporal and anatomical consistency constraints. Our method is a post-processing that takes as input segmented echocardiographic sequences produced by any state-of-the-art method and processes it in two steps to (i) identify spatio-temporal inconsistencies according to the overall dynamics of the cardiac sequence and (ii) correct the inconsistencies. The identification and correction of cardiac inconsistencies relies on a constrained autoencoder trained to learn a physiologically interpretable embedding of cardiac shapes, where we can both detect and fix anomalies. We tested our framework on 98 full-cycle sequences from the CAMUS dataset, which are available alongside this paper. Our temporal regularization method not only improves the accuracy of the segmentation across the whole sequences, but also enforces temporal and anatomical consistency.

MCP-Net: Inter-frame Motion Correction with Patlak Regularization for Whole-body Dynamic PET

Inter-frame patient motion introduces spatial misalignment and degrades parametric imaging in whole-body dynamic positron emission tomography (PET). Most current deep learning inter-frame motion correction works consider only the image registration problem, ignoring tracer kinetics. We propose an inter-frame Motion Correction framework with Patlak regularization (MCP-Net) to directly optimize the Patlak fitting error and further improve model performance. The MCP-Net contains three modules: a motion estimation module consisting of a multiple-frame 3-D U-Net with a convolutional long short-term memory layer combined at the bottleneck; an image warping module that performs spatial transformation; and an analytical Patlak module that estimates Patlak fitting with the motion-corrected frames and the individual input function. A Patlak loss penalization term using mean squared percentage fitting error is introduced to the loss function in addition to image similarity measurement and displacement gradient loss. Following motion correction, the parametric images were generated by standard Patlak analysis. Compared with both traditional and deep learning benchmarks, our network further corrected the residual spatial mismatch in the dynamic frames, improved the spatial alignment of Patlak Ki/Vb images, and reduced normalized fitting error. With the utilization of tracer dynamics and enhanced network performance, MCP-Net has the potential for further improving the quantitative accuracy of dynamic PET. Our code is released at https://github.com/gxq1998/MCP-Net.

Unsupervised inter-frame motion correction for whole-body dynamic PET using convolutional long short-term memory in a convolutional neural network

Subject motion in whole-body dynamic PET introduces inter-frame mismatch and seriously impacts parametric imaging. Traditional non-rigid registration methods are generally computationally intense and time-consuming. Deep learning approaches are promising in achieving high accuracy with fast speed, but have yet been investigated with consideration for tracer distribution changes or in the whole-body scope. In this work, we developed an unsupervised automatic deep learning-based framework to correct inter-frame body motion. The motion estimation network is a convolutional neural network with a combined convolutional long short-term memory layer, fully utilizing dynamic temporal features and spatial information. Our dataset contains 27 subjects each under a 90-min FDG whole-body dynamic PET scan. Evaluating performance in motion simulation studies and a 9-fold cross-validation on the human subject dataset, compared with both traditional and deep learning baselines, we demonstrated that the proposed network achieved the lowest motion prediction error, obtained superior performance in enhanced qualitative and quantitative spatial alignment between parametric Ki and Vb images, and significantly reduced parametric fitting error. We also showed the potential of the proposed motion correction method for impacting downstream analysis of the estimated parametric images, improving the ability to distinguish malignant from benign hypermetabolic regions of interest. Once trained, the motion estimation inference time of our proposed network was around 460 times faster than the conventional registration baseline, showing its potential to be easily applied in clinical settings.

Deep learning-based framework for segmentation of multiclass rib fractures in CT utilizing a multi-angle projection network

PURPOSE

Clinical rib fracture diagnosis via computed tomography (CT) screening has attracted much attention in recent years. However, automated and accurate segmentation solutions remain a challenging task due to the large sets of 3D CT data to deal with. Down-sampling is often required to face computer constraints, but the performance of the segmentation may decrease in this case.

METHODS

A new multi-angle projection network (MAPNet) method is proposed for accurately segmenting rib fractures by means of a deep learning approach. The proposed method incorporates multi-angle projection images to complementarily and comprehensively extract the rib characteristics using a rib extraction (RE) module and the fracture features using a fracture segmentation (FS) module. A multi-angle projection fusion (MPF) module is designed for fusing multi-angle spatial features. RESULTS: It is shown that MAPNet can capture more detailed rib fracture features than some commonly used segmentation networks. Our method achieves a better performance in accuracy (88.06 ± 6.97%), sensitivity (89.26 ± 5.69%), specificity (87.58% ± 7.66%) and in terms of classical criteria like dice (85.41 ± 3.35%), intersection over union (IoU, 80.37 ± 4.63%), and Hausdorff distance (HD, 4.34 ± 3.1).

CONCLUSION

We propose a rib fracture segmentation technique to deal with the problem of automatic fracture diagnosis. The proposed method avoids the down-sampling of 3D CT data through a projection technique. Experimental results show that it has excellent potential for clinical applications.

Direct left-ventricular global longitudinal strain (GLS) computation with a fully convolutional network

This study's purpose was to develop a direct MRI-based, deep-learning semantic segmentation approach for computing global longitudinal strain (GLS), a known metric for detecting left-ventricular (LV) cardiotoxicity in breast cancer. Displacement Encoding with Stimulated Echoes cardiac image phases acquired from 30 breast cancer patients and 30 healthy females were unwrapped via a DeepLabV3 + fully convolutional network (FCN). Myocardial strains were directly computed from the unwrapped phases with the Radial Point Interpolation Method. FCN-unwrapped phases of a phantom's rotating gel were validated against quality-guided phase-unwrapping (QGPU) and robust transport of intensity equation (RTIE) phase-unwrapping. FCN performance on unwrapping human LV data was measured with F1 and Dice scores versus QGPU ground-truth. The reliability of FCN-based strains was assessed against RTIE-based strains with Cronbach's alpha (C-α) intraclass correlation coefficient. Mean squared error (MSE) of unwrapping the phantom experiment data at 0 dB signal-to-noise ratio were 1.6, 2.7 and 6.1 with FCN, QGPU and RTIE techniques. Human data classification accuracies were F1 = 0.95 (Dice = 0.96) with FCN and F1 = 0.94 (Dice = 0.95) with RTIE. GLS results from FCN and RTIE were -16 ± 3% vs. -16 ± 3% (C-α = 0.9) for patients and -20 ± 3% vs. -20 ± 3% (C-α = 0.9) for healthy subjects. The low MSE from the phantom validation demonstrates accuracy of phase-unwrapping with the FCN and comparable human subject results versus RTIE demonstrate GLS analysis accuracy. A deep-learning methodology for phase-unwrapping in medical images and GLS computation was developed and validated in a heterogeneous cohort.

Constraint-based Unsupervised Domain Adaptation network for Multi-Modality Cardiac Image Segmentation

The cardiac CT and MRI images depict the various structures of the heart, which are very valuable for analyzing heart function. However, due to the difference in the shape of the cardiac images and imaging techniques, automatic segmentation is challenging. To solve this challenge, in this paper, we propose a new constraint-based unsupervised domain adaptation network. This network first performs mutual translation of images between different domains, it can provide training data for the segmentation model, and ensure domain invariance at the image level. Then, we input the target domain into the source domain segmentation model to obtain pseudo-labels and introduce cross-domain self-supervised learning between the two segmentation models. Here, a new loss function is designed to ensure the accuracy of the pseudo-labels. In addition, a cross-domain consistency loss is also introduced. Finally, we construct a multi-level aggregation segmentation network to obtain more refined target domain information. We validate our method on the public whole heart image segmentation challenge dataset and obtain experimental results of 82.9% and 5.5 on dice and average symmetric surface distance (ASSD), respectively. These experimental results prove that our method can provide important assistance in the clinical evaluation of unannotated cardiac datasets.

Artificial intelligence applied to support medical decisions for the automatic analysis of echocardiogram images: A systematic review

The echocardiogram is a test that is widely used in Heart Disease Diagnoses. However, its analysis is largely dependent on the physician's experience. In this regard, artificial intelligence has become an essential technology to assist physicians. This study is a Systematic Literature Review (SLR) of primary state-of-the-art studies that used Artificial Intelligence (AI) techniques to automate echocardiogram analyses. Searches on the leading scientific article indexing platforms using a search string returned approximately 1400 articles. After applying the inclusion and exclusion criteria, 118 articles were selected to compose the detailed SLR. This SLR presents a thorough investigation of AI applied to support medical decisions for the main types of echocardiogram (Transthoracic, Transesophageal, Doppler, Stress, and Fetal). The article's data extraction indicated that the primary research interest of the studies comprised four groups: 1) Improvement of image quality; 2) identification of the cardiac window vision plane; 3) quantification and analysis of cardiac functions, and; 4) detection and classification of cardiac diseases. The articles were categorized and grouped to show the main contributions of the literature to each type of ECHO. The results indicate that the Deep Learning (DL) methods presented the best results for the detection and segmentation of the heart walls, right and left atrium and ventricles, and classification of heart diseases using images/videos obtained by echocardiography. The models that used Convolutional Neural Network (CNN) and its variations showed the best results for all groups. The evidence produced by the results presented in the tabulation of the studies indicates that the DL contributed significantly to advances in echocardiogram automated analysis processes. Although several solutions were presented regarding the automated analysis of ECHO, this area of research still has great potential for further studies to improve the accuracy of results already known in the literature.

Automatic Meniscus Segmentation Using Adversarial Learning-Based Segmentation Network with Object-Aware Map in Knee MR Images

Meniscus segmentation from knee MR images is an essential step when analyzing the length, width, height, cross-sectional area, surface area for meniscus allograft transplantation using a 3D reconstruction model based on the patient's normal meniscus. In this paper, we propose a two-stage DCNN that combines a 2D U-Net-based meniscus localization network with a conditional generative adversarial network-based segmentation network using an object-aware map. First, the 2D U-Net segments knee MR images into six classes including bone and cartilage with whole MR images at a resolution of 512 × 512 to localize the medial and lateral meniscus. Second, adversarial learning with a generator based on the 2D U-Net and a discriminator based on the 2D DCNN using an object-aware map segments the meniscus into localized regions-of-interest with a resolution of 64 × 64. The average Dice similarity coefficient of the meniscus was 85.18% at the medial meniscus and 84.33% at the lateral meniscus; these values were 10.79%p and 1.14%p, and 7.78%p and 1.12%p higher than the segmentation method without adversarial learning and without the use of an object-aware map with the Dice similarity coefficient at the medial meniscus and lateral meniscus, respectively. The proposed automatic meniscus localization through multi-class can prevent the class imbalance problem by focusing on local regions. The proposed adversarial learning using an object-aware map can prevent under-segmentation by repeatedly judging and improving the segmentation results, and over-segmentation by considering information only from the meniscus regions. Our method can be used to identify and analyze the shape of the meniscus for allograft transplantation using a 3D reconstruction model of the patient's unruptured meniscus.

Clinical Analysis of Improved Particle Swarm Algorithm-Based Magnetic Resonance Imaging Diagnosis of Placenta Accreta

The magnetic resonance imaging (MRI) image processing capabilities were investigated based on the improved particle swarm optimization (IPSO) algorithm, and the clinical application analysis of MRI images in the diagnosis of placenta accreta (PA) was evaluated in this study. The MRI uterine images were detected on the basis of IPSO. Besides, the clinical data of 89 patients with PA were selected and collected, who were diagnosed by clinical cesarean section surgery and pathological comprehensive diagnosis in hospital from January 2018 to July 2020. Then, all of them underwent the ultrasound (US) and MRI examinations, and the differences of sensitivity, specificity, and accuracy between MRI and US under IPSO in the diagnosis of PA were compared, as well as the differences in the diagnosis of adhesive, implantable, and penetrated PA. The results showed that the difference in detection between IPSO-based MRI images and US images was not statistically substantial (p > 0.05), but the number of initial detections was higher than the number of US examination. MRI examination had higher sensitivity and specificity in the diagnosis of PA during pregnancy, especially for implantable PA, compared with US examination (p < 0.05). In conclusion, MRI images based on the improved particle swarm optimization algorithm showed a good application effect in the diagnosis of placental implantation diseases, which was worthy of further promotion in clinical practice.

A 3D multiscale view convolutional neural network with attention for mental disease diagnosis on MRI images

Computer Assisted Diagnosis (CAD) based on brain Magnetic Resonance Imaging (MRI) is a popular research field for the computer science and medical engineering. Traditional machine learning and deep learning methods were employed in the classification of brain MRI images in the previous studies. However, the current algorithms rarely take into consideration the influence of multi-scale brain connectivity disorders on some mental diseases. To improve this defect, a deep learning structure was proposed based on MRI images, which was designed to consider the brain's connections at different sizes and the attention of connections. In this work, a Multiscale View (MV) module was proposed, which was designed to detect multi-scale brain network disorders. On the basis of the MV module, the path attention module was also proposed to simulate the attention selection of the parallel paths in the MV module. Based on the two modules, we proposed a 3D Multiscale View Convolutional Neural Network with Attention (3D MVA-CNN) for classification of MRI images for mental disease. The proposed method outperformed the previous 3D CNN structures in the structural MRI data of ADHD-200 and the functional MRI data of schizophrenia. Finally, we also proposed a preliminary framework for clinical application using 3D CNN, and discussed its limitations on data accessing and reliability. This work promoted the assisted diagnosis of mental diseases based on deep learning and provided a novel 3D CNN method based on MRI data.

Discrete light sheet microscopic segmentation of left ventricle using morphological tuning and active contours

Left ventricular segmentation using cardiovascular MR scan is required for the diagnosis and further cure of cardiac diseases. Automatic systems for left ventricle segmentation are being studied for attaining more accurate results in a shorter period of time. A novel algorithm introducing discrete segmentation of left ventricle achieves an independent processing of images swiftly. The workflow consists of four segments; first, automated localization is performed on the MR image. Second, performing preprocessing intimately improves and enhances the quality of image using mean contrast adjustment. Central segmentation of endocardium and epicardium layers includes novel MTAC (Morphological tuning using active contours) segmentation algorithm that provides a perfect combination of active contours and morphological tuning to bring an adequate and desirable segmentation. The prospective snake model is a restrained progression, which takes iterations for an impulse throughout the left ventricle contours. At the end, contrast based refining overcomes minor edge problems for both outer and inner boundaries. Proposed algorithm is evaluated via Sunnybrook cardiac MR images by producing an overall average perpendicular distance 2.45 mm, an average dice matrix (endo: 91.3%; epi: 92.16%) and 91.7% dice matrix of overall endocardium and epicardium contours from ground truth contours.

Multi-level semantic adaptation for few-shot segmentation on cardiac image sequences

Obtaining manual labels is time-consuming and labor-intensive on cardiac image sequences. Few-shot segmentation can utilize limited labels to learn new tasks. However, it suffers from two challenges: spatial-temporal distribution bias and long-term information bias. These challenges derive from the impact of the time dimension on cardiac image sequences, resulting in serious over-adaptation. In this paper, we propose the multi-level semantic adaptation (MSA) for few-shot segmentation on cardiac image sequences. The MSA addresses the two biases by exploring the domain adaptation and the weight adaptation on the semantic features in multiple levels, including sequence-level, frame-level, and pixel-level. First, the MSA proposes the dual-level feature adjustment for domain adaptation in spatial and temporal directions. This adjustment explicitly aligns the frame-level feature and the sequence-level feature to improve the model adaptation on diverse modalities. Second, the MSA explores the hierarchical attention metric for weight adaptation in the frame-level feature and the pixel-level feature. This metric focuses on the similar frame and the target region to promote the model discrimination on the border features. The extensive experiments demonstrate that our MSA is effective in few-shot segmentation on cardiac image sequences with three modalities, i.e. MR, CT, and Echo (e.g. the average Dice is 0.9243), as well as superior to the ten state-of-the-art methods.

TransMed: Transformers Advance Multi-Modal Medical Image Classification

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.

TransMed: Transformers Advance Multi-Modal Medical Image Classification

Over the past decade, convolutional neural networks (CNN) have shown very competitive performance in medical image analysis tasks, such as disease classification, tumor segmentation, and lesion detection. CNN has great advantages in extracting local features of images. However, due to the locality of convolution operation, it cannot deal with long-range relationships well. Recently, transformers have been applied to computer vision and achieved remarkable success in large-scale datasets. Compared with natural images, multi-modal medical images have explicit and important long-range dependencies, and effective multi-modal fusion strategies can greatly improve the performance of deep models. This prompts us to study transformer-based structures and apply them to multi-modal medical images. Existing transformer-based network architectures require large-scale datasets to achieve better performance. However, medical imaging datasets are relatively small, which makes it difficult to apply pure transformers to medical image analysis. Therefore, we propose TransMed for multi-modal medical image classification. TransMed combines the advantages of CNN and transformer to efficiently extract low-level features of images and establish long-range dependencies between modalities. We evaluated our model on two datasets, parotid gland tumors classification and knee injury classification. Combining our contributions, we achieve an improvement of 10.1% and 1.9% in average accuracy, respectively, outperforming other state-of-the-art CNN-based models. The results of the proposed method are promising and have tremendous potential to be applied to a large number of medical image analysis tasks. To our best knowledge, this is the first work to apply transformers to multi-modal medical image classification.

A Deep Learning Approach for Segmentation, Classification, and Visualization of 3-D High-Frequency Ultrasound Images of Mouse Embryos

Segmentation and mutant classification of high-frequency ultrasound (HFU) mouse embryo brain ventricle (BV) and body images can provide valuable information for developmental biologists. However, manual segmentation and identification of BV and body requires substantial time and expertise. This article proposes an accurate, efficient and explainable deep learning pipeline for automatic segmentation and classification of the BV and body. For segmentation, a two-stage framework is implemented. The first stage produces a low-resolution segmentation map, which is then used to crop a region of interest (ROI) around the target object and serve as the probability map of the autocontext input for the second-stage fine-resolution refinement network. The segmentation then becomes tractable on high-resolution 3-D images without time-consuming sliding windows. The proposed segmentation method significantly reduces inference time (102.36-0.09 s/volume ≈ 1000× faster) while maintaining high accuracy comparable to previous sliding-window approaches. Based on the BV and body segmentation map, a volumetric convolutional neural network (CNN) is trained to perform a mutant classification task. Through backpropagating the gradients of the predictions to the input BV and body segmentation map, the trained classifier is found to largely focus on the region where the Engrailed-1 (En1) mutation phenotype is known to manifest itself. This suggests that gradient backpropagation of deep learning classifiers may provide a powerful tool for automatically detecting unknown phenotypes associated with a known genetic mutation.

Which GAN? A comparative study of generative adversarial network-based fast MRI reconstruction

Fast magnetic resonance imaging (MRI) is crucial for clinical applications that can alleviate motion artefacts and increase patient throughput. K-space undersampling is an obvious approach to accelerate MR acquisition. However, undersampling of k-space data can result in blurring and aliasing artefacts for the reconstructed images. Recently, several studies have been proposed to use deep learning-based data-driven models for MRI reconstruction and have obtained promising results. However, the comparison of these methods remains limited because the models have not been trained on the same datasets and the validation strategies may be different. The purpose of this work is to conduct a comparative study to investigate the generative adversarial network (GAN)-based models for MRI reconstruction. We reimplemented and benchmarked four widely used GAN-based architectures including DAGAN, ReconGAN, RefineGAN and KIGAN. These four frameworks were trained and tested on brain, knee and liver MRI images using twofold, fourfold and sixfold accelerations, respectively, with a random undersampling mask. Both quantitative evaluations and qualitative visualization have shown that the RefineGAN method has achieved superior performance in reconstruction with better accuracy and perceptual quality compared to other GAN-based methods. This article is part of the theme issue 'Synergistic tomographic image reconstruction: part 1'.

CMMCSegNet: Cross-Modality Multicascade Indirect LGE Segmentation on Multimodal Cardiac MR

Since Late-Gadolinium Enhancement (LGE) of cardiac magnetic resonance (CMR) visualizes myocardial infarction, and the balanced-Steady State Free Precession (bSSFP) cine sequence can capture cardiac motions and present clear boundaries; multimodal CMR segmentation has played an important role in the assessment of myocardial viability and clinical diagnosis, while automatic and accurate CMR segmentation still remains challenging due to a very small amount of labeled LGE data and the relatively low contrasts of LGE. The main purpose of our work is to learn the real/fake bSSFP modality with ground truths to indirectly segment the LGE modality of cardiac MR by using a proposed cross-modality multicascade framework: cross-modality translation network and automatic segmentation network, respectively. In the segmentation stage, a novel multicascade pix2pix network is designed to segment the fake bSSFP sequence obtained from a cross-modality translation network. Moreover, we propose perceptual loss measuring features between ground truth and prediction, which are extracted from the pretrained vgg network in the segmentation stage. We evaluate the performance of the proposed method on the multimodal CMR dataset and verify its superiority over other state-of-the-art approaches under different network structures and different types of adversarial losses in terms of dice accuracy in testing. Therefore, the proposed network is promising for Indirect Cardiac LGE Segmentation in clinical applications.

Hybrid dilation and attention residual U-Net for medical image segmentation

Medical image segmentation is a typical task in medical image processing and critical foundation in medical image analysis. U-Net is well-liked in medical image segmentation, but it doesn't fully explore useful features of the channel and capitalize on the contextual information. Therefore, we present an improved U-Net with residual connections, adding a plug-and-play, very portable channel attention (CA) block and a hybrid dilated attention convolutional (HDAC) layer to perform medical image segmentation for different tasks accurately and effectively, and call it HDA-ResUNet, in which we fully utilize advantages of U-Net, attention mechanism and dilated convolution. In contrast to the simple copy splicing of U-Net in the skip connection, the channel attention block is inserted into the extracted feature map of the encoding path before decoding operation. Since this block is lightweight, we can apply it to multiple layers in the backbone network to optimize the channel effect of this layer's coding operation. In addition, the convolutional layer at the bottom of the "U"-shaped network is replaced by a hybrid dilated attention convolutional (HDAC) layer to fuse information from different sizes of receptive fields. The proposed HDA-ResUNet is evaluated on four datasets: liver and tumor segmentation (LiTS 2017), lung segmentation (Lung dataset), nuclear segmentation in microscope images (DSB 2018) and neuron structure segmentation (ISBI 2012). The dice global scores of liver and tumor segmentation (LiTS 2017) reach 0.949 and 0.799. The dice coefficients of lung segmentation and nuclear segmentation are 0.9797 and 0.9081 respectively, and the information theoretic score for the last one is 0.9703. The segmentation results are all more accurate than U-Net with fewer parameters, and the problem of slow convergence speed of U-Net on DBS 2018 is solved.

Automatic segmentation of the left ventricle in echocardiographic images using convolutional neural networks

Background

Two-dimensional echocardiography (2D echo) is the most widely used non-invasive imaging modality due to its fast acquisition time, low cost, and high temporal resolution. Boundary identification of left ventricle (LV) in 2D echo, i.e., image segmentation, is the first step to calculate relevant clinical parameters. Currently, LV segmentation in 2D echo is primarily conducted semi-manually. A fully-automatic segmentation of the LV wall needs further development.

Methods

We evaluated the performance of the state-of-the-art convolutional neural networks (CNNs) for the segmentation of 2D echo images from 6 standard projections of the LV. We used two segmentation algorithms: U-net and segAN. The models were trained using an in-house dataset, which consists of 1,649 porcine images from 6 to 8 different pigs. In addition, a transfer learning approach was used for the segmentation of long-axis projections by training models with our database based on the previously trained weights obtained from Cardiac Acquisitions for Multi-structure Ultrasound Segmentation (CAMUS) dataset. The models were tested on a separate set of images from two other pigs by computing several metrics. The segmentation process was combined with a 3D reconstruction framework to quantify the physiological indices such as LV volumes and ejection fraction (EF).

Results

The average dice metric for the LV cavity was 0.90 and 0.91 for the U-net and segAN, respectively, which was higher than 0.82 for the level-set (P value: 3.31×10^-25). The average Hausdorff distance for the LV cavity was 2.71 mm and 2.82 mm for the U-net and segAN, respectively, which was lower than 3.64 mm for the level-set (P value: 4.86×10^-16). The LV shapes and volumes obtained using the CNN segmentation models were in good agreement with the results segmented by the experts. In addition, the differences of the calculated physiological parameters between two 3D reconstruction models segmented by the experts and CNNs were less than 15%.

Conclusions

The results showed that both CNN models achieve higher performance on LV segmentation than the level-set method. The error of the reconstruction from automatic segmentation compared to the expert segmentation is less than 15%, which is within the 20% error of echo compared to the gold standard.

Coronary Vessel Segmentation by Coarse-to-Fine Strategy Using U-nets

Each level of the coronary artery has different sizes and properties. The primary coronary arteries usually have high contrast to the background, while the secondary coronary arteries have low contrast to the background and thin structures. Furthermore, several small vessels are disconnected or broken up vascular segments. It is a challenging task to use a single model to segment all coronary artery sizes. To overcome this problem, we propose a novel segmenting method for coronary artery extraction from angiograms based on the primary and secondary coronary artery. Our method is a coarse-to-fine strategic approach for extracting coronary arteries in many different sizes. We construct the first U-net model to segment the main coronary artery extraction and build a new algorithm to determine the junctions of the main coronary artery with the secondary coronary artery. Using these junctions, we determine regions of the secondary coronary arteries (rectangular regions) for a secondary coronary artery-extracted segment with the second U-net model. The experiment result is 76.40% in terms of Dice coefficient on coronary X-ray datasets. The proposed approach presents its potential in coronary vessel segmentation.

An 8-layer residual U-Net with deep supervision for segmentation of the left ventricle in cardiac CT angiography

BACKGROUND AND OBJECTIVES

Accurate segmentation of left ventricle (LV) is a fundamental step in evaluation of cardiac function. Cardiac CT angiography (CCTA) has become an important clinical diagnostic method for cardio-vascular disease (CVD) due to its non-invasive, short exam time, and low cost. To obtain the segmentation of the LV in CCTA scans, we present a deep learning method based on an 8-layer residual U-Net with deep supervision.

METHODS

Based on the original 4-layer U-Net, our method deepened the network to eight layers, which increased the fitting capacity of the network, thus greatly improved its LV recognition capability. Residual blocks were incorporated to optimize the network from the increased depth. Auxiliary paths as deep supervision were introduced to supervise the intermediate information to improve the segmentation quality. In this study, we collected CCTA scans of 100 patients. Eighty patients with 1600 discrete slices were used to train the LV segmentation and the remaining 20 patients with 400 discrete slices were used for testing our method. An interactive graph cut algorithm was utilized reliably to annotate the LV reference standard that was further confirmed by cardiologists. Online data augmentation was performed in the training process to improve the generalization and robustness of our method.

RESULTS

Compared with the segmentation results from the original U-Net and FC-DenseNet56 with Dice similarity coefficient (DSC) of 0.878±0.230 and 0.897±0.189, respectively, our method demonstrated higher segmentation accuracy and robustness for varying LV shape, size, and contrast, achieving DSC of 0.927±0.139. Without online data augmentation, our method resulted in inferior performance with DSC of 0.911±0.170. In addition, compared with the provided results from other existing studies in the LV segmentation of cardiac CT images, our method achieved a competitive performance for the LV segmentation.

CONCLUSIONS

The proposed 8-layer residual U-Net with deep supervision accurately and efficiently segments the LV in CCTA scans. This method has potential advantages to be a reliable segmentation method and useful for the evaluation of cardiac function in the future study.

Deep learning in spatiotemporal cardiac imaging: A review of methodologies and clinical usability

The use of different cardiac imaging modalities such as MRI, CT or ultrasound enables the visualization and interpretation of altered morphological structures and function of the heart. In recent years, there has been an increasing interest in AI and deep learning that take into account spatial and temporal information in medical image analysis. In particular, deep learning tools using temporal information in image processing have not yet found their way into daily clinical practice, despite its presumed high diagnostic and prognostic value. This review aims to synthesize the most relevant deep learning methods and discuss their clinical usability in dynamic cardiac imaging using for example the complete spatiotemporal image information of the heart cycle. Selected articles were categorized according to the following indicators: clinical applications, quality of datasets, preprocessing and annotation, learning methods and training strategy, and test performance. Clinical usability was evaluated based on these criteria by classifying the selected papers into (i) clinical level, (ii) robust candidate and (iii) proof of concept applications. Interestingly, not a single one of the reviewed papers was classified as a "clinical level" study. Almost 39% of the articles achieved a "robust candidate" and as many as 61% a "proof of concept" status. In summary, deep learning in spatiotemporal cardiac imaging is still strongly research-oriented and its implementation in clinical application still requires considerable efforts. Challenges that need to be addressed are the quality of datasets together with clinical verification and validation of the performance achieved by the used method.

PWD-3DNet: A Deep Learning-Based Fully-Automated Segmentation of Multiple Structures on Temporal Bone CT Scans

The temporal bone is a part of the lateral skull surface that contains organs responsible for hearing and balance. Mastering surgery of the temporal bone is challenging because of this complex and microscopic three-dimensional anatomy. Segmentation of intra-temporal anatomy based on computed tomography (CT) images is necessary for applications such as surgical training and rehearsal, amongst others. However, temporal bone segmentation is challenging due to the similar intensities and complicated anatomical relationships among critical structures, undetectable small structures on standard clinical CT, and the amount of time required for manual segmentation. This paper describes a single multi-class deep learning-based pipeline as the first fully automated algorithm for segmenting multiple temporal bone structures from CT volumes, including the sigmoid sinus, facial nerve, inner ear, malleus, incus, stapes, internal carotid artery and internal auditory canal. The proposed fully convolutional network, PWD-3DNet, is a patch-wise densely connected (PWD) three-dimensional (3D) network. The accuracy and speed of the proposed algorithm was shown to surpass current manual and semi-automated segmentation techniques. The experimental results yielded significantly high Dice similarity scores and low Hausdorff distances for all temporal bone structures with an average of 86% and 0.755 millimeter (mm), respectively. We illustrated that overlapping in the inference sub-volumes improves the segmentation performance. Moreover, we proposed augmentation layers by using samples with various transformations and image artefacts to increase the robustness of PWD-3DNet against image acquisition protocols, such as smoothing caused by soft tissue scanner settings and larger voxel sizes used for radiation reduction. The proposed algorithm was tested on low-resolution CTs acquired by another center with different scanner parameters than the ones used to create the algorithm and shows potential for application beyond the particular training data used in the study.

LU-Net: A Multistage Attention Network to Improve the Robustness of Segmentation of Left Ventricular Structures in 2-D Echocardiography

Segmentation of cardiac structures is one of the fundamental steps to estimate volumetric indices of the heart. This step is still performed semiautomatically in clinical routine and is, thus, prone to interobserver and intraobserver variabilities. Recent studies have shown that deep learning has the potential to perform fully automatic segmentation. However, the current best solutions still suffer from a lack of robustness in terms of accuracy and number of outliers. The goal of this work is to introduce a novel network designed to improve the overall segmentation accuracy of left ventricular structures (endocardial and epicardial borders) while enhancing the estimation of the corresponding clinical indices and reducing the number of outliers. This network is based on a multistage framework where both the localization and segmentation steps are optimized jointly through an end-to-end scheme. Results obtained on a large open access data set show that our method outperforms the current best-performing deep learning solution with a lighter architecture and achieved an overall segmentation accuracy lower than the intraobserver variability for the epicardial border (i.e., on average a mean absolute error of 1.5 mm and a Hausdorff distance of 5.1mm) with 11% of outliers. Moreover, we demonstrate that our method can closely reproduce the expert analysis for the end-diastolic and end-systolic left ventricular volumes, with a mean correlation of 0.96 and a mean absolute error of 7.6 ml. Concerning the ejection fraction of the left ventricle, results are more contrasted with a mean correlation coefficient of 0.83 and an absolute mean error of 5.0%, producing scores that are slightly below the intraobserver margin. Based on this observation, areas for improvement are suggested.