Echocardiography Segmentation With Enforced Temporal Consistency

Multiple token rearrangement Transformer network with explicit superpixel constraint for segmentation of echocardiography

Diagnostic cardiologists have considerable clinical demand for precise segmentation of echocardiography to diagnose cardiovascular disease. The paradox is that manual segmentation of echocardiography is a time-consuming and operator-dependent task. Computer-aided segmentation can reduce the workflow greatly. However, it is challenging to segment multi-type echocardiography, which is reflected in differential anatomic structures, artifacts, and blurred borderline. This study proposes the multiple token rearrangement Transformer network (MTRT-Net) embedded in three novel modules to address the corresponding three challenges. First, the depthwise deformable attention module can extract flexible features to adapt to anatomic structures of echocardiography with different ages and diseases. Second, the superpixel supervised module can cluster similar features and keep discriminative features away to make the segmentation regions tend to be an entire body. The artifacts have the influence in separating the complete internal region. Third, the atrous affinity aggregation module can integrate affinity features near the borderline to judge the blurred regions. Overall, the three modules rearrange the relationships of tokens and broaden the diversity of features. Besides, the explicit constraint brought by the superpixel supervised module enhances the performance of fitting ability. This study has 13747 echocardiography to train and test the MTRT-Net. Abundant experiments also validate the performance of MTRT-Net. Therefore, MTRT-Net can assist the diagnostician in segmenting the echocardiography precisely.

M4S-Net: a motion-enhanced shape-aware semi-supervised network for echocardiography sequence segmentation

Sequence segmentation of echocardiograms is of great significance for the diagnosis and treatment of cardiovascular diseases. However, the low quality of ultrasound imaging and the complexity of cardiac motion pose great challenges to it. In addition, the difficulty and cost of labeling echocardiography sequences limit the performance of supervised learning methods. In this paper, we proposed a Motion-enhanced Shape-aware Semi-supervised Sequence Segmentation Network named M4S-Net. First, multi-level shape priors are used to enhance the model's shape representation capabilities, overcoming the low image quality and improving single-frame segmentation. Then, a motion-enhanced optimization module utilizes optical flows to assist segmentation in a geometric sense, which robustly responds to the complex motions and ensures the temporal consistency of sequence segmentation. A hybrid loss function is devised to maximize the effectiveness of each module and further improve the temporal stability of predicted masks. Furthermore, the parameter-sharing strategy allows it to perform sequence segmentation in a semi-supervised manner. Massive experiments on both public and in-house datasets show that M4S-Net outperforms the state-of-the-art methods in both spatial and temporal segmentation performance. A downstream apical rocking recognition task based on M4S-Net also achieves an AUC of 0.944, which significantly exceeds specialized physicians.

SimLVSeg: Simplifying Left Ventricular Segmentation in 2-D+Time Echocardiograms With Self- and Weakly Supervised Learning

OBJECTIVE

Achieving reliable automatic left ventricle (LV) segmentation from echocardiograms is challenging due to the inherent sparsity of annotations in the dataset, as clinicians typically only annotate two specific frames for diagnostic purposes. Here we aim to address this challenge by introducing simplified LV segmentation (SimLVSeg), a novel paradigm that enables video-based networks for consistent LV segmentation from sparsely annotated echocardiogram videos.

METHODS

SimLVSeg consists of two training stages: (i) self-supervised pre-training with temporal masking, which involves pre-training a video segmentation network by capturing the cyclic patterns of echocardiograms from largely unannotated echocardiogram frames, and (ii) weakly supervised learning tailored for LV segmentation from sparse annotations.

RESULTS

We extensively evaluated SimLVSeg using EchoNet-Dynamic, the largest echocardiography dataset. SimLVSeg outperformed state-of-the-art solutions by achieving a 93.32% (95% confidence interval: 93.21-93.43%) dice score while being more efficient. We further conducted an out-of-distribution test to showcase SimLVSeg's generalizability on distribution shifts (CAM US dataset).

CONCLUSION

Our findings show that SimLVSeg exhibits excellent performance on LV segmentation with a relatively cheaper computational cost. This suggests that adopting video-based networks for LV segmentation is a promising research direction to achieve reliable LV segmentation. Our code is publicly available at https://github.com/BioMedIA-MBZUAI/SimLVSeg.

U-shape-based network for left ventricular segmentation in echocardiograms with contrastive pretraining

Cardiovascular diseases, characterized by high morbidity, disability, and mortality rates, are a collective term for disorders affecting the heart's structure or function. In clinical practice, physicians often manually delineate the left ventricular border on echocardiograms to obtain critical physiological parameters such as left ventricular volume and ejection fraction, which are essential for accurate cardiac function assessment. However, most state-of-the-art models focus excessively on pushing the boundaries of segmentation accuracy at the expense of computational complexity, overlooking the substantial demand for high-performance computing resources required for model inference in clinical applications. This paper introduces a novel left ventricle echocardiographic segmentation model that efficiently combines the SwiftFormer Encoder and U-Lite Decoder to reduce network parameter count and computational complexity. Additionally, we incorporate the Spatial and Channel reconstruction Convolution (SCConv) module through spatial and channel reconstruction during downsampling and replace the Binary Cross Entropy Loss (BCELoss) with Polynomial Loss (PolyLoss) to achieve superior segmentation performance. On the EchoNet-Dynamic dataset, our network achieves a Dice similarity coefficient of 0.92714 for left ventricle segmentation, with FLoating-point Operations Per Second (FLOPs) and Parameters of just 4472.55 M and 28.96 M respectively. Extensive experimental results on the EchoNet-Dynamic dataset demonstrate that the proposed modifications deliver competitive performance at a lower computational cost.

Dynamic-Guided Spatiotemporal Attention for Echocardiography Video Segmentation

Left ventricle (LV) endocardium segmentation in echocardiography video has received much attention as an important step in quantifying LV ejection fraction. Most existing methods are dedicated to exploiting temporal information on top of 2D convolutional networks. In addition to single appearance semantic learning, some research attempted to introduce motion cues through the optical flow estimation (OFE) task to enhance temporal consistency modeling. However, OFE in these methods is tightly coupled to LV endocardium segmentation, resulting in noisy inter-frame flow prediction, and post-optimization based on these flows accumulates errors. To address these drawbacks, we propose dynamic-guided spatiotemporal attention (DSA) for semi-supervised echocardiography video segmentation. We first fine-tune the off-the-shelf OFE network RAFT on echocardiography data to provide dynamic information. Taking inter-frame flows as additional input, we use a dual-encoder structure to extract motion and appearance features separately. Based on the connection between dynamic continuity and semantic consistency, we propose a bilateral feature calibration module to enhance both features. For temporal consistency modeling, the DSA is proposed to aggregate neighboring frame context using deformable attention that is realized by offsets grid attention. Dynamic information is introduced into DSA through a bilateral offset estimation module to effectively combine with appearance semantics and predict attention offsets, thereby guiding semantic-based spatiotemporal attention. We evaluated our method on two popular echocardiography datasets, CAMUS and EchoNet-Dynamic, and achieved state-of-the-art.

Left Ventricular Segmentation, Warping, and Myocardial Registration for Automated Strain Measurement

The left ventricular global longitudinal strain (LVGLS) is a crucial prognostic indicator. However, inconsistencies in measurements due to the speckle tracking algorithm and manual adjustments have hindered its standardization and democratization. To solve this issue, we proposed a fully automated strain measurement by artificial intelligence-assisted LV segmentation contours. The LV segmentation model was trained from echocardiograms of 368 adults (11,125 frames). We compared the registration-like effects of dynamic time warping (DTW) with speckle tracking on a synthetic echocardiographic dataset in experiment-1. In experiment-2, we enrolled 80 patients to compare the DTW method with commercially available software. In experiment-3, we combined the segmentation model and DTW method to create the artificial intelligence (AI)-DTW method, which was then tested on 40 patients with general LV morphology, 20 with dilated cardiomyopathy (DCMP), and 20 with transthyretin-associated cardiac amyloidosis (ATTR-CA), 20 with severe aortic stenosis (AS), and 20 with severe mitral regurgitation (MR). Experiments-1 and -2 revealed that the DTW method is consistent with dedicated software. In experiment-3, the AI-DTW strain method showed comparable results for general LV morphology (bias - 0.137 ± 0.398%), DCMP (- 0.397 ± 0.607%), ATTR-CA (0.095 ± 0.581%), AS (0.334 ± 0.358%), and MR (0.237 ± 0.490%). Moreover, the strain curves showed a high correlation in their characteristics, with R-squared values of 0.8879-0.9452 for those LV morphology in experiment-3. Measuring LVGLS through dynamic warping of segmentation contour is a feasible method compared to traditional tracking techniques. This approach has the potential to decrease the need for manual demarcation and make LVGLS measurements more efficient and user-friendly for daily practice.

Real-Time Automatic M-Mode Echocardiography Measurement With Panel Attention

Motion mode (M-mode) echocardiography is essential for measuring cardiac dimension and ejection fraction. However, the current diagnosis is time-consuming and suffers from diagnosis accuracy variance. This work resorts to building an automatic scheme through well-designed and well-trained deep learning to conquer the situation. That is, we proposed RAMEM, an automatic scheme of real-time M-mode echocardiography, which contributes three aspects to address the challenges: 1) provide MEIS, the first dataset of M-mode echocardiograms, to enable consistent results and support developing an automatic scheme; For detecting objects accurately in echocardiograms, it requires big receptive field for covering long-range diastole to systole cycle. However, the limited receptive field in the typical backbone of convolutional neural networks (CNN) and the losing information risk in non-local block (NL) equipped CNN risk the accuracy requirement. Therefore, we 2) propose panel attention embedding with updated UPANets V2, a convolutional backbone network, in a real-time instance segmentation (RIS) scheme for boosting big object detection performance; 3) introduce AMEM, an efficient algorithm of automatic M-mode echocardiography measurement, for automatic diagnosis; The experimental results show that RAMEM surpasses existing RIS schemes (CNNs with NL & Transformers as the backbone) in PASCAL 2012 SBD and human performances in MEIS.

A review of evaluation approaches for explainable AI with applications in cardiology

Explainable artificial intelligence (XAI) elucidates the decision-making process of complex AI models and is important in building trust in model predictions. XAI explanations themselves require evaluation as to accuracy and reasonableness and in the context of use of the underlying AI model. This review details the evaluation of XAI in cardiac AI applications and has found that, of the studies examined, 37% evaluated XAI quality using literature results, 11% used clinicians as domain-experts, 11% used proxies or statistical analysis, with the remaining 43% not assessing the XAI used at all. We aim to inspire additional studies within healthcare, urging researchers not only to apply XAI methods but to systematically assess the resulting explanations, as a step towards developing trustworthy and safe models.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10462-024-10852-w.

Multi-granularity learning of explicit geometric constraint and contrast for label-efficient medical image segmentation and differentiable clinical function assessment

Automated segmentation is a challenging task in medical image analysis that usually requires a large amount of manually labeled data. However, most current supervised learning based algorithms suffer from insufficient manual annotations, posing a significant difficulty for accurate and robust segmentation. In addition, most current semi-supervised methods lack explicit representations of geometric structure and semantic information, restricting segmentation accuracy. In this work, we propose a hybrid framework to learn polygon vertices, region masks, and their boundaries in a weakly/semi-supervised manner that significantly advances geometric and semantic representations. Firstly, we propose multi-granularity learning of explicit geometric structure constraints via polygon vertices (PolyV) and pixel-wise region (PixelR) segmentation masks in a semi-supervised manner. Secondly, we propose eliminating boundary ambiguity by using an explicit contrastive objective to learn a discriminative feature space of boundary contours at the pixel level with limited annotations. Thirdly, we exploit the task-specific clinical domain knowledge to differentiate the clinical function assessment end-to-end. The ground truth of clinical function assessment, on the other hand, can serve as auxiliary weak supervision for PolyV and PixelR learning. We evaluate the proposed framework on two tasks, including optic disc (OD) and cup (OC) segmentation along with vertical cup-to-disc ratio (vCDR) estimation in fundus images; left ventricle (LV) segmentation at end-diastolic and end-systolic frames along with ejection fraction (LVEF) estimation in two-dimensional echocardiography images. Experiments on nine large-scale datasets of the two tasks under different label settings demonstrate our model's superior performance on segmentation and clinical function assessment.

Deep learning from latent spatiotemporal information of the heart: Identifying advanced bioimaging markers from echocardiograms

A key strength of echocardiography lies in its integration of comprehensive spatiotemporal cardiac imaging data in real-time, to aid frontline or bedside patient risk stratification and management. Nonetheless, its acquisition, processing, and interpretation are known to all be subject to heterogeneity from its reliance on manual and subjective human tracings, which challenges workflow and protocol standardization and final interpretation accuracy. In the era of advanced computational power, utilization of machine learning algorithms for big data analytics in echocardiography promises reduction in cost, cognitive errors, and intra- and inter-observer variability. Novel spatiotemporal deep learning (DL) models allow the integration of temporal arm information based on unlabeled pixel echocardiographic data for convolution of an adaptive semantic spatiotemporal calibration to construct personalized 4D heart meshes, assess global and regional cardiac function, detect early valve pathology, and differentiate uncommon cardiovascular disorders. Meanwhile, data visualization on spatiotemporal DL prediction models helps extract latent temporal imaging features to develop advanced imaging biomarkers in early disease stages and advance our understanding of pathophysiology to support the development of personalized prevention or treatment strategies. Since portable echocardiograms have been increasingly used as point-of-care imaging tools to aid rural care delivery, the application of these new spatiotemporal DL techniques show the potentials in streamlining echocardiographic acquisition, processing, and data analysis to improve workflow standardization and efficiencies, and provide risk stratification and decision supporting tools in real-time, to prompt the building of new imaging diagnostic networks to enhance rural healthcare engagement.

Automated Measurement of Ovary Development in Atlantic Salmon Using Deep Learning

OBJECTIVE

Salmon breeding companies control the egg stripping period through environmental change, which triggers the need to identify the state of maturation. Ultrasound imaging of the salmon ovary is a proven non-invasive tool for this purpose; however, the process is laborious, and the interpretation of the ultrasound scans is subjective. Real-time ultrasound image segmentation of Atlantic salmon ovary provides an opportunity to overcome these limitations. However, several application challenges need to be addressed to achieve this goal. These challenges include the potential for false-positive and false-negative predictions, accurate prediction of attenuated lower ovary parts and resolution of inconsistencies in predicted ovary shape.

METHODS

We describe an approach designed to tackle these obstacles by employing targeted pre-training of a modified U-Net, capable of performing both segmentation and classification. In addition, a variational autoencoder (VAE) and generative adversarial network (GAN) were incorporated to rectify shape inconsistencies in the segmentation output. To train the proposed model, a data set of Atlantic salmon ovaries throughout two maturation periods was recorded.

RESULTS

We then tested our model and compared its performance with that of conventional and novel U-Nets. The method was also tested in a salmon on-site ultrasound examination setting. The results of our application indicate that our method is able to efficiently segment salmon ovary with an average Dice score of 0.885 per individual in real-time.

CONCLUSION

These results represent a competitive performance for this specific application, which enables us to design an automated system for smart monitoring of maturation state in Atlantic salmon.

Enhancing Arrhythmogenic Right Ventricular Cardiomyopathy Detection and Risk Stratification: Insights from Advanced Echocardiographic Techniques

INTRODUCTION

The echocardiographic diagnosis criteria for arrhythmogenic right ventricular cardiomyopathy (ARVC) are highly specific but sensitivity is low, especially in the early stages of the disease. The role of echocardiographic strain in ARVC has not been fully elucidated, although prior studies suggest that it can improve the detection of subtle functional abnormalities. The purposes of the study were to determine whether these advanced measures of right ventricular (RV) dysfunction on echocardiogram, including RV strain, increase diagnostic value for ARVC disease detection and to evaluate the association of echocardiographic parameters with arrhythmic outcomes.

METHODS

The study included 28 patients from the Heart Institute of São Paulo ARVC cohort with a definite diagnosis of ARVC established according to the 2010 Task Force Criteria. All patients were submitted to ECHO's advanced techniques including RV strain, and the parameters were compared to prior conventional visual ECHO and CMR.

RESULTS

In total, 28 patients were enrolled in order to perform ECHO's advanced techniques. A total of 2/28 (7%) patients died due to a cardiovascular cause, 2/28 (7%) underwent heart transplantation, and 14/28 (50%) patients developed sustained ventricular arrhythmic events. Among ECHO's parameters, RV dilatation, measured by RVDd (p = 0.018) and RVOT PSAX (p = 0.044), was significantly associated with arrhythmic outcomes. RV free wall longitudinal strain < 14.35% in absolute value was associated with arrhythmic outcomes (p = 0.033).

CONCLUSION

Our data suggest that ECHO's advanced techniques improve ARVC detection and that abnormal RV strain can be associated with arrhythmic risk stratification. Further studies are necessary to better demonstrate these findings and contribute to risk stratification in ARVC, in addition to other well-known risk markers.

Explainable Artificial Intelligence and Cardiac Imaging: Toward More Interpretable Models

Artificial intelligence applications have shown success in different medical and health care domains, and cardiac imaging is no exception. However, some machine learning models, especially deep learning, are considered black box as they do not provide an explanation or rationale for model outcomes. Complexity and vagueness in these models necessitate a transition to explainable artificial intelligence (XAI) methods to ensure that model results are both transparent and understandable to end users. In cardiac imaging studies, there are a limited number of papers that use XAI methodologies. This article provides a comprehensive literature review of state-of-the-art works using XAI methods for cardiac imaging. Moreover, it provides simple and comprehensive guidelines on XAI. Finally, open issues and directions for XAI in cardiac imaging are discussed.

Cross-Domain Echocardiography Segmentation with Multi-Space Joint Adaptation

The segmentation of the left ventricle endocardium (LVendo) and the left ventricle epicardium (LVepi) in echocardiography plays an important role in clinical diagnosis. Recently, deep neural networks have been the most commonly used approach for echocardiography segmentation. However, the performance of a well-trained segmentation network may degrade in unseen domain datasets due to the distribution shift of the data. Adaptation algorithms can improve the generalization of deep neural networks to different domains. In this paper, we present a multi-space adaptation-segmentation-joint framework, named MACS, for cross-domain echocardiography segmentation. It adopts a generative adversarial architecture; the generator fulfills the segmentation task and the multi-space discriminators align the two domains on both the feature space and output space. We evaluated the MACS method on two echocardiography datasets from different medical centers and vendors, the publicly available CAMUS dataset and our self-acquired dataset. The experimental results indicated that the MACS could handle unseen domain datasets well, without requirements for manual annotations, and improve the generalization performance by 2.2% in the Dice metric.

MITEA: A dataset for machine learning segmentation of the left ventricle in 3D echocardiography using subject-specific labels from cardiac magnetic resonance imaging

Segmentation of the left ventricle (LV) in echocardiography is an important task for the quantification of volume and mass in heart disease. Continuing advances in echocardiography have extended imaging capabilities into the 3D domain, subsequently overcoming the geometric assumptions associated with conventional 2D acquisitions. Nevertheless, the analysis of 3D echocardiography (3DE) poses several challenges associated with limited spatial resolution, poor contrast-to-noise ratio, complex noise characteristics, and image anisotropy. To develop automated methods for 3DE analysis, a sufficiently large, labeled dataset is typically required. However, ground truth segmentations have historically been difficult to obtain due to the high inter-observer variability associated with manual analysis. We address this lack of expert consensus by registering labels derived from higher-resolution subject-specific cardiac magnetic resonance (CMR) images, producing 536 annotated 3DE images from 143 human subjects (10 of which were excluded). This heterogeneous population consists of healthy controls and patients with cardiac disease, across a range of demographics. To demonstrate the utility of such a dataset, a state-of-the-art, self-configuring deep learning network for semantic segmentation was employed for automated 3DE analysis. Using the proposed dataset for training, the network produced measurement biases of -9 ± 16 ml, -1 ± 10 ml, -2 ± 5 %, and 5 ± 23 g, for end-diastolic volume, end-systolic volume, ejection fraction, and mass, respectively, outperforming an expert human observer in terms of accuracy as well as scan-rescan reproducibility. As part of the Cardiac Atlas Project, we present here a large, publicly available 3DE dataset with ground truth labels that leverage the higher resolution and contrast of CMR, to provide a new benchmark for automated 3DE analysis. Such an approach not only reduces the effect of observer-specific bias present in manual 3DE annotations, but also enables the development of analysis techniques which exhibit better agreement with CMR compared to conventional methods. This represents an important step for enabling more efficient and accurate diagnostic and prognostic information to be obtained from echocardiography.

Left ventricle analysis in echocardiographic images using transfer learning

The segmentation of cardiac boundaries, specifically Left Ventricle (LV) segmentation in 2D echocardiographic images, is a critical step in LV segmentation and cardiac function assessment. These images are generally of poor quality and present low contrast, making daily clinical delineation difficult, time-consuming, and often inaccurate. Thus, it is necessary to design an intelligent automatic endocardium segmentation system. The present work aims to examine and assess the performance of some deep learning-based architectures such as U-Net1, U-Net2, LinkNet, Attention U-Net, and TransUNet using the public CAMUS (Cardiac Acquisitions for Multi-structure Ultrasound Segmentation) dataset. The adopted approach emphasizes the advantage of using transfer learning and resorting to pre-trained backbones in the encoder part of a segmentation network for echocardiographic image analysis. The experimental findings indicated that the proposed framework with the [Formula: see text]-[Formula: see text] is quite promising; it outperforms other more recent approaches with a Dice similarity coefficient of 93.30% and a Hausdorff Distance of 4.01 mm. In addition, a good agreement between the clinical indices calculated from the automatic segmentation and those calculated from the ground truth segmentation. For instance, the mean absolute errors for the left ventricular end-diastolic volume, end-systolic volume, and ejection fraction are equal to 7.9 ml, 5.4 ml, and 6.6%, respectively. These results are encouraging and point out additional perspectives for further improvement.