Machine Learning Approaches for Myocardial Motion and Deformation Analysis

Improving the efficiency and accuracy of cardiovascular magnetic resonance with artificial intelligence-review of evidence and proposition of a roadmap to clinical translation

BACKGROUND

Cardiovascular magnetic resonance (CMR) is an important imaging modality for the assessment of heart disease; however, limitations of CMR include long exam times and high complexity compared to other cardiac imaging modalities. Recently advancements in artificial intelligence (AI) technology have shown great potential to address many CMR limitations. While the developments are remarkable, translation of AI-based methods into real-world CMR clinical practice remains at a nascent stage and much work lies ahead to realize the full potential of AI for CMR.

METHODS

Herein we review recent cutting-edge and representative examples demonstrating how AI can advance CMR in areas such as exam planning, accelerated image reconstruction, post-processing, quality control, classification and diagnosis.

RESULTS

These advances can be applied to speed up and simplify essentially every application including cine, strain, late gadolinium enhancement, parametric mapping, 3D whole heart, flow, perfusion and others. AI is a unique technology based on training models using data. Beyond reviewing the literature, this paper discusses important AI-specific issues in the context of CMR, including (1) properties and characteristics of datasets for training and validation, (2) previously published guidelines for reporting CMR AI research, (3) considerations around clinical deployment, (4) responsibilities of clinicians and the need for multi-disciplinary teams in the development and deployment of AI in CMR, (5) industry considerations, and (6) regulatory perspectives.

CONCLUSIONS

Understanding and consideration of all these factors will contribute to the effective and ethical deployment of AI to improve clinical CMR.

DeepMesh: Mesh-Based Cardiac Motion Tracking Using Deep Learning

3D motion estimation from cine cardiac magnetic resonance (CMR) images is important for the assessment of cardiac function and the diagnosis of cardiovascular diseases. Current state-of-the art methods focus on estimating dense pixel-/voxel-wise motion fields in image space, which ignores the fact that motion estimation is only relevant and useful within the anatomical objects of interest, e.g., the heart. In this work, we model the heart as a 3D mesh consisting of epi- and endocardial surfaces. We propose a novel learning framework, DeepMesh, which propagates a template heart mesh to a subject space and estimates the 3D motion of the heart mesh from CMR images for individual subjects. In DeepMesh, the heart mesh of the end-diastolic frame of an individual subject is first reconstructed from the template mesh. Mesh-based 3D motion fields with respect to the end-diastolic frame are then estimated from 2D short- and long-axis CMR images. By developing a differentiable mesh-to-image rasterizer, DeepMesh is able to leverage 2D shape information from multiple anatomical views for 3D mesh reconstruction and mesh motion estimation. The proposed method estimates vertex-wise displacement and thus maintains vertex correspondences between time frames, which is important for the quantitative assessment of cardiac function across different subjects and populations. We evaluate DeepMesh on CMR images acquired from the UK Biobank. We focus on 3D motion estimation of the left ventricle in this work. Experimental results show that the proposed method quantitatively and qualitatively outperforms other image-based and mesh-based cardiac motion tracking methods.

Automated segmentation of the left-ventricle from MRI with a fully convolutional network to investigate CTRCD in breast cancer patients

Purpose: The goal of this study was to develop a fully convolutional network (FCN) tool to automatedly segment the left-ventricular (LV) myocardium in displacement encoding with stimulated echoes MRI. The segmentation results are used for LV chamber quantification and strain analyses in breast cancer patients susceptible to cancer therapy-related cardiac dysfunction (CTRCD). Approach: A DeepLabV3+ FCN with a ResNet-101 backbone was custom-designed to conduct chamber quantification on 45 female breast cancer datasets (23 training, 11 validation, and 11 test sets). LV structural parameters and LV ejection fraction (LVEF) were measured, and myocardial strains estimated with the radial point interpolation method. Myocardial classification validation was against quantization-based ground-truth with computations of accuracy, Dice score, average perpendicular distance (APD), Hausdorff-distance, and others. Additional validations were conducted with equivalence tests and Cronbach's alpha (C - α ) intraclass correlation coefficients between the FCN and a vendor tool on chamber quantification and myocardial strain computations. Results: Myocardial classification results against ground-truth were Dice = 0.89 , APD = 2.4    mm , and accuracy = 97 % for the validation set and Dice = 0.90 , APD = 2.5    mm , and accuracy = 97 % for the test set. The confidence intervals (CI) and two one-sided t-test results of equivalence tests between the FCN and vendor-tool were CI = - 1.36 % to 2.42%, p-value < 0.001 for LVEF (58 ± 5 % versus 57 ± 6 % ), and CI = - 0.71 % to 0.63%, p-value < 0.001 for longitudinal strain (- 15 ± 2 % versus - 15 ± 3 % ). Conclusions: The validation results were found equivalent to the vendor tool-based parameter estimates, which show that accurate LV chamber quantification followed by strain analysis for CTRCD investigation can be achieved with our proposed FCN methodology.

Machine Learning in Cardio-Oncology: New Insights from an Emerging Discipline

A growing body of evidence on a wide spectrum of adverse cardiac events following oncologic therapies has led to the emergence of cardio-oncology as an increasingly relevant interdisciplinary specialty. This also calls for better risk-stratification for patients undergoing cancer treatment. Machine learning (ML), a popular branch discipline of artificial intelligence that tackles complex big data problems by identifying interaction patterns among variables, has seen increasing usage in cardio-oncology studies for risk stratification. The objective of this comprehensive review is to outline the application of ML approaches in cardio-oncology, including deep learning, artificial neural networks, random forest and summarize the cardiotoxicity identified by ML. The current literature shows that ML has been applied for the prediction, diagnosis and treatment of cardiotoxicity in cancer patients. In addition, role of ML in gender and racial disparities for cardiac outcomes and potential future directions of cardio-oncology are discussed. It is essential to establish dedicated multidisciplinary teams in the hospital and educate medical professionals to become familiar and proficient in ML in the future.

Pixel-wise statistical analysis of myocardial injury in STEMI patients with delayed enhancement MRI

Objectives

Myocardial injury assessment from delayed enhancement magnetic resonance images is routinely limited to global descriptors such as size and transmurality. Statistical tools from computational anatomy can drastically improve this characterization, and refine the assessment of therapeutic procedures aiming at infarct size reduction. Based on these techniques, we propose a new characterization of myocardial injury up to the pixel resolution. We demonstrate it on the imaging data from the Minimalist Immediate Mechanical Intervention randomized clinical trial (MIMI: NCT01360242), which aimed at comparing immediate and delayed stenting in acute ST-Elevation Myocardial Infarction (STEMI) patients.

Methods

We analyzed 123 patients from the MIMI trial (62 ± 12 years, 98 male, 65 immediate 58 delayed stenting). Early and late enhancement images were transported onto a common geometry using techniques inspired by statistical atlases, allowing pixel-wise comparisons across population subgroups. A practical visualization of lesion patterns against specific clinical and therapeutic characteristics was also proposed using state-of-the-art dimensionality reduction.

Results

Infarct patterns were roughly comparable between the two treatments across the whole myocardium. Subtle but significant local differences were observed for the LCX and RCA territories with higher transmurality for delayed stenting at lateral and inferior/inferoseptal locations, respectively (15% and 23% of myocardial locations with a p-value <0.05, mainly in these regions). In contrast, global measurements were comparable for all territories (no statistically significant differences for all-except-one measurements before standardization / for all after standardization), although immediate stenting resulted in more subjects without reperfusion injury.

Conclusion

Our approach substantially empowers the analysis of lesion patterns with standardized comparisons up to the pixel resolution, and may reveal subtle differences not accessible with global observations. On the MIMI trial data as illustrative case, it confirmed its general conclusions regarding the lack of benefit of delayed stenting, but revealed subgroups differences thanks to the standardized and finer analysis scale.

Generative myocardial motion tracking via latent space exploration with biomechanics-informed prior

Myocardial motion and deformation are rich descriptors that characterize cardiac function. Image registration, as the most commonly used technique for myocardial motion tracking, is an ill-posed inverse problem which often requires prior assumptions on the solution space. In contrast to most existing approaches which impose explicit generic regularization such as smoothness, in this work we propose a novel method that can implicitly learn an application-specific biomechanics-informed prior and embed it into a neural network-parameterized transformation model. Particularly, the proposed method leverages a variational autoencoder-based generative model to learn a manifold for biomechanically plausible deformations. The motion tracking then can be performed via traversing the learnt manifold to search for the optimal transformations while considering the sequence information. The proposed method is validated on three public cardiac cine MRI datasets with comprehensive evaluations. The results demonstrate that the proposed method can outperform other approaches, yielding higher motion tracking accuracy with reasonable volume preservation and better generalizability to varying data distributions. It also enables better estimates of myocardial strains, which indicates the potential of the method in characterizing spatiotemporal signatures for understanding cardiovascular diseases.

Forecasting the Applied Deep Learning Tools in Enhancing Food Quality for Heart Related Diseases Effectively: A Study Using Structural Equation Model Analysis

The term heart-related disease is stated as the range of condition that impacts an individual heart negatively. In the current scenario, cardiovascular diseases are causing more deaths when compared with other ailments, it has been estimated that there are nearly 18 million deaths annually as per the recent report released by World Health Organization (WHO). It has been stated that unhealthy habits and other related aspects adopted by individuals are considered as the primary reasons for an increase in the risk of heart diseases. High cholesterol, eating more junk foods, hypertension, etc., created the issue related to heart diseases. Hence, addressing food quality and suggesting better eating habits enable individuals to enhance their living and support better health. The application of new technologies like machine learning, deep learning, and other models support doctors, nurses, and radiologists to predict heart disease effectively. Studies have stated that the various models are used mainly for the classification and forecasting of the diagnosis of heart-related diseases. The researchers have identified that critical algorithms like CART support the predictability of the disease by 93.3% whereas the conventional models possess vert less specificity. Furthermore, deep neural networks can be applied for analyzing and detecting heart failures effectively and supporting medical practitioners in making better and more critical clinical decisions making. The researchers focus on using a descriptive research study for performing the study; moreover, the researcher collates the data using the questionnaire method, which enables sourcing the critical information from the medical practitioners and supports in making critical data analysis effectively. The researchers also use secondary data modes for sourcing the information related to past studies on the related topic. The researchers use the frequency analysis, correlation analysis, and structural equation model analysis for performing the study, and the results are stated in detail in the respective sections.

Motion Estimation by Deep Learning in 2D Echocardiography: Synthetic Dataset and Validation

Motion estimation in echocardiography plays an important role in the characterization of cardiac function, allowing the computation of myocardial deformation indices. However, there exist limitations in clinical practice, particularly with regard to the accuracy and robustness of measurements extracted from images. We therefore propose a novel deep learning solution for motion estimation in echocardiography. Our network corresponds to a modified version of PWC-Net which achieves high performance on ultrasound sequences. In parallel, we designed a novel simulation pipeline allowing the generation of a large amount of realistic B-mode sequences. These synthetic data, together with strategies during training and inference, were used to improve the performance of our deep learning solution, which achieved an average endpoint error of 0.07 ± 0.06 mm per frame and 1.20 ± 0.67 mm between ED and ES on our simulated dataset. The performance of our method was further investigated on 30 patients from a publicly available clinical dataset acquired from a GE system. The method showed promise by achieving a mean absolute error of the global longitudinal strain of 2.5 ± 2.1% and a correlation of 0.77 compared to GLS derived from manual segmentation, much better than one of the most efficient methods in the state-of-the-art (namely the FFT-Xcorr block-matching method). We finally evaluated our method on an auxiliary dataset including 30 patients from another center and acquired with a different system. Comparable results were achieved, illustrating the ability of our method to maintain high performance regardless of the echocardiographic data processed.

Myocardial strain analysis of echocardiography based on deep learning

BACKGROUND

Strain analysis provides more thorough spatiotemporal signatures for myocardial contraction, which is helpful for early detection of cardiac insufficiency. The use of deep learning (DL) to automatically measure myocardial strain from echocardiogram videos has garnered recent attention. However, the development of key techniques including segmentation and motion estimation remains a challenge. In this work, we developed a novel DL-based framework for myocardial segmentation and motion estimation to generate strain measures from echocardiogram videos.

METHODS

Three-dimensional (3D) Convolutional Neural Network (CNN) was developed for myocardial segmentation and optical flow network for motion estimation. The segmentation network was used to define the region of interest (ROI), and the optical flow network was used to estimate the pixel motion in the ROI. We performed a model architecture search to identify the optimal base architecture for motion estimation. The final workflow design and associated hyperparameters are the result of a careful implementation. In addition, we compared the DL model with a traditional speck tracking algorithm on an independent, external clinical data. Each video was double-blind measured by an ultrasound expert and a DL expert using speck tracking echocardiography (STE) and DL method, respectively.

RESULTS

The DL method successfully performed automatic segmentation, motion estimation, and global longitudinal strain (GLS) measurements in all examinations. The 3D segmentation has better spatio-temporal smoothness, average dice correlation reaches 0.82, and the effect of target frame is better than that of previous 2D networks. The best motion estimation network achieved an average end-point error of 0.05 ± 0.03 mm per frame, better than previously reported state-of-the-art. The DL method showed no significant difference relative to the traditional method in GLS measurement, Spearman correlation of 0.90 (p < 0.001) and mean bias -1.2 ± 1.5%.

CONCLUSION

In conclusion, our method exhibits better segmentation and motion estimation performance and demonstrates the feasibility of DL method for automatic strain analysis. The DL approach helps reduce time consumption and human effort, which holds great promise for translational research and precision medicine efforts.

Direct left-ventricular global longitudinal strain (GLS) computation with a fully convolutional network

This study's purpose was to develop a direct MRI-based, deep-learning semantic segmentation approach for computing global longitudinal strain (GLS), a known metric for detecting left-ventricular (LV) cardiotoxicity in breast cancer. Displacement Encoding with Stimulated Echoes cardiac image phases acquired from 30 breast cancer patients and 30 healthy females were unwrapped via a DeepLabV3 + fully convolutional network (FCN). Myocardial strains were directly computed from the unwrapped phases with the Radial Point Interpolation Method. FCN-unwrapped phases of a phantom's rotating gel were validated against quality-guided phase-unwrapping (QGPU) and robust transport of intensity equation (RTIE) phase-unwrapping. FCN performance on unwrapping human LV data was measured with F1 and Dice scores versus QGPU ground-truth. The reliability of FCN-based strains was assessed against RTIE-based strains with Cronbach's alpha (C-α) intraclass correlation coefficient. Mean squared error (MSE) of unwrapping the phantom experiment data at 0 dB signal-to-noise ratio were 1.6, 2.7 and 6.1 with FCN, QGPU and RTIE techniques. Human data classification accuracies were F1 = 0.95 (Dice = 0.96) with FCN and F1 = 0.94 (Dice = 0.95) with RTIE. GLS results from FCN and RTIE were -16 ± 3% vs. -16 ± 3% (C-α = 0.9) for patients and -20 ± 3% vs. -20 ± 3% (C-α = 0.9) for healthy subjects. The low MSE from the phantom validation demonstrates accuracy of phase-unwrapping with the FCN and comparable human subject results versus RTIE demonstrate GLS analysis accuracy. A deep-learning methodology for phase-unwrapping in medical images and GLS computation was developed and validated in a heterogeneous cohort.

Characterizing interactions between cardiac shape and deformation by non-linear manifold learning

In clinical routine, high-dimensional descriptors of the cardiac function such as shape and deformation are reduced to scalars (e.g. volumes or ejection fraction), which limit the characterization of complex diseases. Besides, these descriptors undergo interactions depending on disease, which may bias their computational analysis. In this paper, we aim at characterizing such interactions by unsupervised manifold learning. We propose to use a sparsified version of Multiple Manifold Learning to align the latent spaces encoding each descriptor and weighting the strength of the alignment depending on each pair of samples. While this framework was up to now only applied to link different datasets from the same manifold, we demonstrate its relevance to characterize the interactions between different but partially related descriptors of the cardiac function (shape and deformation). We benchmark our approach against linear and non-linear embedding strategies, among which the fusion of manifolds by Multiple Kernel Learning, the independent embedding of each descriptor by Diffusion Maps, and a strict alignment based on pairwise correspondences. We first evaluated the methods on a synthetic dataset from a 0D cardiac model where the interactions between descriptors are fully controlled. Then, we transfered them to a population of right ventricular meshes from 310 subjects (100 healthy and 210 patients with right ventricular disease) obtained from 3D echocardiography, where the link between shape and deformation is key for disease understanding. Our experiments underline the relevance of jointly considering shape and deformation descriptors, and that manifold alignment is preferable over fusion for our application. They also confirm at a finer scale the characteristic traits of the right ventricular diseases in our population.

Applications of artificial intelligence in cardiovascular imaging

Research into artificial intelligence (AI) has made tremendous progress over the past decade. In particular, the AI-powered analysis of images and signals has reached human-level performance in many applications owing to the efficiency of modern machine learning methods, in particular deep learning using convolutional neural networks. Research into the application of AI to medical imaging is now very active, especially in the field of cardiovascular imaging because of the challenges associated with acquiring and analysing images of this dynamic organ. In this Review, we discuss the clinical questions in cardiovascular imaging that AI can be used to address and the principal methodological AI approaches that have been developed to solve the related image analysis problems. Some approaches are purely data-driven and rely mainly on statistical associations, whereas others integrate anatomical and physiological information through additional statistical, geometric and biophysical models of the human heart. In a structured manner, we provide representative examples of each of these approaches, with particular attention to the underlying computational imaging challenges. Finally, we discuss the remaining limitations of AI approaches in cardiovascular imaging (such as generalizability and explainability) and how they can be overcome.

Myocardial Function Imaging in Echocardiography Using Deep Learning

Deformation imaging in echocardiography has been shown to have better diagnostic and prognostic value than conventional anatomical measures such as ejection fraction. However, despite clinical availability and demonstrated efficacy, everyday clinical use remains limited at many hospitals. The reasons are complex, but practical robustness has been questioned, and a large inter-vendor variability has been demonstrated. In this work, we propose a novel deep learning based framework for motion estimation in echocardiography, and use this to fully automate myocardial function imaging. A motion estimator was developed based on a PWC-Net architecture, which achieved an average end point error of (0.06±0.04) mm per frame using simulated data from an open access database, on par or better compared to previously reported state of the art. We further demonstrate unique adaptability to image artifacts such as signal dropouts, made possible using trained models that incorporate relevant image augmentations. Further, a fully automatic pipeline consisting of cardiac view classification, event detection, myocardial segmentation and motion estimation was developed and used to estimate left ventricular longitudinal strain in vivo. The method showed promise by achieving a mean deviation of (-0.7±1.6)% compared to a semi-automatic commercial solution for N=30 patients with relevant disease, within the expected limits of agreement. We thus believe that learning-based motion estimation can facilitate extended use of strain imaging in clinical practice.

Validation of a deep-learning semantic segmentation approach to fully automate MRI-based left-ventricular deformation analysis in cardiotoxicity

OBJECTIVE

Left-ventricular (LV) strain measurements with the Displacement Encoding with Stimulated Echoes (DENSE) MRI sequence provide accurate estimates of cardiotoxicity damage related to chemotherapy for breast cancer. This study investigated an automated and supervised deep convolutional neural network (DCNN) model for LV chamber quantification before strain analysis in DENSE images.

METHODS

The DeepLabV3 +DCNN with three versions of ResNet-50 backbone was designed to conduct chamber quantification on 42 female breast cancer data sets. The convolutional layers in the three ResNet-50 backbones were varied as non-atrous, atrous and modified, atrous with accuracy improvements like using Laplacian of Gaussian filters. Parameters such as LV end-diastolic diameter (LVEDD) and ejection fraction (LVEF) were quantified, and myocardial strains analyzed with the Radial Point Interpolation Method (RPIM). Myocardial classification was validated with the performance metrics of accuracy, Dice, average perpendicular distance (APD) and others. Repeated measures ANOVA and intraclass correlation (ICC) with Cronbach's α (C-Alpha) tests were conducted between the three DCNNs and a vendor tool on chamber quantification and myocardial strain analysis.

RESULTS

Validation results in the same test-set for myocardial classification were accuracy = 97%, Dice = 0.92, APD = 1.2 mm with the modified ResNet-50, and accuracy = 95%, Dice = 0.90, APD = 1.7 mm with the atrous ResNet-50. The ICC results between the modified ResNet-50, atrous ResNet-50 and vendor-tool were C-Alpha = 0.97 for LVEF (55±7%, 54±7%, 54±7%, p = 0.6), and C-Alpha = 0.87 for LVEDD (4.6 ± 0.3 cm, 4.6 ± 0.3 cm, 4.6 ± 0.4 cm, p = 0.7).

CONCLUSION

Similar performance metrics and equivalent parameters obtained from comparisons between the atrous networks and vendor tool show that segmentation with the modified, atrous DCNN is applicable for automated LV chamber quantification and subsequent strain analysis in cardiotoxicity.

ADVANCES IN KNOWLEDGE

A novel deep-learning technique for segmenting DENSE images was developed and validated for LV chamber quantification and strain analysis in cardiotoxicity detection.

A deep-learning semantic segmentation approach to fully automated MRI-based left-ventricular deformation analysis in cardiotoxicity

Left-ventricular (LV) strain measurements with the Displacement Encoding with Stimulated Echoes (DENSE) MRI sequence provide accurate estimates of cardiotoxicity damage related to breast cancer chemotherapy. This study investigated an automated LV chamber quantification tool via segmentation with a supervised deep convolutional neural network (DCNN) before strain analysis with DENSE images. Segmentation for chamber quantification analysis was conducted with a custom DeepLabV3+ DCNN with ResNet-50 backbone on 42 female breast cancer datasets (22 training-sets, eight validation-sets and 12 independent test-sets). Parameters such as LV end-diastolic diameter (LVEDD) and ejection fraction (LVEF) were quantified, and myocardial strains analyzed with the Radial Point Interpolation Method (RPIM). Myocardial classification was validated against ground-truth with sensitivity-specificity analysis, the metrics of Dice, average perpendicular distance (APD) and Hausdorff-distance. Following segmentation, validation was conducted with the Cronbach's Alpha (C-Alpha) intraclass correlation coefficient between LV chamber quantification results with DENSE and Steady State Free Precession (SSFP) acquisitions and a vendor tool-based method to segment the DENSE data, and similarly for myocardial strain analysis in the chambers. The results of myocardial classification from segmentation of the DENSE data were accuracy = 97%, Dice = 0.89 and APD = 2.4 mm in the test-set. The C-Alpha correlations from comparing chamber quantification results between the segmented DENSE and SSFP data and vendor tool-based method were 0.97 for LVEF (56 ± 7% vs 55 ± 7% vs 55 ± 6%, p = 0.6) and 0.77 for LVEDD (4.6 ± 0.4 cm vs 4.5 ± 0.3 cm vs 4.5 ± 0.3 cm, p = 0.8). The validation metrics against ground-truth and equivalent parameters obtained from the SSFP segmentation and vendor tool-based comparisons show that the DCNN approach is applicable for automated LV chamber quantification and subsequent strain analysis in cardiotoxicity.