Deep learning evaluation of biomarkers from echocardiogram videos

Opportunistic Screening of Chronic Liver Disease with Deep Learning Enhanced Echocardiography

Importance

Chronic liver disease affects more than 1.5 billion adults worldwide, however the majority of cases are asymptomatic and undiagnosed. Echocardiography is broadly performed and visualizes the liver; but this information is not leveraged.

Objective

To develop and evaluate a deep learning algorithm on echocardiography videos to enable opportunistic screening for chronic liver disease.

Design

Retrospective observational cohorts.

Setting

Two large urban academic medical centers.

Participants

Adult patients who received echocardiography and abdominal imaging (either abdominal ultrasound or abdominal magnetic resonance imaging) with ≤30 days between tests, between July 4, 2012, to June 4, 2022.

Exposure

Deep learning model predictions from a deep-learning computer vision pipeline that identifies subcostal view echocardiogram videos and detects the presence of cirrhosis or steatotic liver disease (SLD).

Main Outcome and Measures

Clinical diagnosis by paired abdominal ultrasound or magnetic resonance imaging (MRI).

Results

A total of 1,596,640 echocardiogram videos (66,922 studies from 24,276 patients) from Cedars-Sinai Medical Center (CSMC) were used to develop EchoNet-Liver, an automated pipeline that identifies high quality subcostal images from echocardiogram studies and detects the presence of cirrhosis or SLD. In the held-out CSMC test cohort, EchoNet-Liver was able to detect the presence of cirrhosis with an AUC of 0.837 (0.789 - 0.880) and SLD with an AUC of 0.799 (0.758 - 0.837). In a separate test cohort with paired abdominal MRIs, cirrhosis was detected with an AUC of 0.704 (0.689-0.718) and SLD was detected with an AUC of 0.726 (0.659-0.790). In an external test cohort of 106 patients (n = 5,280 videos), the model detected cirrhosis with an AUC of 0.830 (0.738 - 0.909) and SLD with an AUC of 0.768 (0.652 - 0.875).

Conclusions and Relevance

Deep learning assessment of clinical echocardiography enables opportunistic screening of SLD and cirrhosis. Application of this algorithm may identify patients who may benefit from further diagnostic testing and treatment for chronic liver disease.

Broadening Perspectives of Artificial Intelligence in Echocardiography

Echocardiography frequently serves as the first-line treatment of diagnostic imaging for several pathological entities in cardiology. Artificial intelligence (AI) has been growing substantially in information technology and various commercial industries. Machine learning (ML), a branch of AI, has been shown to expand the capabilities and potential of echocardiography. ML algorithms expand the field of echocardiography by automated assessment of the ejection fraction and left ventricular function, integrating novel approaches such as speckle tracking or tissue Doppler echocardiography or vector flow mapping, improved phenotyping, distinguishing between cardiac conditions, and incorporating information from mobile health and genomics. In this review article, we assess the impact of AI and ML in echocardiography.

Overview and Clinical Applications of Artificial Intelligence and Machine Learning in Cardiac Anesthesiology

Artificial intelligence- (AI) and machine learning (ML)-based applications are becoming increasingly pervasive in the healthcare setting. This has in turn challenged clinicians, hospital administrators, and health policymakers to understand such technologies and develop frameworks for safe and sustained clinical implementation. Within cardiac anesthesiology, challenges and opportunities for AI/ML to support patient care are presented by the vast amounts of electronic health data, which are collected rapidly, interpreted, and acted upon within the periprocedural area. To address such challenges and opportunities, in this article, the authors review 3 recent applications relevant to cardiac anesthesiology, including depth of anesthesia monitoring, operating room resource optimization, and transthoracic/transesophageal echocardiography, as conceptual examples to explore strengths and limitations of AI/ML within healthcare, and characterize this evolving landscape. Through reviewing such applications, the authors introduce basic AI/ML concepts and methodologies, as well as practical considerations and ethical concerns for initiating and maintaining safe clinical implementation of AI/ML-based algorithms for cardiac anesthesia patient care.

Using Deep learning to Predict Cardiovascular Magnetic Resonance Findings from Echocardiography Videos

Background

Echocardiography is the most common modality for assessing cardiac structure and function. While cardiac magnetic resonance (CMR) imaging is less accessible, CMR can provide unique tissue characterization including late gadolinium enhancement (LGE), T1 and T2 mapping, and extracellular volume (ECV) which are associated with tissue fibrosis, infiltration, and inflammation. While deep learning has been shown to uncover findings not recognized by clinicians, it is unknown whether CMR-based tissue characteristics can be derived from echocardiography videos using deep learning. We hypothesized that deep learning applied to echocardiography could predict CMR-based measurements.

Methods

In a retrospective single-center study, adult patients with CMRs and echocardiography studies within 30 days were included. A video-based convolutional neural network was trained on echocardiography videos to predict CMR-derived labels including wall motion abnormality (WMA) presence, LGE presence, and abnormal T1, T2 or ECV across echocardiography views. The model performance was evaluated in a held-out test dataset not used for training.

Results

The study population included 1,453 adult patients (mean age 56±18 years, 42% female) with 2,556 paired echocardiography studies occurring on average 2 days after CMR (interquartile range 2 days prior to 6 days after). The model had high predictive capability for presence of WMA (AUC 0.873 [95%CI 0.816-0.922]), however, the model was unable to reliably detect the presence of LGE (AUC 0.699 [0.613-0.780]), native T1 (AUC 0.614 [0.500-0.715]), T2 0.553 [0.420-0.692], or ECV 0.564 [0.455-0.691]).

Conclusions

Deep learning applied to echocardiography accurately identified CMR-based WMA, but was unable to predict tissue characteristics, suggesting that signal for these tissue characteristics may not be present within ultrasound videos, and that the use of CMR for tissue characterization remains essential within cardiology.

Clinical Perspective

Tissue characterization of the heart muscle is useful for clinical diagnosis and prognosis by identifying myocardial fibrosis, inflammation, and infiltration, and can be measured using cardiac MRI. While echocardiography is highly accessible and provides excellent functional information, its ability to provide tissue characterization information is limited at this time. Our study using a deep learning approach to predict cardiac MRI-based tissue characteristics from echocardiography showed limited ability to do so, suggesting that alternative approaches, including non-deep learning methods should be considered in future research.

Deep learning from latent spatiotemporal information of the heart: Identifying advanced bioimaging markers from echocardiograms

A key strength of echocardiography lies in its integration of comprehensive spatiotemporal cardiac imaging data in real-time, to aid frontline or bedside patient risk stratification and management. Nonetheless, its acquisition, processing, and interpretation are known to all be subject to heterogeneity from its reliance on manual and subjective human tracings, which challenges workflow and protocol standardization and final interpretation accuracy. In the era of advanced computational power, utilization of machine learning algorithms for big data analytics in echocardiography promises reduction in cost, cognitive errors, and intra- and inter-observer variability. Novel spatiotemporal deep learning (DL) models allow the integration of temporal arm information based on unlabeled pixel echocardiographic data for convolution of an adaptive semantic spatiotemporal calibration to construct personalized 4D heart meshes, assess global and regional cardiac function, detect early valve pathology, and differentiate uncommon cardiovascular disorders. Meanwhile, data visualization on spatiotemporal DL prediction models helps extract latent temporal imaging features to develop advanced imaging biomarkers in early disease stages and advance our understanding of pathophysiology to support the development of personalized prevention or treatment strategies. Since portable echocardiograms have been increasingly used as point-of-care imaging tools to aid rural care delivery, the application of these new spatiotemporal DL techniques show the potentials in streamlining echocardiographic acquisition, processing, and data analysis to improve workflow standardization and efficiencies, and provide risk stratification and decision supporting tools in real-time, to prompt the building of new imaging diagnostic networks to enhance rural healthcare engagement.

Deep learning from atrioventricular plane displacement in patients with Takotsubo syndrome: lighting up the black-box

Aims

The spatiotemporal deep convolutional neural network (DCNN) helps reduce echocardiographic readers' erroneous 'judgement calls' on Takotsubo syndrome (TTS). The aim of this study was to improve the interpretability of the spatiotemporal DCNN to discover latent imaging features associated with causative TTS pathophysiology.

Methods and results

We applied gradient-weighted class activation mapping analysis to visualize an established spatiotemporal DCNN based on the echocardiographic videos to differentiate TTS (150 patients) from anterior wall ST-segment elevation myocardial infarction (STEMI, 150 patients). Forty-eight human expert readers interpreted the same echocardiographic videos and prioritized the regions of interest on myocardium for the differentiation. Based on visualization results, we completed optical flow measurement, myocardial strain, and Doppler/tissue Doppler echocardiography studies to investigate regional myocardial temporal dynamics and diastology. While human readers' visualization predominantly focused on the apex of the heart in TTS patients, the DCNN temporal arm's saliency visualization was attentive on the base of the heart, particularly at the atrioventricular (AV) plane. Compared with STEMI patients, TTS patients consistently showed weaker peak longitudinal displacement (in pixels) in the basal inferoseptal (systolic: 2.15 ± 1.41 vs. 3.10 ± 1.66, P < 0.001; diastolic: 2.36 ± 1.71 vs. 2.97 ± 1.69, P = 0.004) and basal anterolateral (systolic: 2.70 ± 1.96 vs. 3.44 ± 2.13, P = 0.003; diastolic: 2.73 ± 1.70 vs. 3.45 ± 2.20, P = 0.002) segments, and worse longitudinal myocardial strain in the basal inferoseptal (-8.5 ± 3.8% vs. -9.9 ± 4.1%, P = 0.013) and basal anterolateral (-8.6 ± 4.2% vs. -10.4 ± 4.1%, P = 0.006) segments. Meanwhile, TTS patients showed worse diastolic mechanics than STEMI patients (E'/septal: 5.1 ± 1.2 cm/s vs. 6.3 ± 1.5 cm/s, P < 0.001; S'/septal: 5.8 ± 1.3 cm/s vs. 6.8 ± 1.4 cm/s, P < 0.001; E'/lateral: 6.0 ± 1.4 cm/s vs. 7.9 ± 1.6 cm/s, P < 0.001; S'/lateral: 6.3 ± 1.4 cm/s vs. 7.3 ± 1.5 cm/s, P < 0.001; E/E': 15.5 ± 5.6 vs. 12.5 ± 3.5, P < 0.001).

Conclusion

The spatiotemporal DCNN saliency visualization helps identify the pattern of myocardial temporal dynamics and navigates the quantification of regional myocardial mechanics. Reduced AV plane displacement in TTS patients likely correlates with impaired diastolic mechanics.

Machine Learning and Bias in Medical Imaging: Opportunities and Challenges

Bias in health care has been well documented and results in disparate and worsened outcomes for at-risk groups. Medical imaging plays a critical role in facilitating patient diagnoses but involves multiple sources of bias including factors related to access to imaging modalities, acquisition of images, and assessment (ie, interpretation) of imaging data. Machine learning (ML) applied to diagnostic imaging has demonstrated the potential to improve the quality of imaging-based diagnosis and the precision of measuring imaging-based traits. Algorithms can leverage subtle information not visible to the human eye to detect underdiagnosed conditions or derive new disease phenotypes by linking imaging features with clinical outcomes, all while mitigating cognitive bias in interpretation. Importantly, however, the application of ML to diagnostic imaging has the potential to either reduce or propagate bias. Understanding the potential gain as well as the potential risks requires an understanding of how and what ML models learn. Common risks of propagating bias can arise from unbalanced training, suboptimal architecture design or selection, and uneven application of models. Notwithstanding these risks, ML may yet be applied to improve gain from imaging across all 3A's (access, acquisition, and assessment) for all patients. In this review, we present a framework for understanding the balance of opportunities and challenges for minimizing bias in medical imaging, how ML may improve current approaches to imaging, and what specific design considerations should be made as part of efforts to maximize the quality of health care for all.

AI-ENABLED ASSESSMENT OF CARDIAC FUNCTION AND VIDEO QUALITY IN EMERGENCY DEPARTMENT POINT-OF-CARE ECHOCARDIOGRAMS

BACKGROUND

The adoption of point-of-care ultrasound (POCUS) has greatly improved the ability to rapidly evaluate unstable emergency department (ED) patients at the bedside. One major use of POCUS is to obtain echocardiograms to assess cardiac function.

OBJECTIVES

We developed EchoNet-POCUS, a novel deep learning system, to aid emergency physicians (EPs) in interpreting POCUS echocardiograms and to reduce operator-to-operator variability.

METHODS

We collected a new dataset of POCUS echocardiogram videos obtained in the ED by EPs and annotated the cardiac function and quality of each video. Using this dataset, we train EchoNet-POCUS to evaluate both cardiac function and video quality in POCUS echocardiograms.

RESULTS

EchoNet-POCUS achieves an area under the receiver operating characteristic curve (AUROC) of 0.92 (0.89-0.94) for predicting whether cardiac function is abnormal and an AUROC of 0.81 (0.78-0.85) for predicting video quality.

CONCLUSIONS

EchoNet-POCUS can be applied to bedside echocardiogram videos in real time using commodity hardware, as we demonstrate in a prospective pilot study.

Classification of chemically modified red blood cells in microflow using machine learning video analysis

We classify native and chemically modified red blood cells with an AI based video classifier. Using TensorFlow video analysis enables us to capture not only the morphology of the cell but also the trajectories of motion of individual red blood cells and their dynamics. We chemically modify cells in three different ways to model different pathological conditions and obtain classification accuracies for all three classification tasks of more than 90% between native and modified cells. Unlike standard cytometers that are based on immunophenotyping our microfluidic cytometer allows to rapidly categorize cells without any fluorescence labels simply by analysing the shape and flow of red blood cells.

Deep learning for transesophageal echocardiography view classification

Transesophageal echocardiography (TEE) imaging is a vital tool used in the evaluation of complex cardiac pathology and the management of cardiac surgery patients. A key limitation to the application of deep learning strategies to intraoperative and intraprocedural TEE data is the complexity and unstructured nature of these images. In the present study, we developed a deep learning-based, multi-category TEE view classification model that can be used to add structure to intraoperative and intraprocedural TEE imaging data. More specifically, we trained a convolutional neural network (CNN) to predict standardized TEE views using labeled intraoperative and intraprocedural TEE videos from Cedars-Sinai Medical Center (CSMC). We externally validated our model on intraoperative TEE videos from Stanford University Medical Center (SUMC). Accuracy of our model was high across all labeled views. The highest performance was achieved for the Trans-Gastric Left Ventricular Short Axis View (area under the receiver operating curve [AUC] = 0.971 at CSMC, 0.957 at SUMC), the Mid-Esophageal Long Axis View (AUC = 0.954 at CSMC, 0.905 at SUMC), the Mid-Esophageal Aortic Valve Short Axis View (AUC = 0.946 at CSMC, 0.898 at SUMC), and the Mid-Esophageal 4-Chamber View (AUC = 0.939 at CSMC, 0.902 at SUMC). Ultimately, we demonstrate that our deep learning model can accurately classify standardized TEE views, which will facilitate further downstream deep learning analyses for intraoperative and intraprocedural TEE imaging.

Successes and challenges of artificial intelligence in cardiology

In the past decades there has been a substantial evolution in data management and data processing techniques. New data architectures made analysis of big data feasible, healthcare is orienting towards personalized medicine with digital health initiatives, and artificial intelligence (AI) is becoming of increasing importance. Despite being a trendy research topic, only very few applications reach the stage where they are implemented in clinical practice. This review provides an overview of current methodologies and identifies clinical and organizational challenges for AI in healthcare.

Artificial Intelligence in Ventricular Arrhythmias and Sudden Death

Sudden cardiac arrest due to lethal ventricular arrhythmias is a major cause of mortality worldwide and results in more years of potential life lost than any individual cancer. Most of these sudden cardiac arrest events occur unexpectedly in individuals who have not been identified as high-risk due to the inadequacy of current risk stratification tools. Artificial intelligence tools are increasingly being used to solve complex problems and are poised to help with this major unmet need in the field of clinical electrophysiology. By leveraging large and detailed datasets, artificial intelligence-based prediction models have the potential to enhance the risk stratification of lethal ventricular arrhythmias. This review presents a synthesis of the published literature and a discussion of future directions in this field.

Cyclical Self-Supervision for Semi-Supervised Ejection Fraction Prediction From Echocardiogram Videos

Left-ventricular ejection fraction (LVEF) is an important indicator of heart failure. Existing methods for LVEF estimation from video require large amounts of annotated data to achieve high performance, e.g. using 10,030 labeled echocardiogram videos to achieve mean absolute error (MAE) of 4.10. Labeling these videos is time-consuming however and limits potential downstream applications to other heart diseases. This paper presents the first semi-supervised approach for LVEF prediction. Unlike general video prediction tasks, LVEF prediction is specifically related to changes in the left ventricle (LV) in echocardiogram videos. By incorporating knowledge learned from predicting LV segmentations into LVEF regression, we can provide additional context to the model for better predictions. To this end, we propose a novel Cyclical Self-Supervision (CSS) method for learning video-based LV segmentation, which is motivated by the observation that the heartbeat is a cyclical process with temporal repetition. Prediction masks from our segmentation model can then be used as additional input for LVEF regression to provide spatial context for the LV region. We also introduce teacher-student distillation to distill the information from LV segmentation masks into an end-to-end LVEF regression model that only requires video inputs. Results show our method outperforms alternative semi-supervised methods and can achieve MAE of 4.17, which is competitive with state-of-the-art supervised performance, using half the number of labels. Validation on an external dataset also shows improved generalization ability from using our method.

A deep learning model for novel systemic biomarkers in photographs of the external eye: a retrospective study

BACKGROUND

Photographs of the external eye were recently shown to reveal signs of diabetic retinal disease and elevated glycated haemoglobin. This study aimed to test the hypothesis that external eye photographs contain information about additional systemic medical conditions.

METHODS

We developed a deep learning system (DLS) that takes external eye photographs as input and predicts systemic parameters, such as those related to the liver (albumin, aspartate aminotransferase [AST]); kidney (estimated glomerular filtration rate [eGFR], urine albumin-to-creatinine ratio [ACR]); bone or mineral (calcium); thyroid (thyroid stimulating hormone); and blood (haemoglobin, white blood cells [WBC], platelets). This DLS was trained using 123 130 images from 38 398 patients with diabetes undergoing diabetic eye screening in 11 sites across Los Angeles county, CA, USA. Evaluation focused on nine prespecified systemic parameters and leveraged three validation sets (A, B, C) spanning 25 510 patients with and without diabetes undergoing eye screening in three independent sites in Los Angeles county, CA, and the greater Atlanta area, GA, USA. We compared performance against baseline models incorporating available clinicodemographic variables (eg, age, sex, race and ethnicity, years with diabetes).

FINDINGS

Relative to the baseline, the DLS achieved statistically significant superior performance at detecting AST >36·0 U/L, calcium <8·6 mg/dL, eGFR <60·0 mL/min/1·73 m2, haemoglobin <11·0 g/dL, platelets <150·0 × 103/μL, ACR ≥300 mg/g, and WBC <4·0 × 103/μL on validation set A (a population resembling the development datasets), with the area under the receiver operating characteristic curve (AUC) of the DLS exceeding that of the baseline by 5·3-19·9% (absolute differences in AUC). On validation sets B and C, with substantial patient population differences compared with the development datasets, the DLS outperformed the baseline for ACR ≥300·0 mg/g and haemoglobin <11·0 g/dL by 7·3-13·2%.

INTERPRETATION

We found further evidence that external eye photographs contain biomarkers spanning multiple organ systems. Such biomarkers could enable accessible and non-invasive screening of disease. Further work is needed to understand the translational implications.

FUNDING

Google.

Confounders mediate AI prediction of demographics in medical imaging

Deep learning has been shown to accurately assess "hidden" phenotypes from medical imaging beyond traditional clinician interpretation. Using large echocardiography datasets from two healthcare systems, we test whether it is possible to predict age, race, and sex from cardiac ultrasound images using deep learning algorithms and assess the impact of varying confounding variables. Using a total of 433,469 videos from Cedars-Sinai Medical Center and 99,909 videos from Stanford Medical Center, we trained video-based convolutional neural networks to predict age, sex, and race. We found that deep learning models were able to identify age and sex, while unable to reliably predict race. Without considering confounding differences between categories, the AI model predicted sex with an AUC of 0.85 (95% CI 0.84-0.86), age with a mean absolute error of 9.12 years (95% CI 9.00-9.25), and race with AUCs ranging from 0.63 to 0.71. When predicting race, we show that tuning the proportion of confounding variables (age or sex) in the training data significantly impacts model AUC (ranging from 0.53 to 0.85), while sex and age prediction was not particularly impacted by adjusting race proportion in the training dataset AUC of 0.81-0.83 and 0.80-0.84, respectively. This suggests significant proportion of AI's performance on predicting race could come from confounding features being detected. Further work remains to identify the particular imaging features that associate with demographic information and to better understand the risks of demographic identification in medical AI as it pertains to potentially perpetuating bias and disparities.

Advancing cardiovascular medicine with machine learning: Progress, potential, and perspective

Recent advances in machine learning (ML) have made it possible to analyze high-dimensional and complex data-such as free text, images, waveforms, videos, and sound-in an automated manner by successfully learning complex associations within these data. Cardiovascular medicine is particularly well poised to take advantage of these ML advances, due to the widespread digitization of medical data and the large number of diagnostic tests used to evaluate cardiovascular disease. Various ML approaches have successfully been applied to cardiovascular tests and diseases to automate interpretation, accurately perform measurements, and, in some cases, predict novel diagnoses from less invasive tests, effectively expanding the utility of more widely accessible diagnostic tests. Here, we present examples of some impactful advances in cardiovascular medicine using ML across a variety of modalities, with a focus on deep learning applications.

Artificial intelligence in histopathology: enhancing cancer research and clinical oncology

Artificial intelligence (AI) methods have multiplied our capabilities to extract quantitative information from digital histopathology images. AI is expected to reduce workload for human experts, improve the objectivity and consistency of pathology reports, and have a clinical impact by extracting hidden information from routinely available data. Here, we describe how AI can be used to predict cancer outcome, treatment response, genetic alterations and gene expression from digitized histopathology slides. We summarize the underlying technologies and emerging approaches, noting limitations, including the need for data sharing and standards. Finally, we discuss the broader implications of AI in cancer research and oncology.

Artificial intelligence applications in cardio-oncology: Leveraging high dimensional cardiovascular data

Growing evidence suggests a wide spectrum of potential cardiovascular complications following cancer therapies, leading to an urgent need for better risk-stratifying and disease screening in patients undergoing oncological treatment. As many cancer patients undergo frequent surveillance through imaging as well as other diagnostic testing, there is a wealth of information that can be utilized to assess one's risk for cardiovascular complications of cancer therapies. Over the past decade, there have been remarkable advances in applying artificial intelligence (AI) to analyze cardiovascular data obtained from electrocardiograms, echocardiograms, computed tomography, and cardiac magnetic resonance imaging to detect early signs or future risk of cardiovascular diseases. Studies have shown AI-guided cardiovascular image analysis can accurately, reliably and inexpensively identify and quantify cardiovascular risk, leading to better detection of at-risk or disease features, which may open preventive and therapeutic opportunities in cardio-oncology. In this perspective, we discuss the potential for the use of AI in analyzing cardiovascular data to identify cancer patients at risk for cardiovascular complications early in treatment which would allow for rapid intervention to prevent adverse cardiovascular outcomes.

Recent Advances in Machine Learning Applied to Ultrasound Imaging

Machine learning (ML) methods are pervading an increasing number of fields of application because of their capacity to effectively solve a wide variety of challenging problems. The employment of ML techniques in ultrasound imaging applications started several years ago but the scientific interest in this issue has increased exponentially in the last few years. The present work reviews the most recent (2019 onwards) implementations of machine learning techniques for two of the most popular ultrasound imaging fields, medical diagnostics and non-destructive evaluation. The former, which covers the major part of the review, was analyzed by classifying studies according to the human organ investigated and the methodology (e.g., detection, segmentation, and/or classification) adopted, while for the latter, some solutions to the detection/classification of material defects or particular patterns are reported. Finally, the main merits of machine learning that emerged from the study analysis are summarized and discussed.