Deep neural networks for A-line-based plaque classification in coronary intravascular optical coherence tomography images

A transformer-based pyramid network for coronary calcified plaque segmentation in intravascular optical coherence tomography images

Characterizing coronary calcified plaque (CCP) provides essential insight into diagnosis and treatment of atherosclerosis. Intravascular optical coherence tomography (OCT) offers significant advantages for detecting CCP and even automated segmentation with recent advances in deep learning techniques. Most of current methods have achieved promising results by adopting existing convolution neural networks (CNNs) in computer vision domain. However, their performance can be detrimentally affected by unseen plaque patterns and artifacts due to inherent limitation of CNNs in contextual reasoning. To overcome this obstacle, we proposed a Transformer-based pyramid network called AFS-TPNet for robust, end-to-end segmentation of CCP from OCT images. Its encoder is built upon CSWin Transformer architecture, allowing for better perceptual understanding of calcified arteries at a higher semantic level. Specifically, an augmented feature split (AFS) module and residual convolutional position encoding (RCPE) mechanism are designed to effectively enhance the capability of Transformer in capturing both fine-grained features and global contexts. Extensive experiments showed that AFS-TPNet trained using Lovasz Loss achieved superior performance in segmentation CCP under various contexts, surpassing prior state-of-the-art CNN and Transformer architectures by more than 6.58% intersection over union (IoU) score. The application of this promising method to extract CCP features is expected to enhance clinical intervention and translational research using OCT.

Deep learning segmentation of fibrous cap in intravascular optical coherence tomography images

Thin-cap fibroatheroma (TCFA) is a prominent risk factor for plaque rupture. Intravascular optical coherence tomography (IVOCT) enables identification of fibrous cap (FC), measurement of FC thicknesses, and assessment of plaque vulnerability. We developed a fully-automated deep learning method for FC segmentation. This study included 32,531 images across 227 pullbacks from two registries (TRANSFORM-OCT and UHCMC). Images were semi-automatically labeled using our OCTOPUS with expert editing using established guidelines. We employed preprocessing including guidewire shadow detection, lumen segmentation, pixel-shifting, and Gaussian filtering on raw IVOCT (r,θ) images. Data were augmented in a natural way by changing θ in spiral acquisitions and by changing intensity and noise values. We used a modified SegResNet and comparison networks to segment FCs. We employed transfer learning from our existing much larger, fully-labeled calcification IVOCT dataset to reduce deep-learning training. Postprocessing with a morphological operation enhanced segmentation performance. Overall, our method consistently delivered better FC segmentation results (Dice: 0.837 ± 0.012) than other deep-learning methods. Transfer learning reduced training time by 84% and reduced the need for more training samples. Our method showed a high level of generalizability, evidenced by highly-consistent segmentations across five-fold cross-validation (sensitivity: 85.0 ± 0.3%, Dice: 0.846 ± 0.011) and the held-out test (sensitivity: 84.9%, Dice: 0.816) sets. In addition, we found excellent agreement of FC thickness with ground truth (2.95 ± 20.73 µm), giving clinically insignificant bias. There was excellent reproducibility in pre- and post-stenting pullbacks (average FC angle: 200.9 ± 128.0°/202.0 ± 121.1°). Our fully automated, deep-learning FC segmentation method demonstrated excellent performance, generalizability, and reproducibility on multi-center datasets. It will be useful for multiple research purposes and potentially for planning stent deployments that avoid placing a stent edge over an FC.

Validation of biomechanical assessment of coronary plaque vulnerability based on intravascular optical coherence tomography and digital subtraction angiography

Background

It has been suggested that biomechanical factors may influence plaque development. However, key determinants for assessing plaque vulnerability remain speculative.

Methods

In this study, a two-dimensional (2D) structural mechanical analysis and a three-dimensional (3D) fluid-structure interaction (FSI) analysis were conducted based on intravascular optical coherence tomography (IV-OCT) and digital subtraction angiography (DSA) data sets. In the 2D study, 103 IV-OCT slices were analyzed. An in-depth morpho-mechanic analysis and a weighted least absolute shrinkage and selection operator (LASSO) regression analysis were conducted to identify the crucial features related to plaque vulnerability via the tuning parameter (λ). In the 3D study, the coronary model was reconstructed by fusing the IV-OCT and DSA data, and a FSI analysis was subsequently performed. The relationship between vulnerable plaque and wall shear stress (WSS) was investigated.

Results

The influential factors were selected using the minimum criteria (λ-min) and one-standard error criteria (λ-1se). In addition to the common vulnerable factor of the minimum fibrous cap thickness (FCTmin), four biomechanical factors were selected by λ-min, including the average/maximal displacements and average/maximal stress, and two biomechanical factors were selected by λ-1se, including the average/maximal displacements. Additionally, the positions of the vulnerable plaques were consistent with the sites of high WSS.

Conclusions

Functional indices are crucial for plaque status assessment. An evaluation based on biomechanical simulations might provide insights into risk identification and guide therapeutic decisions.

Machine Learning-Assisted Short-Wave InfraRed (SWIR) Techniques for Biomedical Applications: Towards Personalized Medicine

Personalized medicine transforms healthcare by adapting interventions to individuals' unique genetic, molecular, and clinical profiles. To maximize diagnostic and/or therapeutic efficacy, personalized medicine requires advanced imaging devices and sensors for accurate assessment and monitoring of individual patient conditions or responses to therapeutics. In the field of biomedical optics, short-wave infrared (SWIR) techniques offer an array of capabilities that hold promise to significantly enhance diagnostics, imaging, and therapeutic interventions. SWIR techniques provide in vivo information, which was previously inaccessible, by making use of its capacity to penetrate biological tissues with reduced attenuation and enable researchers and clinicians to delve deeper into anatomical structures, physiological processes, and molecular interactions. Combining SWIR techniques with machine learning (ML), which is a powerful tool for analyzing information, holds the potential to provide unprecedented accuracy for disease detection, precision in treatment guidance, and correlations of complex biological features, opening the way for the data-driven personalized medicine field. Despite numerous biomedical demonstrations that utilize cutting-edge SWIR techniques, the clinical potential of this approach has remained significantly underexplored. This paper demonstrates how the synergy between SWIR imaging and ML is reshaping biomedical research and clinical applications. As the paper showcases the growing significance of SWIR imaging techniques that are empowered by ML, it calls for continued collaboration between researchers, engineers, and clinicians to boost the translation of this technology into clinics, ultimately bridging the gap between cutting-edge technology and its potential for personalized medicine.

Prediction of stent under-expansion in calcified coronary arteries using machine learning on intravascular optical coherence tomography images

It can be difficult/impossible to fully expand a coronary artery stent in a heavily calcified coronary artery lesion. Under-expanded stents are linked to later complications. Here we used machine/deep learning to analyze calcifications in pre-stent intravascular optical coherence tomography (IVOCT) images and predicted the success of vessel expansion. Pre- and post-stent IVOCT image data were obtained from 110 coronary lesions. Lumen and calcifications in pre-stent images were segmented using deep learning, and lesion features were extracted. We analyzed stent expansion along the lesion, enabling frame, segmental, and whole-lesion analyses. We trained regression models to predict the post-stent lumen area and then computed the stent expansion index (SEI). Best performance (root-mean-square-error = 0.04 ± 0.02 mm2, r = 0.94 ± 0.04, p < 0.0001) was achieved when we used features from both lumen and calcification to train a Gaussian regression model for segmental analysis of 31 frames in length. Stents with minimum SEI > 80% were classified as "well-expanded;" others were "under-expanded." Under-expansion classification results (e.g., AUC = 0.85 ± 0.02) were significantly improved over a previous, simple calculation, as well as other machine learning solutions. Promising results suggest that such methods can identify lesions at risk of under-expansion that would be candidates for intervention lesion preparation (e.g., atherectomy).

Pericoronary Adipose Tissue Radiomics from Coronary Computed Tomography Angiography Identifies Vulnerable Plaques

Pericoronary adipose tissue (PCAT) features on Computed Tomography (CT) have been shown to reflect local inflammation and increased cardiovascular risk. Our goal was to determine whether PCAT radiomics extracted from coronary CT angiography (CCTA) images are associated with intravascular optical coherence tomography (IVOCT)-identified vulnerable-plaque characteristics (e.g., microchannels (MC) and thin-cap fibroatheroma (TCFA)). The CCTA and IVOCT images of 30 lesions from 25 patients were registered. The vessels with vulnerable plaques were identified from the registered IVOCT images. The PCAT-radiomics features were extracted from the CCTA images for the lesion region of interest (PCAT-LOI) and the entire vessel (PCAT-Vessel). We extracted 1356 radiomic features, including intensity (first-order), shape, and texture features. The features were reduced using standard approaches (e.g., high feature correlation). Using stratified three-fold cross-validation with 1000 repeats, we determined the ability of PCAT-radiomics features from CCTA to predict IVOCT vulnerable-plaque characteristics. In the identification of TCFA lesions, the PCAT-LOI and PCAT-Vessel radiomics models performed comparably (Area Under the Curve (AUC) ± standard deviation 0.78 ± 0.13, 0.77 ± 0.14). For the identification of MC lesions, the PCAT-Vessel radiomics model (0.89 ± 0.09) was moderately better associated than the PCAT-LOI model (0.83 ± 0.12). In addition, both the PCAT-LOI and the PCAT-Vessel radiomics model identified coronary vessels thought to be highly vulnerable to a similar standard (i.e., both TCFA and MC; 0.88 ± 0.10, 0.91 ± 0.09). The most favorable radiomic features tended to be those describing the texture and size of the PCAT. The application of PCAT radiomics can identify coronary vessels with TCFA or MC, consistent with IVOCT. Furthermore, the use of CCTA radiomics may improve risk stratification by noninvasively detecting vulnerable-plaque characteristics that are only visible with IVOCT.

OCTOPUS - Optical coherence tomography plaque and stent analysis software

Background and objective

Compared with other imaging modalities, intravascular optical coherence tomography (IVOCT) has significant advantages for guiding percutaneous coronary interventions, assessing their outcomes, and characterizing plaque components. To aid IVOCT research studies, we developed the Optical Coherence TOmography PlaqUe and Stent (OCTOPUS) analysis software, which provides highly automated, comprehensive analysis of coronary plaques and stents in IVOCT images.

Methods

User specifications for OCTOPUS were obtained from detailed, iterative discussions with IVOCT analysts in the Cardiovascular Imaging Core Laboratory at University Hospitals Cleveland Medical Center, a leading laboratory for IVOCT image analysis. To automate image analysis results, the software includes several important algorithmic steps: pre-processing, deep learning plaque segmentation, machine learning identification of stent struts, and registration of pullbacks for sequential comparisons. Intuitive, interactive visualization and manual editing of segmentations were included in the software. Quantifications include stent deployment characteristics (e.g., stent area and stent strut malapposition), strut level analysis, calcium angle, and calcium thickness measurements. Interactive visualizations include (x,y) anatomical, en face, and longitudinal views with optional overlays (e.g., segmented calcifications). To compare images over time, linked visualizations were enabled to display up to four registered vessel segments at a time.

Results

OCTOPUS has been deployed for nearly 1 year and is currently being used in multiple IVOCT studies. Underlying plaque segmentation algorithm yielded excellent pixel-wise results (86.2% sensitivity and 0.781 F1 score). Using OCTOPUS on 34 new pullbacks, we determined that following automated segmentation, only 13% and 23% of frames needed any manual touch up for detailed lumen and calcification labeling, respectively. Only up to 3.8% of plaque pixels were modified, leading to an average editing time of only 7.5 s/frame, an approximately 80% reduction compared to manual analysis. Regarding stent analysis, sensitivity and precision were both greater than 90%, and each strut was successfully classified as either covered or uncovered with high sensitivity (94%) and specificity (90%). We demonstrated use cases for sequential analysis. To analyze plaque progression, we loaded multiple pullbacks acquired at different points (e.g., pre-stent, 3-month follow-up, and 18-month follow-up) and evaluated frame-level development of in-stent neo-atherosclerosis. In ex vivo cadaver experiments, the OCTOPUS software enabled visualization and quantitative evaluation of irregular stent deployment in the presence of calcifications identified in pre-stent images.

Conclusions

We introduced and evaluated the clinical application of a highly automated software package, OCTOPUS, for quantitative plaque and stent analysis in IVOCT images. The software is currently used as an offline tool for research purposes; however, the software's embedded algorithms may also be useful for real-time treatment planning.

Pericoronary adipose tissue radiomics from coronary CT angiography identifies vulnerable plaques characteristics in intravascular OCT

Pericoronary adipose tissue (PCAT) features on CT have been shown to reflect local inflammation, and signals increased cardiovascular risk. Our goal was to determine if PCAT radiomics extracted from coronary CT angiography (CCTA) images are associated with intravascular optical coherence tomography (IVOCT)-identified vulnerable plaque characteristics (e.g., microchannels [MC] and thin-cap fibroatheroma [TCFA]). CCTA and IVOCT images of 30 lesions from 25 patients were registered. Vessels with vulnerable plaques were identified from the registered IVOCT images. PCAT radiomics features were extracted from CCTA images for the lesion region of interest (PCAT-LOI) and the entire vessel (PCAT-Vessel). We extracted 1356 radiomics features, including intensity (first-order), shape, and texture features. Features were reduced using standard approaches (e.g., high feature correlation). Using stratified three-fold cross-validation with 1000 repeats, we determined the ability of PCAT radiomics features from CCTA to predict IVOCT vulnerable plaque characteristics. In identification of TCFA lesions, PCAT-LOI and PCAT-Vessel radiomics models performed comparably (AUC±standard deviation 0.78±0.13, 0.77±0.14). For identification of MC lesions, PCAT-Vessel radiomics model (0.89±0.09) was moderately better associated than that of PCAT-LOI model (0.83±0.12). Both PCAT-LOI and PCAT-Vessel radiomics models also similarly identified coronary vessels thought to be highly vulnerable (i.e., both TCFA and MC) (0.88±0.10, 0.91±0.09). Favorable radiomics features tended to be those describing texture and size of PCAT. PCAT radiomics can identify coronary vessels with TCFA or MC, consistent with IVOCT. CCTA radiomics may improve risk stratification by noninvasively detecting vulnerable plaque characteristics that are only visible with IVOCT.

Neoatherosclerosis prediction using plaque markers in intravascular optical coherence tomography images

Introduction

In-stent neoatherosclerosis has emerged as a crucial factor in post-stent complications including late in-stent restenosis and very late stent thrombosis. In this study, we investigated the ability of quantitative plaque characteristics from intravascular optical coherence tomography (IVOCT) images taken just prior to stent implantation to predict neoatherosclerosis after implantation.

Methods

This was a sub-study of the TRiple Assessment of Neointima Stent FOrmation to Reabsorbable polyMer with Optical Coherence Tomography (TRANSFORM-OCT) trial. Images were obtained before and 18 months after stent implantation. Final analysis included images of 180 lesions from 90 patients; each patient had images of two lesions in different coronary arteries. A total of 17 IVOCT plaque features, including lesion length, lumen (e.g., area and diameter); calcium (e.g., angle and thickness); and fibrous cap (FC) features (e.g., thickness, surface area, and burden), were automatically extracted from the baseline IVOCT images before stenting using dedicated software developed by our group (OCTOPUS). The predictive value of baseline IVOCT plaque features for neoatherosclerosis development after stent implantation was assessed using univariate/multivariate logistic regression and receiver operating characteristic (ROC) analyses.

Results

Follow-up IVOCT identified stents with (n = 19) and without (n = 161) neoatherosclerosis. Greater lesion length and maximum calcium angle and features related to FC were associated with a higher prevalence of neoatherosclerosis after stent implantation (p < 0.05). Hierarchical clustering identified six clusters with the best prediction p-values. In univariate logistic regression analysis, maximum calcium angle, minimum calcium thickness, maximum FC angle, maximum FC area, FC surface area, and FC burden were significant predictors of neoatherosclerosis. Lesion length and features related to the lumen were not significantly different between the two groups. In multivariate logistic regression analysis, only larger FC surface area was strongly associated with neoatherosclerosis (odds ratio 1.38, 95% confidence interval [CI] 1.05-1.80, p < 0.05). The area under the ROC curve was 0.901 (95% CI 0.859-0.946, p < 0.05) for FC surface area.

Conclusion

Post-stent neoatherosclerosis can be predicted by quantitative IVOCT imaging of plaque characteristics prior to stent implantation. Our findings highlight the additional clinical benefits of utilizing IVOCT imaging in the catheterization laboratory to inform treatment decision-making and improve outcomes.

Automated analysis of fibrous cap in intravascular optical coherence tomography images of coronary arteries

Thin-cap fibroatheroma (TCFA) and plaque rupture have been recognized as the most frequent risk factor for thrombosis and acute coronary syndrome. Intravascular optical coherence tomography (IVOCT) can identify TCFA and assess cap thickness, which provides an opportunity to assess plaque vulnerability. We developed an automated method that can detect lipidous plaque and assess fibrous cap thickness in IVOCT images. This study analyzed a total of 4360 IVOCT image frames of 77 lesions among 41 patients. Expert cardiologists manually labeled lipidous plaque based on established criteria. To improve segmentation performance, preprocessing included lumen segmentation, pixel-shifting, and noise filtering on the raw polar (r, θ) IVOCT images. We used the DeepLab-v3 plus deep learning model to classify lipidous plaque pixels. After lipid detection, we automatically detected the outer border of the fibrous cap using a special dynamic programming algorithm and assessed the cap thickness. Our method provided excellent discriminability of lipid plaque with a sensitivity of 85.8% and A-line Dice coefficient of 0.837. By comparing lipid angle measurements between two analysts following editing of our automated software, we found good agreement by Bland-Altman analysis (difference 6.7° ± 17°; mean ~ 196°). Our method accurately detected the fibrous cap from the detected lipid plaque. Automated analysis required a significant modification for only 5.5% frames. Furthermore, our method showed a good agreement of fibrous cap thickness between two analysts with Bland-Altman analysis (4.2 ± 14.6 µm; mean ~ 175 µm), indicating little bias between users and good reproducibility of the measurement. We developed a fully automated method for fibrous cap quantification in IVOCT images, resulting in good agreement with determinations by analysts. The method has great potential to enable highly automated, repeatable, and comprehensive evaluations of TCFAs.

A-line-based thin-cap fibroatheroma detection with multi-view IVOCT images using multi-task learning and contrastive learning

Automatic detection of thin-cap fibroatheroma (TCFA) is essential to prevent acute coronary syndrome. Hence, in this paper, a method is proposed to detect TCFAs by directly classifying each A-line using multi-view intravascular optical coherence tomography (IVOCT) images. To solve the problem of false positives, a multi-input-output network was developed to implement image-level classification and A-line-based classification at the same time, and a contrastive consistency term was designed to ensure consistency between two tasks. In addition, to learn spatial and global information and obtain the complete extent of TCFAs, an architecture and a regional connectivity constraint term are proposed to classify each A-line of IVOCT images. Experimental results obtained on the 2017 China Computer Vision Conference IVOCT dataset show that the proposed method achieved state-of-art performance with a total score of 88.7±0.88%, overlap rate of 88.64±0.26%, precision rate of 84.34±0.86%, and recall rate of 93.67±2.29%.

Deep learning applications in coronary anatomy imaging: a systematic review and meta-analysis

Background

The application of deep learning on medical imaging is growing in prevalence in the recent literature. One of the most studied areas is coronary artery disease (CAD). Imaging of coronary artery anatomy is fundamental, which has led to a high number of publications describing a variety of techniques. The aim of this systematic review is to review the evidence behind the accuracy of deep learning applications in coronary anatomy imaging.

Methods

The search for the relevant studies, which applied deep learning on coronary anatomy imaging, was performed in a systematic approach on MEDLINE and EMBASE databases, followed by reviewing of abstracts and full texts. The data from the final studies was retrieved using data extraction forms. A meta-analysis was performed on a subgroup of studies, which looked at fractional flow reserve (FFR) prediction. Heterogeneity was tested using tau², I² and Q tests. Finally, a risk of bias was performed using Quality Assessment of Diagnostic Accuracy Studies (QUADAS) approach.

Results

A total of 81 studies met the inclusion criteria. The most common imaging modality was coronary computed tomography angiography (CCTA) (58%) and the most common deep learning method was convolutional neural network (CNN) (52%). The majority of studies demonstrated good performance metrics. The most common outputs were focused on coronary artery segmentation, clinical outcome prediction, coronary calcium quantification and FFR prediction, and most studies reported area under the curve (AUC) of ≥80%. The pooled diagnostic odds ratio (DOR) derived from 8 studies looking at FFR prediction using CCTA was 12.5 using the Mantel-Haenszel (MH) method. There was no significant heterogeneity amongst studies according to Q test (P=0.2496).

Conclusions

Deep learning has been used in many applications on coronary anatomy imaging, most of which are yet to be externally validated and prepared for clinical use. The performance of deep learning, especially CNN models, proved to be powerful and some applications have already translated into medical practice, such as computed tomography (CT)-FFR. These applications have the potential to translate technology into better care of CAD patients.

Rapid lipid-laden plaque identification in intravascular optical coherence tomography imaging based on time-series deep learning

SIGNIFICANCE

Coronary heart disease has the highest rate of death and morbidity in the Western world. Atherosclerosis is an asymptomatic condition that is considered the primary cause of cardiovascular diseases. The accumulation of low-density lipoprotein triggers an inflammatory process in focal areas of arteries, which leads to the formation of plaques. Lipid-laden plaques containing a necrotic core may eventually rupture, causing heart attack and stroke. Lately, intravascular optical coherence tomography (IV-OCT) imaging has been used for plaque assessment. The interpretation of the IV-OCT images is performed visually, which is burdensome and requires highly trained physicians for accurate plaque identification.

AIM

Our study aims to provide high throughput lipid-laden plaque identification that can assist in vivo imaging by offering faster screening and guided decision making during percutaneous coronary interventions.

APPROACH

An A-line-wise classification methodology based on time-series deep learning is presented to fulfill this aim. The classifier was trained and validated with a database consisting of IV-OCT images of 98 artery sections. A trained physician with expertise in the analysis of IV-OCT imaging provided the visual evaluation of the database that was used as ground truth for training and validation.

RESULTS

This method showed an accuracy, sensitivity, and specificity of 89.6%, 83.6%, and 91.1%, respectively. This deep learning methodology has the potential to increase the speed of lipid-laden plaques identification to provide a high throughput of more than 100 B-scans/s.

CONCLUSIONS

These encouraging results suggest that this method will allow for high throughput video-rate atherosclerotic plaque assessment through automated tissue characterization for in vivo imaging by providing faster screening to assist in guided decision making during percutaneous coronary interventions.

Automated Coronary Optical Coherence Tomography Feature Extraction with Application to Three-Dimensional Reconstruction

Coronary optical coherence tomography (OCT) is an intravascular, near-infrared light-based imaging modality capable of reaching axial resolutions of 10–20 µm. This resolution allows for accurate determination of high-risk plaque features, such as thin cap fibroatheroma; however, visualization of morphological features alone still provides unreliable positive predictive capability for plaque progression or future major adverse cardiovascular events (MACE). Biomechanical simulation could assist in this prediction, but this requires extracting morphological features from intravascular imaging to construct accurate three-dimensional (3D) simulations of patients’ arteries. Extracting these features is a laborious process, often carried out manually by trained experts. To address this challenge, numerous techniques have emerged to automate these processes while simultaneously overcoming difficulties associated with OCT imaging, such as its limited penetration depth. This systematic review summarizes advances in automated segmentation techniques from the past five years (2016–2021) with a focus on their application to the 3D reconstruction of vessels and their subsequent simulation. We discuss four categories based on the feature being processed, namely: coronary lumen; artery layers; plaque characteristics and subtypes; and stents. Areas for future innovation are also discussed as well as their potential for future translation.

Current research and future prospects of IVOCT imaging-based detection of the vascular lumen and vulnerable plaque

Intravascular optical coherence tomography (IVOCT) is an imaging method that has developed rapidly in recent years and is useful in coronary atherosclerosis diagnosis. It is widely used in the assessment of vulnerable plaque. This review summarizes the main research methods used in recent years for blood vessel lumen boundary detection and segmentation and vulnerable plaque segmentation and classification. This article aims to comprehensively and systematically introduce the research progress on internal tissues of blood vessels based on IVOCT images. The characteristics and advantages of various methods have been summarized to provide theoretical ideas and methods for the reference of relevant researchers and scholars.

Comprehensive Assessment of Coronary Calcification in Intravascular OCT Using a Spatial-Temporal Encoder-Decoder Network

Coronary calcification is a strong indicator of coronary artery disease and a key determinant of the outcome of percutaneous coronary intervention. We propose a fully automated method to segment and quantify coronary calcification in intravascular OCT (IVOCT) images based on convolutional neural networks (CNN). All possible calcified plaques were segmented from IVOCT pullbacks using a spatial-temporal encoder-decoder network by exploiting the 3D continuity information of the plaques, which were then screened and classified by a DenseNet network to reduce false positives. A novel data augmentation method based on the IVOCT image acquisition pattern was also proposed to improve the performance and robustness of the segmentation. Clinically relevant metrics including calcification area, depth, angle, thickness, volume, and stent-deployment calcification score, were automatically computed. 13844 IVOCT images with 2627 calcification slices from 45 clinical OCT pullbacks were collected and used to train and test the model. The proposed method performed significantly better than existing state-of-the-art 2D and 3D CNN methods. The data augmentation method improved the Dice similarity coefficient for calcification segmentation from 0.615±0.332 to 0.756±0.222, reaching human-level inter-observer agreement. Our proposed region-based classifier improved image-level calcification classification precision and F1-score from 0.725±0.071 and 0.791±0.041 to 0.964±0.002 and 0.883±0.008, respectively. Bland-Altman analysis showed close agreement between manual and automatic calcification measurements. Our proposed method is valuable for automated assessment of coronary calcification lesions and in-procedure planning of stent deployment.

Artificial Intelligence in Cardiovascular Atherosclerosis Imaging

At present, artificial intelligence (AI) has already been applied in cardiovascular imaging (e.g., image segmentation, automated measurements, and eventually, automated diagnosis) and it has been propelled to the forefront of cardiovascular medical imaging research. In this review, we presented the current status of artificial intelligence applied to image analysis of coronary atherosclerotic plaques, covering multiple areas from plaque component analysis (e.g., identification of plaque properties, identification of vulnerable plaque, detection of myocardial function, and risk prediction) to risk prediction. Additionally, we discuss the current evidence, strengths, limitations, and future directions for AI in cardiac imaging of atherosclerotic plaques, as well as lessons that can be learned from other areas. The continuous development of computer science and technology may further promote the development of this field.

Automatic microchannel detection using deep learning in intravascular optical coherence tomography images

Microchannel formation is known to be a significant marker of plaque vulnerability, plaque rupture, and intraplaque hemorrhage, which are responsible for plaque progression. We developed a fully-automated method for detecting microchannels in intravascular optical coherence tomography (IVOCT) images using deep learning. A total of 3,075 IVOCT image frames across 41 patients having 62 microchannel segments were analyzed. Microchannel was manually annotated by expert cardiologists, according to previously established criteria. In order to improve segmentation performance, pre-processing including guidewire detection/removal, lumen segmentation, pixel-shifting, and noise filtering was applied to the raw (r,θ) IVOCT image. We used the DeepLab-v3 plus deep learning model with the Xception backbone network for identifying microchannel candidates. After microchannel candidate detection, each candidate was classified as either microchannel or no-microchannel using a convolutional neural network (CNN) classification model. Our method provided excellent segmentation of microchannel with a Dice coefficient of 0.811, sensitivity of 92.4%, and specificity of 99.9%. We found that pre-processing and data augmentation were very important to improve results. In addition, a CNN classification step was also helpful to rule out false positives. Furthermore, automated analysis missed only 3% of frames having microchannels and showed no false positives. Our method has great potential to enable highly automated, objective, repeatable, and comprehensive evaluations of vulnerable plaques and treatments. We believe that this method is promising for both research and clinical applications.

Recent Trends in Artificial Intelligence-Assisted Coronary Atherosclerotic Plaque Characterization

Coronary artery disease is a major cause of morbidity and mortality worldwide. Its underlying histopathology is the atherosclerotic plaque, which comprises lipid, fibrous and—when chronic—calcium components. Intravascular ultrasound (IVUS) and intravascular optical coherence tomography (IVOCT) performed during invasive coronary angiography are reference standards for characterizing the atherosclerotic plaque. Fine image spatial resolution attainable with contemporary coronary computed tomographic angiography (CCTA) has enabled noninvasive plaque assessment, including identifying features associated with vulnerable plaques known to presage acute coronary events. Manual interpretation of IVUS, IVOCT and CCTA images demands scarce physician expertise and high time cost. This has motivated recent research into and development of artificial intelligence (AI)-assisted methods for image processing, feature extraction, plaque identification and characterization. We performed parallel searches of the medical and technical literature from 1995 to 2021 focusing respectively on human plaque characterization using various imaging modalities and the use of AI-assisted computer aided diagnosis (CAD) to detect and classify atherosclerotic plaques, including their composition and the presence of high-risk features denoting vulnerable plaques. A total of 122 publications were selected for evaluation and the analysis was summarized in terms of data sources, methods—machine versus deep learning—and performance metrics. Trends in AI-assisted plaque characterization are detailed and prospective research challenges discussed. Future directions for the development of accurate and efficient CAD systems to characterize plaque noninvasively using CCTA are proposed.

Automatic Classification of A-Lines in Intravascular OCT Images Using Deep Learning and Estimation of Attenuation Coefficients

Intravascular Optical Coherence Tomography (IVOCT) images provide important insight into every aspect of atherosclerosis. Specifically, the extent of plaque and its type, which are indicative of the patient’s condition, are better assessed by OCT images in comparison to other in vivo modalities. A large amount of imaging data per patient require automatic methods for rapid results. An effective step towards automatic plaque detection and plaque characterization is axial lines (A-lines) based classification into normal and various plaque types. In this work, a novel automatic method for A-line classification is proposed. The method employed convolutional neural networks (CNNs) for classification in its core and comprised the following pre-processing steps: arterial wall segmentation and an OCT-specific (depth-resolved) transformation and a post-processing step based on the majority of classifications. The important step was the OCT-specific transformation, which was based on the estimation of the attenuation coefficient in every pixel of the OCT image. The dataset used for training and testing consisted of 183 images from 33 patients. In these images, four different plaque types were delineated. The method was evaluated by cross-validation. The mean values of accuracy, sensitivity and specificity were 74.73%, 87.78%, and 61.45%, respectively, when classifying into plaque and normal A-lines. When plaque A-lines were classified into fibrolipidic and fibrocalcific, the overall accuracy was 83.47% for A-lines of OCT-specific transformed images and 74.94% for A-lines of original images. This large improvement in accuracy indicates the advantage of using attenuation coefficients when characterizing plaque types. The proposed automatic deep-learning pipeline constitutes a positive contribution to the accurate classification of A-lines in intravascular OCT images.

Coronary Plaque Characterization From Optical Coherence Tomography Imaging With a Two-Pathway Cascade Convolutional Neural Network Architecture

Background: The morphological structure and tissue composition of a coronary atherosclerotic plaque determine its stability, which can be assessed by intravascular optical coherence tomography (OCT) imaging. However, plaque characterization relies on the interpretation of large datasets by well-trained observers. This study aims to develop a convolutional neural network (CNN) method to automatically extract tissue features from OCT images to characterize the main components of a coronary atherosclerotic plaque (fibrous, lipid, and calcification). The method is based on a novel CNN architecture called TwopathCNN, which is utilized in a cascaded structure. According to the evaluation, this proposed method is effective and robust in the characterization of coronary plaque composition from in vivo OCT imaging. On average, the method achieves 0.86 in F1-score and 0.88 in accuracy. The TwopathCNN architecture and cascaded structure show significant improvement in performance (p < 0.05). CNN with cascaded structure can greatly improve the performance of characterization compared to the conventional CNN methods and machine learning methods. This method has a higher efficiency, which may be proven to be a promising diagnostic tool in the detection of coronary plaques.

Learning With Fewer Images via Image Clustering: Application to Intravascular OCT Image Segmentation

Deep learning based methods are routinely used to segment various structures of interest in varied medical imaging modalities. Acquiring annotations for a large number of images requires a skilled analyst, and the process is both time consuming and challenging. Our approach to reduce effort is to reduce the number of images needing detailed annotation. For intravascular optical coherence tomography (IVOCT) image pullbacks, we tested 10% to 100% of training images derived from two schemes: equally-spaced image subsampling and deep-learning- based image clustering. The first strategy involves selecting images at equally spaced intervals from the volume, accounting for the high spatial correlation between neighboring images. In clustering, we used an autoencoder to create a deep feature space representation, performed k-medoids clustering, and then used the cluster medians for training. For coronary calcifications, a baseline U-net model was trained on all images from volumes of interest (VOIs) and compared with fewer images from the sub-sampling strategies. For a given sampling ratio, the clustering based method performed better or similar as compared to the equally spaced sampling approach (e.g., with 10% of images, mean F1 score for calcific class increased from 0.52 to 0.63, with equal spacing and with clustering, respectively). Additionally, for a fixed number of training images, sampling images from more VOIs performed better than otherwise. In conclusion, we recommend the clustering based approach to annotate a small fraction of images, creating a baseline model, which potentially can be improved further by annotating images selected using methods described in active learning research.

Segmentation of Coronary Calcified Plaque in Intravascular OCT Images Using a Two-Step Deep Learning Approach

We developed a fully automated, two-step deep learning approach for characterizing coronary calcified plaque in intravascular optical coherence tomography (IVOCT) images. First, major calcification lesions were detected from an entire pullback using a 3D convolutional neural network (CNN). Second, a SegNet deep learning model with the Tversky loss function was used to segment calcified plaques in the major calcification lesions. The fully connected conditional random field and the frame interpolation of the missing calcification frames were used to reduce classification errors. We trained/tested the networks on a large dataset comprising 8,231 clinical images from 68 patients with 68 vessels and 4,320 ex vivo cadaveric images from 4 hearts with 4 vessels. The 3D CNN model detected major calcifications with high sensitivity (97.7%), specificity (87.7%), and F1 score (0.922). Compared to the standard one-step approach, our two-step deep learning approach significantly improved sensitivity (from 77.5% to 86.2%), precision (from 73.5% to 75.8%), and F1 score (from 0.749 to 0.781). We investigated segmentation performance for varying numbers of training samples; at least 3,900 images were required to obtain stable segmentation results. We also found very small differences in calcification attributes (e.g., angle, thickness, and depth) and identical calcium scores on repetitive pullbacks, indicating excellent reproducibility. Applied to new clinical pullbacks, our method has implications for real-time treatment planning and imaging research.

Automated classification of coronary plaque calcification in OCT pullbacks with 3D deep neural networks

SIGNIFICANCE

Detection and characterization of coronary atherosclerotic plaques often need reviews of a large number of optical coherence tomography (OCT) imaging slices to make a clinical decision. However, it is a challenge to manually review all the slices and consider the interrelationship between adjacent slices.

APPROACH

Inspired by the recent success of deep convolutional network on the classification of medical images, we proposed a ResNet-3D network for classification of coronary plaque calcification in OCT pullbacks. The ResNet-3D network was initialized with a trained ResNet-50 network and a three-dimensional convolution filter filled with zeros padding and non-zeros padding with a convolutional filter. To retrain ResNet-50, we used a dataset of ∼4860 OCT images, derived by 18 entire pullbacks from different patients. In addition, we investigated a two-phase training method to address the data imbalance. For an improved performance, we evaluated different input sizes for the ResNet-3D network, such as 3, 5, and 7 OCT slices. Furthermore, we integrated all ResNet-3D results by majority voting.

RESULTS

A comparative analysis proved the effectiveness of the proposed ResNet-3D networks against ResNet-2D network in the OCT dataset. The classification performance (F1-scores  =  94  %   for non-zeros padding and F1-score  =  96  %   for zeros padding) demonstrated the potential of convolutional neural networks (CNNs) in classifying plaque calcification.

CONCLUSIONS

This work may provide a foundation for further work in extending the CNN to voxel segmentation, which may lead to a supportive diagnostic tool for assessment of coronary plaque vulnerability.

Towards an Optical Biopsy during Visceral Surgical Interventions

Background

Cancer will replace cardiovascular diseases as the most frequent cause of death. Therefore, the goals of cancer treatment are prevention strategies and early detection by cancer screening and ideal stage therapy. From an oncological point of view, complete tumor resection is a significant prognostic factor. Optical coherence tomography (OCT) and confocal laser microscopy (CLM) are two techniques that have the potential to complement intraoperative frozen section analysis as in vivo and real-time optical biopsies.

Summary

In this review we present both procedures and review the progress of evaluation for intraoperative application in visceral surgery. For visceral surgery, there are promising studies evaluating OCT and CLM; however, application during routine visceral surgical interventions is still lacking.

Key Message

OCT and CLM are not competing but complementary approaches of tissue analysis to intraoperative frozen section analysis. Although intraoperative application of OCT and CLM is at an early stage, they are two promising techniques of intraoperative in vivo and real-time tissue examination. Additionally, deep learning strategies provide a significant supplement for automated tissue detection.

Fully automated plaque characterization in intravascular OCT images using hybrid convolutional and lumen morphology features

For intravascular OCT (IVOCT) images, we developed an automated atherosclerotic plaque characterization method that used a hybrid learning approach, which combined deep-learning convolutional and hand-crafted, lumen morphological features. Processing was done on innate A-line units with labels fibrolipidic (fibrous tissue followed by lipidous tissue), fibrocalcific (fibrous tissue followed by calcification), or other. We trained/tested on an expansive data set (6,556 images), and performed an active learning, relabeling step to improve noisy ground truth labels. Conditional random field was an important post-processing step to reduce classification errors. Sensitivities/specificities were 84.8%/97.8% and 91.4%/95.7% for fibrolipidic and fibrocalcific plaques, respectively. Over lesions, en face classification maps showed automated results that agreed favorably to manually labeled counterparts. Adding lumen morphological features gave statistically significant improvement (p < 0.05), as compared to classification with convolutional features alone. Automated assessments of clinically relevant plaque attributes (arc angle and length), compared favorably to those from manual labels. Our hybrid approach gave statistically improved results as compared to previous A-line classification methods using deep learning or hand-crafted features alone. This plaque characterization approach is fully automated, robust, and promising for live-time treatment planning and research applications.

Application and Evaluation of Highly Automated Software for Comprehensive Stent Analysis in Intravascular Optical Coherence Tomography

Intravascular optical coherence tomography (IVOCT) is used to assess stent tissue coverage and malapposition in stent evaluation trials. We developed the OCT Image Visualization and Analysis Toolkit for Stent (OCTivat-Stent), for highly automated analysis of IVOCT pullbacks. Algorithms automatically detected the guidewire, lumen boundary, and stent struts; determined the presence of tissue coverage for each strut; and estimated the stent contour for comparison of stent and lumen area. Strut-level tissue thickness, tissue coverage area, and malapposition area were automatically quantified. The software was used to analyze 292 stent pullbacks. The concordance-correlation-coefficients of automatically measured stent and lumen areas and independent manual measurements were 0.97 and 0.99, respectively. Eleven percent of struts were missed by the software and some artifacts were miscalled as struts giving 1% false-positive strut detection. Eighty-two percent of uncovered struts and 99% of covered struts were labeled correctly, as compared to manual analysis. Using the highly automated software, analysis was harmonized, leading to a reduction of inter-observer variability by 30%. With software assistance, analysis time for a full stent analysis was reduced to less than 30 minutes. Application of this software to stent evaluation trials should enable faster, more reliable analysis with improved statistical power for comparing designs.

Co-registration of pre- and post-stent intravascular OCT images for validation of finite element model simulation of stent expansion

Intravascular optical coherence tomography (IVOCT) provides high-resolution images of coronary calcifications and detailed measurements of acute stent deployment following stent implantation. Since pre- and post-stent IVOCT image "pull-back" acquisitions start from different locations, registration of corresponding pullbacks is needed for assessing treatment outcomes. In particular, we are interested in assessing finite element model (FEM) prediction of lumen gain following stenting, requiring registration. We used deep learning to segment calcifications in corresponding pre- and post-stent IVOCT pullbacks. We created 1D representations of calcium thickness as a function of the angle of the helical IVOCT scans. Registration of two scans was done by maximizing the cross correlation of these two 1D representations. Registration was accurate, as determined by visual comparisons of 2D image frames. We used our pre-stent calcification segmentations to create a lesion-specific FEM, which took into account balloon size, balloon pressure, and stent measurements. We then compared simulated lumen gain from FEM analysis to actual stent deployment results. Actual lumen gain across ~200 registered pre and post-stent images was 1.52 ± 0.51, while FEM prediction was 1.43 ± 0.41. Comparison between actual and FEM results showed no significant difference (p < 0.001), suggesting accurate prediction of FEM modeling. Registered image data showed good visual agreement regarding lumen gain and stent strut malapposition. Hence, we have developed a platform for evaluation of FEM prediction of lumen gain. This platform can be used to guide development of FEM prediction software, which could ultimately help physicians with stent treatment planning of calcified lesions.

Automatic A-line coronary plaque classification using combined deep learning and textural features in intravascular OCT images

We developed a fully automated method for classifying A-line coronary plaques in intravascular optical coherence tomography images using combined deep learning and textural features. The proposed method was trained on 4,292 images from 48 pullbacks giving 80 manually labeled, volumes of interest. Preprocessing steps including guidewire/shadow removal, lumen boundary detection, pixel shifting, and noise reduction were employed. We built a convolutional neural network to extract the deep learning features from the preprocessed image. Traditional textural features were also extracted and combined with deep learning features. Feature selection was performed using the minimum redundancy maximum relevance method. Combined features were utilized as inputs for a random forest classifier. After classification, conditional random field (CRF) method was used for classification noise cleaning. We determined a sub-feature set with the most predictive power. With CRF noise cleaning, sensitivities/specificities were 82.2%/90.8% and 82.4%/89.2% for fibrolipidic and fibrocalcific classes, respectively, with good Dice coefficients. The classification noise cleaning step improved performance metrics by nearly 10-15%. The predicted en face classification maps of entire pullbacks agreed favorably to the manually labeled counterparts. Both assessments suggested that our automated measurements gave clinically relevant results. The proposed method is very promising with regards to both clinical treatment planning and research applications.

An automatic diagnostic system of coronary artery lesions in Kawasaki disease using intravascular optical coherence tomography imaging

Intravascular optical coherence tomography (IV-OCT) is a light-based imaging modality with high resolution, which employs near-infrared light to provide tomographic intracoronary images. Morbidity caused by coronary heart disease is a substantial cause of acute coronary syndrome and sudden cardiac death. The most common intracoronay complications caused by coronary artery disease are intimal hyperplasia, calcification, fibrosis, neovascularization and macrophage accumulation, which require efficient prevention strategies. OCT can provide discriminative information of the intracoronary tissues, which can be used to train a robust fully automatic tissue characterization model based on deep learning. In this study, we aimed to design a diagnostic model of coronary artery lesions. Particularly, we trained a random forest using convolutional neural network features to distinguish between normal and diseased arterial wall structure. Then, based on the arterial wall structure, fully convolutional network is designed to extract the tissue layers in normal cases, and pathological tissues regardless of lesion type in pathological cases. Then, the type of the lesions can be characterized with high precision using our previous model. The results demonstrate the robustness of the model with the approximate overall accuracy up to 90%.

Automated plaque characterization using deep learning on coronary intravascular optical coherence tomographic images

Accurate identification of coronary plaque is very important for cardiologists when treating patients with advanced atherosclerosis. We developed fully-automated semantic segmentation of plaque in intravascular OCT images. We trained/tested a deep learning model on a folded, large, manually annotated clinical dataset. The sensitivities/specificities were 87.4%/89.5% and 85.1%/94.2% for pixel-wise classification of lipidous and calcified plaque, respectively. Automated clinical lesion metrics, potentially useful for treatment planning and research, compared favorably (<4%) with those derived from ground-truth labels. When we converted the results to A-line classification, they were significantly better (p < 0.05) than those obtained previously by using deep learning classifications of A-lines.

Coronary calcification segmentation in intravascular OCT images using deep learning: application to calcification scoring

Major calcifications are of great concern when performing percutaneous coronary interventions because they inhibit proper stent deployment. We created a comprehensive software to segment calcifications in intravascular optical coherence tomography (IVOCT) images and to calculate their impact using the stent-deployment calcification score, as reported by Fujino et al. We segmented the vascular lumen and calcifications using the pretrained SegNet, convolutional neural network, which was refined for our task. We cleaned segmentation results using conditional random field processing. We evaluated the method on manually annotated IVOCT volumes of interest (VOIs) without lesions and with calcifications, lipidous, or mixed lesions. The dataset included 48 VOIs taken from 34 clinical pullbacks, giving a total of 2640 in vivo images. Annotations were determined from consensus between two expert analysts. Keeping VOIs intact, we performed 10-fold cross-validation over all data. Following segmentation noise cleaning, we obtained sensitivities of 0.85 ± 0.04 , 0.99 ± 0.01 , and 0.97 ± 0.01 for calcified, lumen, and other tissue classes, respectively. From segmented regions, we automatically determined calcification depth, angle, and thickness attributes. Bland-Altman analysis suggested strong correlation between manually and automatically obtained lumen and calcification attributes. Agreement between manually and automatically obtained stent-deployment calcification scores was good (four of five lesions gave exact agreement). Results are encouraging and suggest our classification approach could be applied clinically for assessment and treatment planning of coronary calcification lesions.

Automated A-line coronary plaque classification of intravascular optical coherence tomography images using handcrafted features and large datasets

We developed machine learning methods to identify fibrolipidic and fibrocalcific A-lines in intravascular optical coherence tomography (IVOCT) images using a comprehensive set of handcrafted features. We incorporated features developed in previous studies (e.g., optical attenuation and A-line peaks). In addition, we included vascular lumen morphology and three-dimensional (3-D) digital edge and texture features. Classification methods were developed using expansive datasets (∼7000  images), consisting of both clinical in-vivo images and an ex-vivo dataset, which was validated using 3-D cryo-imaging/histology. Conditional random field was used to perform 3-D classification noise cleaning of classification results. We tested various multiclass approaches, classifiers, and feature selection schemes and found that a three-class support vector machine with minimal-redundancy-maximal-relevance feature selection gave the best performance. We found that inclusion of our morphological and 3-D features improved overall classification accuracy. On a held-out test set consisting of >1700 images, we obtained an overall accuracy of 81.58%, with the following (sensitivity/specificity) for each class: other (81.43/89.59), fibrolipidic (94.48/87.32), and fibrocalcific (74.82/95.28). The en-face views of classification results showed that automated classification easily captured the preponderance of a disease segment (e.g., a calcified segment had large regions of fibrocalcific classifications). Finally, we demonstrated proof-of-concept for streamlining A-line classification output with existing fibrolipidic and fibrocalcific boundary segmentation methods, to enable fully automated plaque quantification. The results suggest that our classification approach is a viable step toward fully automated IVOCT plaque classification and segmentation for live-time treatment planning and for offline assessment of drug and biologic therapeutics.

Automated stent coverage analysis in intravascular OCT (IVOCT) image volumes using a support vector machine and mesh growing

Absence of vascular-stent tissue coverage by IVOCT is a biomarker for potential stent-related thrombosis. We developed highly-automated algorithms to classify covered and uncovered struts and quantitatively evaluate stent apposition. We trained a machine learning model on 7,125 images, and included an active learning, relabeling step to improve noisy labels. We obtained uncovered strut classification sensitivity/specificity (94%/90%) comparable to analyst inter-and-intra-observer variability and AUC (0.97), and tissue coverage thickness measurement arguably better than the commercial product. By comparing classification models from regular and relabeled data sets, we observed robustness of the support vector machine to noisy data. A graph-based algorithm detected clusters of uncovered struts thought to pose a greater risk than isolated uncovered struts. The software enables highly-automated, objective, repeatable, comprehensive stent analysis.

Deep learning segmentation of coronary calcified plaque from intravascular optical coherence tomography (IVOCT) images with application to finite element modeling of stent deployment

Because coronary artery calcified plaques can hinder or eliminate stent deployment, interventional cardiologists need a better way to plan interventions, which might include one of the many methods for calcification modification (e.g., atherectomy). We are imaging calcifications with intravascular optical coherence tomography (IVOCT), which is the lone intravascular imaging technique with the ability to image the extent of a calcification, and using results to build vessel-specific finite element models for stent deployment. We applied methods to a large set of image data (>45 lesions and > 2,600 image frames) of calcified plaques, manually segmented by experts into calcified, lumen and "other" tissue classes. In optimization experiments, we evaluated anatomical (x, y) versus acquisition (r,θ) views, augmentation methods, and classification noise cleaning. Noisy semantic segmentations are cleaned by applying a conditional random field (CRF). We achieve an accuracy of 0.85 ± 0.04, 0.99 ± 0.01, and 0.97 ± 0.01, and F-score of 0.88 ± 0.07, 0.97 ± 0.01, and 0.91 ± 0.04 for calcified, lumen, and other tissues classes respectively across all folds following CRF noise cleaning. As a proof of concept, we applied our methods to cadaver heart experiments on highly calcified plaques. Following limited manual correction, we used our calcification segmentations to create a lesion-specific finite element model (FEM) and used it to predict direct stenting deployment at multiple pressure steps. FEM modeling of stent deployment captured many features found in the actual stent deployment (e.g., lumen shape, lumen area, and location and number of apposed stent struts).