A multistage approach to improve performance of computer-aided detection of pulmonary embolisms depicted on CT images: preliminary investigation

Seeking an optimal approach for Computer-aided Diagnosis of Pulmonary Embolism

Pulmonary Embolism (PE) represents a thrombus ("blood clot"), usually originating from a lower extremity vein, that travels to the blood vessels in the lung, causing vascular obstruction and in some patients death. This disorder is commonly diagnosed using Computed Tomography Pulmonary Angiography (CTPA). Deep learning holds great promise for the Computer-aided Diagnosis (CAD) of PE. However, numerous deep learning methods, such as Convolutional Neural Networks (CNN) and Transformer-based models, exist for a given task, causing great confusion regarding the development of CAD systems for PE. To address this confusion, we present a comprehensive analysis of competing deep learning methods applicable to PE diagnosis based on four datasets. First, we use the RSNA PE dataset, which includes (weak) slice-level and exam-level labels, for PE classification and diagnosis, respectively. At the slice level, we compare CNNs with the Vision Transformer (ViT) and the Swin Transformer. We also investigate the impact of self-supervised versus (fully) supervised ImageNet pre-training, and transfer learning over training models from scratch. Additionally, at the exam level, we compare sequence model learning with our proposed transformer-based architecture, Embedding-based ViT (E-ViT). For the second and third datasets, we utilize the CAD-PE Challenge Dataset and Ferdowsi University of Mashad's PE Dataset, where we convert (strong) clot-level masks into slice-level annotations to evaluate the optimal CNN model for slice-level PE classification. Finally, we use our in-house PE-CAD dataset, which contains (strong) clot-level masks. Here, we investigate the impact of our vessel-oriented image representations and self-supervised pre-training on PE false positive reduction at the clot level across image dimensions (2D, 2.5D, and 3D). Our experiments show that (1) transfer learning boosts performance despite differences between photographic images and CTPA scans; (2) self-supervised pre-training can surpass (fully) supervised pre-training; (3) transformer-based models demonstrate comparable performance but slower convergence compared with CNNs for slice-level PE classification; (4) model trained on the RSNA PE dataset demonstrates promising performance when tested on unseen datasets for slice-level PE classification; (5) our E-ViT framework excels in handling variable numbers of slices and outperforms sequence model learning for exam-level diagnosis; and (6) vessel-oriented image representation and self-supervised pre-training both enhance performance for PE false positive reduction across image dimensions. Our optimal approach surpasses state-of-the-art results on the RSNA PE dataset, enhancing AUC by 0.62% (slice-level) and 2.22% (exam-level). On our in-house PE-CAD dataset, 3D vessel-oriented images improve performance from 80.07% to 91.35%, a remarkable 11% gain. Codes are available at GitHub.com/JLiangLab/CAD_PE.

Automated detection and segmentation of pulmonary embolisms on computed tomography pulmonary angiography (CTPA) using deep learning but without manual outlining

We present a novel computer algorithm to automatically detect and segment pulmonary embolisms (PEs) on computed tomography pulmonary angiography (CTPA). This algorithm is based on deep learning but does not require manual outlines of the PE regions. Given a CTPA scan, both intra- and extra-pulmonary arteries were firstly segmented. The arteries were then partitioned into several parts based on size (radius). Adaptive thresholding and constrained morphological operations were used to identify suspicious PE regions within each part. The confidence of a suspicious region to be PE was scored based on its contrast in the arteries. This approach was applied to the publicly available RSNA Pulmonary Embolism CT Dataset (RSNA-PE) to identify three-dimensional (3-D) PE negative and positive image patches, which were used to train a 3-D Recurrent Residual U-Net (R2-Unet) to automatically segment PE. The feasibility of this computer algorithm was validated on an independent test set consisting of 91 CTPA scans acquired from a different medical institute, where the PE regions were manually located and outlined by a thoracic radiologist (>18 years' experience). An R2-Unet model was also trained and validated on the manual outlines using a 5-fold cross-validation method. The CNN model trained on the high-confident PE regions showed a Dice coefficient of 0.676±0.168 and a false positive rate of 1.86 per CT scan, while the CNN model trained on the manual outlines demonstrated a Dice coefficient of 0.647±0.192 and a false positive rate of 4.20 per CT scan. The former model performed significantly better than the latter model (p<0.01). The promising performance of the developed PE detection and segmentation algorithm suggests the feasibility of training a deep learning network without dedicating significant efforts to manual annotations of the PE regions on CTPA scans.

Pixel-level annotated dataset of computed tomography angiography images of acute pulmonary embolism

Pulmonary embolism has a high incidence and mortality, especially if undiagnosed. The examination of choice for diagnosing the disease is computed tomography pulmonary angiography. As many factors can lead to misinterpretations and diagnostic errors, different groups are utilizing deep learning methods to help improve this process. The diagnostic accuracy of these methods tends to increase by augmenting the training dataset. Deep learning methods can potentially benefit from the use of images acquired with devices from different vendors. To the best of our knowledge, we have developed the first public dataset annotated at the pixel and image levels and the first pixel-level annotated dataset to contain examinations performed with equipment from Toshiba and GE. This dataset includes 40 examinations, half performed with each piece of equipment, representing samples from two medical services. We also included measurements related to the cardiac and circulatory consequences of pulmonary embolism. We encourage the use of this dataset to develop, evaluate and compare the performance of new AI algorithms designed to diagnose PE.

Weakly supervised attention model for RV strain classification from volumetric CTPA scans

BACKGROUND AND OBJECTIVE

Evaluation of the right ventricle (RV) is a key component of the clinical assessment of many cardiovascular and pulmonary disorders. In this work, we focus on RV strain classification from patients who were diagnosed with pulmonary embolism (PE) in computed tomography pulmonary angiography (CTPA) scans. PE is a life-threatening condition, often without warning signs or symptoms. Early diagnosis and accurate risk stratification are critical for decreasing mortality rates. High-risk PE relies on the presence of RV dysfunction resulting from acute pressure overload. PE severity classification and specifically, high-risk PE diagnosis are crucial for appropriate therapy. CTPA is the golden standard in the diagnostic workup of suspected PE. Therefore, it can link between diagnosis and risk stratification strategies.

METHODS

We retrieved data of consecutive patients who underwent CTPA and were diagnosed with PE and extracted a single binary label of "RV strain biomarker" from the CTPA scan report. This label was used as a weak label for classification. Our solution applies a 3D DenseNet network architecture, further improved by integrating residual attention blocks into the network's layers.

RESULTS

This model achieved an area under the receiver operating characteristic curve (AUC) of 0.88 for classifying RV strain. For Youden's index, the model showed a sensitivity of 87% and specificity of 83.7%. Our solution outperforms state-of-the-art 3D CNN networks. The proposed design allows for a fully automated network that can be trained easily in an end-to-end manner without requiring computationally intensive and time-consuming preprocessing or strenuous labeling of the data.

CONCLUSIONS

This current solution demonstrates that a small dataset of readily available unmarked CTPAs can be used for effective RV strain classification. To our knowledge, this is the first work that attempts to solve the problem of RV strain classification from CTPA scans and this is the first work where medical images are used in such an architecture. Our generalized self-attention blocks can be incorporated into various existing classification architectures making this a general methodology that can be applied to 3D medical datasets.

Artificial Intelligence Evidence-Based Current Status and Potential for Lower Limb Vascular Management

Consultation prioritization is fundamental in optimal healthcare management and its performance can be helped by artificial intelligence (AI)-dedicated software and by digital medicine in general. The need for remote consultation has been demonstrated not only in the pandemic-induced lock-down but also in rurality conditions for which access to health centers is constantly limited. The term “AI” indicates the use of a computer to simulate human intellectual behavior with minimal human intervention. AI is based on a “machine learning” process or on an artificial neural network. AI provides accurate diagnostic algorithms and personalized treatments in many fields, including oncology, ophthalmology, traumatology, and dermatology. AI can help vascular specialists in diagnostics of peripheral artery disease, cerebrovascular disease, and deep vein thrombosis by analyzing contrast-enhanced magnetic resonance imaging or ultrasound data and in diagnostics of pulmonary embolism on multi-slice computed angiograms. Automatic methods based on AI may be applied to detect the presence and determine the clinical class of chronic venous disease. Nevertheless, data on using AI in this field are still scarce. In this narrative review, the authors discuss available data on AI implementation in arterial and venous disease diagnostics and care.

From Early Morphometrics to Machine Learning-What Future for Cardiovascular Imaging of the Pulmonary Circulation?

Imaging plays a cardinal role in the diagnosis and management of diseases of the pulmonary circulation. Behind the picture itself, every digital image contains a wealth of quantitative data, which are hardly analysed in current routine clinical practice and this is now being transformed by radiomics. Mathematical analyses of these data using novel techniques, such as vascular morphometry (including vascular tortuosity and vascular volumes), blood flow imaging (including quantitative lung perfusion and computational flow dynamics), and artificial intelligence, are opening a window on the complex pathophysiology and structure-function relationships of pulmonary vascular diseases. They have the potential to make dramatic alterations to how clinicians investigate the pulmonary circulation, with the consequences of more rapid diagnosis and a reduction in the need for invasive procedures in the future. Applied to multimodality imaging, they can provide new information to improve disease characterization and increase diagnostic accuracy. These new technologies may be used as sophisticated biomarkers for risk prediction modelling of prognosis and for optimising the long-term management of pulmonary circulatory diseases. These innovative techniques will require evaluation in clinical trials and may in themselves serve as successful surrogate end points in trials in the years to come.

PENet-a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging

Pulmonary embolism (PE) is a life-threatening clinical problem and computed tomography pulmonary angiography (CTPA) is the gold standard for diagnosis. Prompt diagnosis and immediate treatment are critical to avoid high morbidity and mortality rates, yet PE remains among the diagnoses most frequently missed or delayed. In this study, we developed a deep learning model-PENet, to automatically detect PE on volumetric CTPA scans as an end-to-end solution for this purpose. The PENet is a 77-layer 3D convolutional neural network (CNN) pretrained on the Kinetics-600 dataset and fine-tuned on a retrospective CTPA dataset collected from a single academic institution. The PENet model performance was evaluated in detecting PE on data from two different institutions: one as a hold-out dataset from the same institution as the training data and a second collected from an external institution to evaluate model generalizability to an unrelated population dataset. PENet achieved an AUROC of 0.84 [0.82-0.87] on detecting PE on the hold out internal test set and 0.85 [0.81-0.88] on external dataset. PENet also outperformed current state-of-the-art 3D CNN models. The results represent successful application of an end-to-end 3D CNN model for the complex task of PE diagnosis without requiring computationally intensive and time consuming preprocessing and demonstrates sustained performance on data from an external institution. Our model could be applied as a triage tool to automatically identify clinically important PEs allowing for prioritization for diagnostic radiology interpretation and improved care pathways via more efficient diagnosis.

Survey on deep learning for pulmonary medical imaging

As a promising method in artificial intelligence, deep learning has been proven successful in several domains ranging from acoustics and images to natural language processing. With medical imaging becoming an important part of disease screening and diagnosis, deep learning-based approaches have emerged as powerful techniques in medical image areas. In this process, feature representations are learned directly and automatically from data, leading to remarkable breakthroughs in the medical field. Deep learning has been widely applied in medical imaging for improved image analysis. This paper reviews the major deep learning techniques in this time of rapid evolution and summarizes some of its key contributions and state-of-the-art outcomes. The topics include classification, detection, and segmentation tasks on medical image analysis with respect to pulmonary medical images, datasets, and benchmarks. A comprehensive overview of these methods implemented on various lung diseases consisting of pulmonary nodule diseases, pulmonary embolism, pneumonia, and interstitial lung disease is also provided. Lastly, the application of deep learning techniques to the medical image and an analysis of their future challenges and potential directions are discussed.

Computer-aided detection and visualization of pulmonary embolism using a novel, compact, and discriminative image representation

Diagnosing pulmonary embolism (PE) and excluding disorders that may clinically and radiologically simulate PE poses a challenging task for both human and machine perception. In this paper, we propose a novel vessel-oriented image representation (VOIR) that can improve the machine perception of PE through a consistent, compact, and discriminative image representation, and can also improve radiologists' diagnostic capabilities for PE assessment by serving as the backbone of an effective PE visualization system. Specifically, our image representation can be used to train more effective convolutional neural networks for distinguishing PE from PE mimics, and also allows radiologists to inspect the vessel lumen from multiple perspectives, so that they can report filling defects (PE), if any, with confidence. Our image representation offers four advantages: (1) Efficiency and compactness-concisely summarizing the 3D contextual information around an embolus in only three image channels, (2) consistency-automatically aligning the embolus in the 3-channel images according to the orientation of the affected vessel, (3) expandability-naturally supporting data augmentation for training CNNs, and (4) multi-view visualization-maximally revealing filling defects. To evaluate the effectiveness of VOIR for PE diagnosis, we use 121 CTPA datasets with a total of 326 emboli. We first compare VOIR with two other compact alternatives using six CNN architectures of varying depths and under varying amounts of labeled training data. Our experiments demonstrate that VOIR enables faster training of a higher-performing model compared to the other compact representations, even in the absence of deep architectures and large labeled training sets. Our experiments comparing VOIR with the 3D image representation further demonstrate that the 2D CNN trained with VOIR achieves a significant performance gain over the 3D CNNs. Our robustness analyses also show that the suggested PE CAD is robust to the choice of CT scanner machines and the physical size of crops used for training. Finally, our PE CAD is ranked second at the PE challenge in the category of 0 mm localization error.

Segmentation of pulmonary vascular tree by incorporating vessel enhancement filter and variational region-growing

BACKGROUND

Automatic segmentation of pulmonary vascular tree in the thoracic computed tomography (CT) image is a promising but challenging task with great clinical potential values. It is difficult to segment the whole vascular tree in reasonable time and acceptable accuracy.

OBJECTIVE

To develop a novel pulmonary vessel segmentation approach by incorporating vessel enhancement filters and the anisotropic diffusion filter with the variational region growing.

METHODS

First, the airway wall from the lung lobes is eliminated from CT images by using multi-scale morphological operations. Second, a Hessian-based multi-scale vesselness filter and medialness filter are applied to detect and enhance the potential vessel. Third, an anisotropic diffusion filter is used to remove noise and enhance the tube-like structures in CT images. Last, the vascular tree is segmented by applying variational region growing algorithm.

RESULTS

Applying to the CT images collected from the entire dataset of VESSEL12 challenge, we achieved an average sensitivity of 92.9%, specificity of 91.6% and the area under the ROC curve of AUC = 0.972.

CONCLUSIONS

This study demonstrated feasibility of segmenting the pulmonary vessel effectively by incorporating vessel enhancement filters and the anisotropic diffusion filter with the variational region growing algorithm. Our method cannot only segment both large and peripheral vessels, but also distinguish the vessels from the adjacent tissues, especially the airway walls.

A new dataset of computed-tomography angiography images for computer-aided detection of pulmonary embolism

The lack of publicly available datasets of computed-tomography angiography (CTA) images for pulmonary embolism (PE) is a problem felt by physicians and researchers. Although a number of computer-aided detection (CAD) systems have been developed for PE diagnosis, their performance is often evaluated using private datasets. In this paper, we introduce a new public dataset called FUMPE (standing for Ferdowsi University of Mashhad's PE dataset) which consists of three-dimensional PE-CTA images of 35 different subjects with 8792 slices in total. For each benchmark image, two expert radiologists provided the ground-truth with the assistance of a semi-automated image processing software tool. FUMPE is a challenging benchmark for CAD methods because of the large number (i.e., 3438) of PE regions and, more especially, because of the location of most of them (i.e., 67%) in lung peripheral arteries. Moreover, due to the reporting of the Qanadli score for each PE-CTA image, FUMPE is the first public dataset which can be used for the analysis of mortality and morbidity risks associated with PE. We also report some complementary prognosis information for each subject.

Fusion of Quantitative Image and Genomic Biomarkers to Improve Prognosis Assessment of Early Stage Lung Cancer Patients

OBJECTIVE

This study aims to develop a new quantitative image feature analysis scheme and investigate its role along with two genomic biomarkers, namely protein expression of the excision repair cross-complementing 1 genes and a regulatory subunit of ribonucleotide reductase (RRM1), in predicting cancer recurrence risk of stage I nonsmall-cell lung cancer (NSCLC) patients after surgery.

METHODS

By using chest computed tomography images, we developed a computer-aided detection scheme to segment lung tumors and computed tumor-related image features. After feature selection, we trained a Naïve Bayesian network-based classifier using eight image features and a multilayer perceptron classifier using two genomic biomarkers to predict cancer recurrence risk, respectively. Two classifiers were trained and tested using a dataset with 79 stage I NSCLC cases, a synthetic minority oversampling technique and a leave-one-case-out validation method. A fusion method was also applied to combine prediction scores of two classifiers.

RESULTS

Areas under ROC curves (AUC) values are 0.78 ± 0.06 and 0.68 ± 0.07 when using the image feature and genomic biomarker-based classifiers, respectively. AUC value significantly increased to 0.84 ± 0.05 ( ) when fusion of two classifier-generated prediction scores using an equal weighting factor.

CONCLUSION

A quantitative image feature-based classifier yielded significantly higher discriminatory power than a genomic biomarker-based classifier in predicting cancer recurrence risk. Fusion of prediction scores generated by the two classifiers further improved prediction performance.

SIGNIFICANCE

We demonstrated a new approach that has potential to assist clinicians in more effectively managing stage I NSCLC patients to reduce cancer recurrence risk.

Adhesion Pulmonary Nodules Detection Based on Dot-Filter and Extracting Centerline Algorithm

A suspected pulmonary nodule detection method was proposed based on dot-filter and extracting centerline algorithm. In this paper, we focus on the distinguishing adhesion pulmonary nodules attached to vessels in two-dimensional (2D) lung computed tomography (CT) images. Firstly, the dot-filter based on Hessian matrix was constructed to enhance the circular area of the pulmonary CT images, which enhanced the circular suspected pulmonary nodule and suppresses the line-like areas. Secondly, to detect the nondistinguishable attached pulmonary nodules by the dot-filter, an algorithm based on extracting centerline was developed to enhance the circle area formed by the end or head of the vessels including the intersection of the lines. 20 sets of CT images were used in the experiments. In addition, 20 true/false nodules extracted were used to test the function of classifier. The experimental results show that the method based on dot-filter and extracting centerline algorithm can detect the attached pulmonary nodules accurately, which is a basis for further studies on the pulmonary nodule detection and diagnose.

Max-AUC feature selection in computer-aided detection of polyps in CT colonography

We propose a feature selection method based on a sequential forward floating selection (SFFS) procedure to improve the performance of a classifier in computerized detection of polyps in CT colonography (CTC). The feature selection method is coupled with a nonlinear support vector machine (SVM) classifier. Unlike the conventional linear method based on Wilks' lambda, the proposed method selected the most relevant features that would maximize the area under the receiver operating characteristic curve (AUC), which directly maximizes classification performance, evaluated based on AUC value, in the computer-aided detection (CADe) scheme. We presented two variants of the proposed method with different stopping criteria used in the SFFS procedure. The first variant searched all feature combinations allowed in the SFFS procedure and selected the subsets that maximize the AUC values. The second variant performed a statistical test at each step during the SFFS procedure, and it was terminated if the increase in the AUC value was not statistically significant. The advantage of the second variant is its lower computational cost. To test the performance of the proposed method, we compared it against the popular stepwise feature selection method based on Wilks' lambda for a colonic-polyp database (25 polyps and 2624 nonpolyps). We extracted 75 morphologic, gray-level-based, and texture features from the segmented lesion candidate regions. The two variants of the proposed feature selection method chose 29 and 7 features, respectively. Two SVM classifiers trained with these selected features yielded a 96% by-polyp sensitivity at false-positive (FP) rates of 4.1 and 6.5 per patient, respectively. Experiments showed a significant improvement in the performance of the classifier with the proposed feature selection method over that with the popular stepwise feature selection based on Wilks' lambda that yielded 18.0 FPs per patient at the same sensitivity level.