Computer-assisted detection of pulmonary embolism: evaluation of pulmonary CT angiograms performed in an on-call setting

Enhanced classification performance using deep learning based segmentation for pulmonary embolism detection in CT angiography

Purpose

To develop a deep learning-based algorithm that automatically and accurately classifies patients as either having pulmonary emboli or not in CT pulmonary angiography (CTPA) examinations.

Materials and methods

For model development, 700 CTPA examinations from 652 patients performed at a single institution were used, of which 149 examinations contained 1497 PE traced by radiologists. The nnU-Net deep learning-based segmentation framework was trained using 5-fold cross-validation. To enhance classification, we applied logical rules based on PE volume and probability thresholds. External model evaluation was performed in 770 and 34 CTPAs from two independent datasets.

Results

A total of 1483 CTPA examinations were evaluated. In internal cross-validation and test set, the trained model correctly classified 123 of 128 examinations as positive for PE (sensitivity 96.1 %; 95 % C.I. 91-98 %; P < .05) and 521 of 551 as negative (specificity 94.6 %; 95 % C.I. 92-96 %; P < .05), achieving an area under the receiver operating characteristic (AUROC) of 96.4 % (95 % C.I. 79-99 %; P < .05). In the first external test dataset, the trained model correctly classified 31 of 32 examinations as positive (sensitivity 96.9 %; 95 % C.I. 84-99 %; P < .05) and 2 of 2 as negative (specificity 100 %; 95 % C.I. 34-100 %; P < .05), achieving an AUROC of 98.6 % (95 % C.I. 83-100 %; P < .05). In the second external test dataset, the trained model correctly classified 379 of 385 examinations as positive (sensitivity 98.4 %; 95 % C.I. 97-99 %; P < .05) and 346 of 385 as negative (specificity 89.9 %; 95 % C.I. 86-93 %; P < .05), achieving an AUROC of 98.5 % (95 % C.I. 83-100 %; P < .05).

Conclusion

Our automatic pipeline achieved beyond state-of-the-art diagnostic performance of PE in CTPA using nnU-Net for segmentation and volume- and probability-based post-processing for classification.

Automated detection and segmentation of pulmonary embolisms on computed tomography pulmonary angiography (CTPA) using deep learning but without manual outlining

We present a novel computer algorithm to automatically detect and segment pulmonary embolisms (PEs) on computed tomography pulmonary angiography (CTPA). This algorithm is based on deep learning but does not require manual outlines of the PE regions. Given a CTPA scan, both intra- and extra-pulmonary arteries were firstly segmented. The arteries were then partitioned into several parts based on size (radius). Adaptive thresholding and constrained morphological operations were used to identify suspicious PE regions within each part. The confidence of a suspicious region to be PE was scored based on its contrast in the arteries. This approach was applied to the publicly available RSNA Pulmonary Embolism CT Dataset (RSNA-PE) to identify three-dimensional (3-D) PE negative and positive image patches, which were used to train a 3-D Recurrent Residual U-Net (R2-Unet) to automatically segment PE. The feasibility of this computer algorithm was validated on an independent test set consisting of 91 CTPA scans acquired from a different medical institute, where the PE regions were manually located and outlined by a thoracic radiologist (>18 years' experience). An R2-Unet model was also trained and validated on the manual outlines using a 5-fold cross-validation method. The CNN model trained on the high-confident PE regions showed a Dice coefficient of 0.676±0.168 and a false positive rate of 1.86 per CT scan, while the CNN model trained on the manual outlines demonstrated a Dice coefficient of 0.647±0.192 and a false positive rate of 4.20 per CT scan. The former model performed significantly better than the latter model (p<0.01). The promising performance of the developed PE detection and segmentation algorithm suggests the feasibility of training a deep learning network without dedicating significant efforts to manual annotations of the PE regions on CTPA scans.

Pixel-level annotated dataset of computed tomography angiography images of acute pulmonary embolism

Pulmonary embolism has a high incidence and mortality, especially if undiagnosed. The examination of choice for diagnosing the disease is computed tomography pulmonary angiography. As many factors can lead to misinterpretations and diagnostic errors, different groups are utilizing deep learning methods to help improve this process. The diagnostic accuracy of these methods tends to increase by augmenting the training dataset. Deep learning methods can potentially benefit from the use of images acquired with devices from different vendors. To the best of our knowledge, we have developed the first public dataset annotated at the pixel and image levels and the first pixel-level annotated dataset to contain examinations performed with equipment from Toshiba and GE. This dataset includes 40 examinations, half performed with each piece of equipment, representing samples from two medical services. We also included measurements related to the cardiac and circulatory consequences of pulmonary embolism. We encourage the use of this dataset to develop, evaluate and compare the performance of new AI algorithms designed to diagnose PE.

Pulmonary artery diameter correlates with echocardiographic parameters of right ventricular dysfunction in patients with acute pulmonary embolism

Introduction

Right ventricular dysfunction (RVD) is a key component in the process of risk stratification in patients with acute pulmonary embolism (PE). Echocardiography remains the gold standard for RVD assessment, however, measures of RVD may be seen on CTPA imaging, including increased pulmonary artery diameter (PAD). The aim of our study was to evaluate the association between PAD and echocardiographic parameters of RVD in patients with acute PE.

Methods

Retrospective analysis of patients diagnosed with acute PE was conducted at large academic center with an established pulmonary embolism response team (PERT). Patients with available clinical, imaging, and echocardiographic data were included. PAD was compared to echocardiographic markers of RVD. Statistical analysis was performed using the Student's t test, Chi-square test, or one-way analysis of variance (ANOVA); P < 0.05 was considered statistically significant.

Results

270 patients with acute PE were identified. Patients with a PAD >30 mm measured on CTPA had higher rates of RV dilation (73.1% vs 48.7%, P < 0.005), RV systolic dysfunction (65.4% vs 43.7%, P < 0.005), and RVSP >30 mmHg (90.2% vs 68%, P = 0.004), but not TAPSE ≤1.6 cm (39.1% vs 26.1%, P = 0.086). A weak increasing linear relationship between PAD and RVSP was noted (r = 0.379, P = 0.001).

Conclusions

Increased PAD in patients with acute PE was significantly associated with echocardiographic markers of RVD. Increased PAD on CTPA in acute PE can serve as a rapid prognostic tool and assist with PE risk stratification at the time of diagnosis, allowing rapid mobilization of a PERT team and appropriate resource utilization.

How artificial intelligence improves radiological interpretation in suspected pulmonary embolism

Objectives

To evaluate and compare the diagnostic performances of a commercialized artificial intelligence (AI) algorithm for diagnosing pulmonary embolism (PE) on CT pulmonary angiogram (CTPA) with those of emergency radiologists in routine clinical practice.

Methods

This was an IRB-approved retrospective multicentric study including patients with suspected PE from September to December 2019 (i.e., during a preliminary evaluation period of an approved AI algorithm). CTPA quality and conclusions by emergency radiologists were retrieved from radiological reports. The gold standard was a retrospective review of CTPA, radiological and clinical reports, AI outputs, and patient outcomes. Diagnostic performance metrics for AI and radiologists were assessed in the entire cohort and depending on CTPA quality.

Results

Overall, 1202 patients were included (median age: 66.2 years). PE prevalence was 15.8% (190/1202). The AI algorithm detected 219 suspicious PEs, of which 176 were true PEs, including 19 true PEs missed by radiologists. In the cohort, the highest sensitivity and negative predictive values (NPVs) were obtained with AI (92.6% versus 90% and 98.6% versus 98.1%, respectively), while the highest specificity and positive predictive value (PPV) were found with radiologists (99.1% versus 95.8% and 95% versus 80.4%, respectively). Accuracy, specificity, and PPV were significantly higher for radiologists except in subcohorts with poor-to-average injection quality. Radiologists positively evaluated the AI algorithm to improve their diagnostic comfort (55/79 [69.6%]).

Conclusion

Instead of replacing radiologists, AI for PE detection appears to be a safety net in emergency radiology practice due to high sensitivity and NPV, thereby increasing the self-confidence of radiologists.

Key Points

• Both the AI algorithm and emergency radiologists showed excellent performance in diagnosing PE on CTPA (sensitivity and specificity ≥ 90%; accuracy ≥ 95%).

• The AI algorithm for PE detection can help increase the sensitivity and NPV of emergency radiologists in clinical practice, especially in cases of poor-to-moderate injection quality.

• Emergency radiologists recommended the use of AI for PE detection in satisfaction surveys to increase their confidence and comfort in their final diagnosis.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00330-022-08645-2.

Performance of a 3D convolutional neural network in the detection of hypoperfusion at CT pulmonary angiography in patients with chronic pulmonary embolism: a feasibility study

BACKGROUND

Chronic pulmonary embolism (CPE) is a life-threatening disease easily misdiagnosed on computed tomography. We investigated a three-dimensional convolutional neural network (CNN) algorithm for detecting hypoperfusion in CPE from computed tomography pulmonary angiography (CTPA).

METHODS

Preoperative CTPA of 25 patients with CPE and 25 without pulmonary embolism were selected. We applied a 48%-12%-40% training-validation-testing split (12 positive and 12 negative CTPA volumes for training, 3 positives and 3 negatives for validation, 10 positives and 10 negatives for testing). The median number of axial images per CTPA was 335 (min-max, 111-570). Expert manual segmentations were used as training and testing targets. The CNN output was compared to a method in which a Hounsfield unit (HU) threshold was used to detect hypoperfusion. Receiver operating characteristic area under the curve (AUC) and Matthew correlation coefficient (MCC) were calculated with their 95% confidence interval (CI).

RESULTS

The predicted segmentations of CNN showed AUC 0.87 (95% CI 0.82-0.91), those of HU-threshold method 0.79 (95% CI 0.74-0.84). The optimal global threshold values were CNN output probability ≥ 0.37 and ≤ -850 HU. Using these values, MCC was 0.46 (95% CI 0.29-0.59) for CNN and 0.35 (95% CI 0.18-0.48) for HU-threshold method (average difference in MCC in the bootstrap samples 0.11 (95% CI 0.05-0.16). A high CNN prediction probability was a strong predictor of CPE.

CONCLUSIONS

We proposed a deep learning method for detecting hypoperfusion in CPE from CTPA. This model may help evaluating disease extent and supporting treatment planning.

Present Limitations of Artificial Intelligence in the Emergency Setting - Performance Study of a Commercial, Computer-Aided Detection Algorithm for Pulmonary Embolism

PURPOSE

 Since artificial intelligence is transitioning from an experimental stage to clinical implementation, the aim of our study was to evaluate the performance of a commercial, computer-aided detection algorithm of computed tomography pulmonary angiograms regarding the presence of pulmonary embolism in the emergency room.

MATERIALS AND METHODS

 This retrospective study includes all pulmonary computed tomography angiogram studies performed in a large emergency department over a period of 36 months that were analyzed by two radiologists experienced in emergency radiology to set a reference standard. Original reports and computer-aided detection results were compared regarding the detection of lobar, segmental, and subsegmental pulmonary embolism. All computer-aided detection findings were analyzed concerning the underlying pathology. False-positive findings were correlated to the contrast-to-noise ratio.

RESULTS

 Expert reading revealed pulmonary embolism in 182 of 1229 patients (49 % men, 10-97 years) with a total of 504 emboli. The computer-aided detection algorithm reported 3331 findings, including 258 (8 %) true-positive findings and 3073 (92 %) false-positive findings. Computer-aided detection analysis showed a sensitivity of 47 % (95 %CI: 33-61 %) on the lobar level and 50 % (95 %CI 43-56 %) on the subsegmental level. On average, there were 2.25 false-positive findings per study (median 2, range 0-25). There was no significant correlation between the number of false-positive findings and the contrast-to-noise ratio (Spearman's Rank Correlation Coefficient = 0.09). Soft tissue (61.0 %) and pulmonary veins (24.1 %) were the most common underlying reasons for false-positive findings.

CONCLUSION

 Applied to a population at a large emergency room, the tested commercial computer-aided detection algorithm faced relevant performance challenges that need to be addressed in future development projects.

KEY POINTS

  · Computed tomography pulmonary angiograms are frequently acquired in emergency radiology.. · Computer-aided detection algorithms (CADs) can support image analysis.. · CADs face challenges regarding false-positive and false-negative findings.. · Radiologists using CADs need to be aware of these limitations.. · Further software improvements are necessary ahead of implementation in the daily routine..

CITATION FORMAT

· Müller-Peltzer K, Kretzschmar L, Negrão de Figueiredo G et al. Present Limitations of Artificial Intelligence in the Emergency Setting - Performance Study of a Commercial, Computer-Aided Detection Algorithm for Pulmonary Embolism. Fortschr Röntgenstr 2021; DOI: 10.1055/a-1515-2923.

Risk stratification of acute pulmonary embolism based on the clot volume and right ventricular dysfunction on CT pulmonary angiography

OBJECTIVE

To test the feasibility of the clot volume and right ventricular dysfunction for risk stratification of acute pulmonary embolism (APE) patients.

METHODS

CT pulmonary angiography (CTPA) images of 158 APE patients were collected. After excluding 38 (24.1%) patients due to unsatisfactory quality, 120 APE patients (61 males and 59 females) were divided into high-risk (n = 37) and non-high-risk (n = 83) groups. Clot burden was measured by an automated programme (clot volume) and by two semi-quantitative systems (Qanadli and Mastora scores). The ratios of the right ventricular diameter to left ventricular diameter (RVd/LVd) and area (RVa/LVa) were obtained. The correlations amongst the above parameters were analysed. Receiver operating characteristic (ROC) curves were calculated to determine the efficacy of high-risk APE. Multivariate analyses were used to identify the independent predictors.

RESULTS

Strong positive correlations were found between the clot volume and both Qanadli score (r²  = 0.696, P < 0.001) and Mastora score (r²  = 0.728, P < 0.001), and moderate correlations were found between the clot volume and both RVd/LVd (r²  = 0.392, P < 0.001) and RVa/LVa (r²  = 0.389, P < 0.001). The clot volume contributed the highest efficacy (AUC = 0.992) for the identification of high-risk cases, followed by Mastora score (0.968), Qanadli score (0.952), RVa/LVa (0.900) and RVd/LVd (0.892). The clot volume and RVd/LVd were two independent factors of high-risk APE.

CONCLUSIONS

The clot volume is correlated with semi-quantitative clot burden scores and CT measured cardiac parameters. The clot volume and RVd/LVd were two independent factors of high-risk APE patients.

Computer-assisted detection of acute pulmonary embolism at CT pulmonary angiography in children and young adults: a diagnostic performance analysis

Background

To diagnose pulmonary embolism (PE) in children and adults since evaluating tiny pulmonary vasculature beyond segmental level is a challenging and demanding task with thousands of images.

Purpose

To evaluate the effect of computer-assisted detection (CAD) on acute PE on CTPA in children and young adults by readers with varying experience levels.

Material and Methods

Six radiologists were retrospectively divided into three groups according to experience levels and assessed the CTPA studies on a per-emboli basis. All readers identified independently the PE presence, and ranked diagnostic confidence on a 5-point scale with and without CAD. Reading time, sensitivities, specificities, accuracies, positive predictive values (PPVs), and negative predictive values (NPVs) were calculated for each reading.

Results

The sensitivities and NPVs differed significantly in most readers ( P = 0.004, 0.001, 0.010, 0.010, and 0.012 for sensitivities and P = 0.011, 0.003, 0.016, 0.017, and 0.019 for NPVs) except for reader 6 ( P = 0.148 and 0.165, respectively), and the accuracies of all readers differed significantly (all P < 0.05) in peripheral PE (beyond segmental level) detection readings with CAD versus without CAD between two reading methods. The overall time using CAD was longer than those without CAD (76.6 ± 54.4 s vs. 49.4 ± 17.7 s, P = 0.000) for all readers. Significant differences were found for confidence scores in inter-group measurements with CAD ( P = 0.045) and without CAD ( P < 0.001).

Conclusion

At the expense of longer reading time, the use of the CAD algorithms improves sensitivities, NPVs, and the accuracies of readers in peripheral PE detection, especially for readers with a poor level of interpretation experience.

A comparison of computer-assisted detection (CAD) programs for the identification of colorectal polyps: performance and sensitivity analysis, current limitations and practical tips for radiologists

AIM

To directly compare the accuracy and speed of analysis of two commercially available computer-assisted detection (CAD) programs in detecting colorectal polyps.

MATERIALS AND METHOD

In this retrospective single-centre study, patients who had colorectal polyps identified on computed tomography colonography (CTC) and subsequent lower gastrointestinal endoscopy, were analysed using two commercially available CAD programs (CAD1 and CAD2). Results were compared against endoscopy to ascertain sensitivity and positive predictive value (PPV) for colorectal polyps. Time taken for CAD analysis was also calculated.

RESULTS

CAD1 demonstrated a sensitivity of 89.8%, PPV of 17.6% and mean analysis time of 125.8 seconds. CAD2 demonstrated a sensitivity of 75.5%, PPV of 44.0% and mean analysis time of 84.6 seconds.

CONCLUSION

The sensitivity and PPV for colorectal polyps and CAD analysis times can vary widely between current commercially available CAD programs. There is still room for improvement. Generally, there is a trade-off between sensitivity and PPV, and so further developments should aim to optimise both. Information on these factors should be made routinely available, so that an informed choice on their use can be made. This information could also potentially influence the radiologist's use of CAD results.

Multidetector computed tomography pulmonary angiography in childhood acute pulmonary embolism

Pulmonary embolism is a life-threatening condition affecting people of all ages. Multidetector row CT pulmonary angiography has improved the imaging of pulmonary embolism in both adults and children and is now regarded as the routine modality for detection of pulmonary embolism. Advanced CT pulmonary angiography techniques developed in recent years, such as dual-energy CT, have been applied as a one-stop modality for pulmonary embolism diagnosis in children, as they can simultaneously provide anatomical and functional information. We discuss CT pulmonary angiography techniques, common and uncommon findings of pulmonary embolism in both conventional and dual-energy CT pulmonary angiography, and radiation dose considerations.

Missed pulmonary emboli on CT angiography: assessment with pulmonary embolism-computer-aided detection

OBJECTIVE

The purpose of this study is to assess the use of a pulmonary embolism (PE)- computer-aided detection (CADx) program in the detection of PE missed in clinical practice.

MATERIALS AND METHODS

Pulmonary CT angiography (CTA) studies (n = 6769) performed between January 2009 and July 2012 were retrospectively assessed by a thoracic radiologist. In studies that were positive for PE, all prior contrast-enhanced pulmonary CTA studies were reviewed. Missed PE was deemed to have occurred if PE was not described in the final interpretation. The presence, proximal extent, and number of PEs were agreed on by three thoracic radiologists. Studies with missed acute PE and available slice thickness of 2 mm or less were assessed with a prototype PE-CADx program. False-positive PE-CADx marks were analyzed. Outcomes of missed acute PEs were assessed in patients with both follow-up imaging and clinical data.

RESULTS

Fifty-three studies with overlooked acute PE met our inclusion criteria for PE-CADx assessment. The PE-CADx program identified at least one PE in 77.4% of instances (41/53). PE-CADx correctly marked at least one PE in 23 of 23 cases (100%) with multiple PEs and 18 of 30 (60%) cases with a solitary PE (p < 0.001). PE-CADx per-study sensitivity was significantly higher for segmental (65.5%) than for subsegmental (91.7%) PEs (p = 0.002). PE-CADx averaged 3.8 false-positive marks per case (range, 0-23 marks). Fourteen patients with missed PE who were not receiving anticoagulation therapy developed new PEs, including nine with an isolated subsegmental PE on the initial CT scan.

CONCLUSION

PE-CADx correctly identified 77.4% of cases of acute PE that were previously missed in clinical practice.

Stand-alone performance of a computer-assisted detection prototype for detection of acute pulmonary embolism: a multi-institutional comparison

OBJECTIVE

To assess whether the performance of a computer-assisted detection (CAD) algorithm for acute pulmonary embolism (PE) differs in pulmonary CT angiographies acquired at various institutions.

METHODS

In this retrospective study, we included 40 consecutive scans with and 40 without PE from 3 institutions (n = 240) using 64-slice scanners made by different manufacturers (General Electric; Philips; Siemens). CAD markers were classified as true or false positive (FP) using independent evaluation by two readers and consultation of a third chest radiologist in discordant cases. Image quality parameters were subjectively scored using 4/5-point scales. Image noise and vascular enhancement were measured. Statistical analysis was done to correlate image quality of the three institutions with CAD stand-alone performance.

RESULTS

Patient groups were comparable with respect to age (p = 0.22), accompanying lung disease (p = 0.12) and inpatient/outpatient ratio (p = 0.67). The sensitivity was 100% (34/34), 97% (37/38) and 92% (33/36), and the specificity was 18% (8/44), 15% (6/41) and 13% (5/39). Neither significantly differed between the institutions (p = 0.21 and p = 0.820, respectively). The mean number of FP findings (4.5, 6.2 and 3.7) significantly varied (p = 0.02 and p = 0.03), but median numbers (2, 3 and 3) were comparable. Image quality parameters were significantly associated with the number of FP findings (p<0.05) but not with sensitivity. After correcting for noise and vascular enhancement, the number of FPs did not significantly differ between the three institutions (p = 0.43).

CONCLUSIONS

CAD stand-alone performance is independent of scanner type but strongly related to image quality and thus scanning protocols.