Comparison of the performance between Frontier ASPECTS software and different levels of radiologists on assessing CT examinations of acute ischaemic stroke patients

A deep learning approach versus expert clinician panel in the classification of posterior circulation infarction

BACKGROUND

Posterior circulation infarction (POCI) is common. Imaging techniques such as non-contrast-CT (NCCT) and diffusion-weighted-magnetic-resonance-imaging commonly fail to detect hyperacute POCI. Studies suggest expert inspection of Computed Tomography Perfusion (CTP) improves diagnosis of POCI. In many settings, there is limited access to specialist expertise. Deep-learning has been successfully applied to automate imaging interpretation. This study aimed to develop and validate a deep-learning approach for the classification of POCI using CTP.

METHODS

Data were analysed from 3541-patients from the International-stroke-perfusion-registry (INSPIRE). All patients with baseline multimodal-CT and follow-up imaging performed at 24-48 h were identified. A cohort of 541-patients was constructed on a 1:3 POCI-to -reference-ratio for model analysis. A 3D-Dense-Convolutional-Network (DenseNet) was trained to classify patients into POCI or non-POCI using CTP-deconvolved-maps. Six-stroke-experts also independently classified patients based upon stepwise access to multimodal CT (mCT) data. DenseNet results were compared against expert clinician results. Model and clinician performance was evaluated using area-under-the-receiver-operating-curve, sensitivity, specificity, accuracy and precision. Clinician agreement was measured with the Fleiss-Kappa-statistic.

RESULTS

Best mean clinician diagnostic accuracy, sensitivity and agreement was demonstrated after review of all mCT data (AUC: 0.81, Sensitivity: 0.65, Fleiss-Kappa-statistic: 0.73). There was a spectrum of individual clinician results with an AUC-range of 0.73-0.86. Best DenseNet performance was recorded with an input combination of NCCT and delay-time maps. The DenseNet model was superior to the best mean clinician performance (AUC: 0.87) and was due to enhanced sensitivity (DenseNET: 0.77, Clinician: 0.65). The degree to which the DenseNet model outperformed each clinician ranged and was clinician specific (AUC improvement 0.01-0.14).

CONCLUSION

Comprehensive review of CTP improves diagnostic performance and agreement amongst clinicians. A DenseNet model was superior to best mean clinician performance. The degree of improvement varied by specific clinician. Development of a clinician-DenseNet approach may improve inter-clinician agreement and diagnostic accuracy. This approach may alleviate limited specialist services in resource constrained settings.

Automatic assessment of DWI-ASPECTS for acute ischemic stroke based on deep learning

BACKGROUND

Alberta Stroke Program Early Computed Tomography Score (ASPECTS) is a standardized semi-quantitative method for early ischemic changes in acute ischemic stroke.

PURPOSE

However, ASPECTS is still affected by expert experience and inconsistent results between readers in clinical. This study aims to propose an automatic ASPECTS scoring model based on diffusion-weighted imaging (DWI) mode to help clinicians make accurate treatment plans.

METHODS

Eighty-two patients with stroke were included in the study. First, we designed a new deep learning network for segmenting ASPECTS scoring brain regions. The network is improved based on U-net, which integrates multiple modules. Second, we proposed using hybrid classifiers to classify brain regions. For brain regions with larger areas, we used brain grayscale comparison algorithm to train machine learning classifiers, while using hybrid feature training for brain regions with smaller areas.

RESULTS

The average DICE coefficient of the segmented hindbrain area can reach 0.864. With the proposed hybrid classifier, our method performs significantly on both region-level ASPECTS and dichotomous ASPECTS. The sensitivity and accuracy on the test set are 95.51% and 93.43%, respectively. For dichotomous ASPECTS, the intraclass correlation coefficient (ICC) between our automated ASPECTS score and the expert reading was 0.87.

CONCLUSIONS

This study proposed an automated model for ASPECTS scoring of patients with acute ischemic stroke based on DWI images. Experimental results show that the method of segmentation first and then classification is feasible. Our method has the potential to assist physicians in the Alberta Stroke Program with early CT scoring and clinical stroke diagnosis.

Region-specific interobserver agreement of the Alberta Stroke Program Early Computed Tomography Score: A meta-analysis

BACKGROUND AND PURPOSE

The Alberta Stroke Program Early CT Score (ASPECTS) is a widely used scoring system for evaluating ischemic stroke to determine therapeutic strategy. However, there is variation in the interobserver agreement of ASPECTS. This systematic review and meta-analysis aimed to investigate the interobserver agreement of total and regional ASPECTS.

METHODS

A comprehensive search was conducted in the Web of Sciences, PubMed, and Scopus databases to identify relevant studies. Inclusion criteria were studies of noncontrast CT performed within 24 hours of ischemic stroke in the middle cerebral artery territory.

RESULTS

A total of 20 studies, with 3482 patients, reporting interobserver agreement of total and regional ASPECTS were included in the meta-analysis. The interobserver agreement for total ASPECTS in studies using Kappa coefficient (κ) analysis was substantial (κ = .67, 95% confidence interval [CI]: .57-.78). In studies using intraclass correlation coefficient (ICC) analysis, agreement was excellent (ICC = .84, 95% CI: .77-.90). Interobserver agreement was higher in studies in which the observer was unblinded to clinical scenario in both groups (κ = .74, 95% CI: .59-.89, and ICC = .82, 95% CI: .79-.85). Per-region analysis showed that the caudate nucleus had the highest agreement (κ = .68, 95% CI: .60-.76, and ICC = .84, 95% CI: .74-.93), while M2 and internal capsule in Kappa studies (κ = .45, 95% CI: .34-.55 and κ = .47, 95% CI: .28-.66), and M4 and internal capsule in ICC studies (ICC = .54, 95% CI: .43-.64 and ICC = .55, 95% CI: .18-.91) had the lowest agreement.

CONCLUSION

This meta-analysis demonstrates substantial to excellent interobserver agreement for total ASPECTS, which supports using this method for stroke treatment. However, findings emphasize the need to consider interobserver agreement in specific regions of ASPECTS for treatment decisions.

Automated ASPECTS calculation may equal the performance of experienced clinicians: a machine learning study based on a large cohort

OBJECTIVES

The Alberta Stroke Program Early CT Score (ASPECTS) is a semi-quantitative method to evaluate the severity of early ischemic change on non-contrast computed tomography (NCCT) in patients with acute ischemic stroke (AIS). In this work, we propose an automated ASPECTS method based on large cohort of data and machine learning.

METHODS

For this study, we collected 3626 NCCT cases from multiple centers and annotated directly on this dataset by neurologists. Based on image analysis and machine learning methods, we constructed a two-stage machine learning model. The validity and reliability of this automated ASPECTS method were tested on an independent external validation set of 300 cases. Statistical analyses on the total ASPECTS, dichotomized ASPECTS, and region-level ASPECTS were presented.

RESULTS

On an independent external validation set of 300 cases, for the total ASPECTS results, the intraclass correlation coefficient between automated ASPECTS and expert-rated was 0.842. The agreement between ASPECTS threshold of ≥ 6 versus < 6 using a dichotomized method was moderate (κ = 0.438, 0.391-0.477), and the detection rate (sensitivity) was 86.5% for patients with ASPECTS threshold of ≥ 6. Compared with the results of previous studies, our method achieved a slight lead in sensitivity (67.8%) and AUC (0.845), with comparable accuracy (78.9%) and specificity (81.2%).

CONCLUSION

The proposed automated ASPECTS method driven by a large cohort of NCCT images performed equally well compared with expert-rated ASPECTS. This work further demonstrates the validity and reliability of automated ASPECTS evaluation method.

CLINICAL RELEVANCE STATEMENT

The automated ASPECTS method proposed by this study may help AIS patients to receive rapid intervention, but should not be used as a stand-alone diagnostic basis.

KEY POINTS

NCCT-based manual ASPECTS scores were poorly consistent. Machine learning can automate the ASPECTS scoring process. Machine learning model design based on large cohort data can effectively improve the consistency of ASPECTS scores.

Artificial intelligence-driven ASPECTS for the detection of early stroke changes in non-contrast CT: a systematic review and meta-analysis

BACKGROUND

Recent advances in machine learning have enabled development of the automated Alberta Stroke Program Early CT Score (ASPECTS) prediction algorithms using non-contrast enhanced computed tomography (NCCT) scans. The applicability of automated ASPECTS in daily clinical practice is yet to be established. The objective of this meta-analysis was to directly compare the performance of automated and manual ASPECTS predictions in recognizing early stroke changes on NCCT.

METHODS

The MEDLINE, Scopus, and Cochrane databases were searched. The last database search was performed on March 10, 2022. Studies reporting the diagnostic performance and validity of automated ASPECTS software compared with expert readers were included. The outcomes were the interobserver reliability of outputs between ASPECTS versus expert readings, experts versus reference standard, and ASPECTS versus reference standard by means of pooled Fisher's Z transformation of the interclass correlation coefficients (ICCs).

RESULTS

Eleven studies were included in the meta-analysis, involving 1976 patients. The meta-analyses showed good interobserver reliability between experts (ICC 0.72 (95% CI 0.63 to 0.79); p<0.001), moderate reliability in the correlation between automated and expert readings (ICC 0.54 (95% CI 0.40 to 0.67); p<0.001), good reliability between the total expert readings and the reference standard (ICC 0.62 (95% CI 0.52 to 0.71); p<0.001), and good reliability between the automated predictions and the reference standard (ICC 0.72 (95% CI 0.61 to 0.80); p<0.001).

CONCLUSIONS

Artificial intelligence-driven ASPECTS software has comparable or better performance than physicians in terms of recognizing early stroke changes on NCCT.

Automatic Alberta Stroke Program Early Computed Tomographic Scoring in patients with acute ischemic stroke using diffusion-weighted imaging

Alberta Stroke Program Early Computed Tomographic Scoring (ASPECTS) is a reliable method for assessing early ischemic changes in the blood supply area of the middle cerebral artery in patients with acute ischemic stroke. This study aims to propose a deep learning based automatic evaluation strategy for DWI-ASPECTS to serve as a reference for clinicians in urgent decision making for endovascular thrombectomy. Ten ASPECTS regions are extracted from the DWI series to train the independent classification network for each region, the accurate training labels of which are confirmed by neuroradiologists. Two classical convolutional neural networks (VGG-16 and ResNet-50) are validated. Subsequently, the innovative CBAM-VGG is designed to improve the accurate scoring of four small-volume DWI-ASPECTS regions, including caudate nucleus, lenticular nucleus, internal capsule, and insular lobe. Average F1-score of 0.929 and 0.840 and the average accuracy of 94.75% and 84.99% are obtained when scoring on six cortical regions M1-M6 and four small ASPECTS regions, respectively. In addition, the modified algorithm CBAM-VGG shows a significant improvement in the accuracy of estimating the four ASPECTS regions with smaller volumes. The experimental results demonstrate that the deep learning methods facilitate the efficiency and robustness of automatic DWI-ASPECTS scoring, which can provide a reference for clinical decision-making.

Accuracy of Automated Computer-Aided Diagnosis for Stroke Imaging: A Critical Evaluation of Current Evidence

There is increasing interest in computer applications, using artificial intelligence methodologies, to perform health care tasks previously performed by humans, particularly in medical imaging for diagnosis. In stroke, there are now commercial artificial intelligence software for use with computed tomography or MR imaging to identify acute ischemic brain tissue pathology, arterial obstruction on computed tomography angiography or as hyperattenuated arteries on computed tomography, brain hemorrhage, or size of perfusion defects. A rapid, accurate diagnosis may aid treatment decisions for individual patients and could improve outcome if it leads to effective and safe treatment; or conversely, to disaster if a delayed or incorrect diagnosis results in inappropriate treatment. Despite this potential clinical impact, diagnostic tools including artificial intelligence methods are not subjected to the same clinical evaluation standards as are mandatory for drugs. Here, we provide an evidence-based review of the pros and cons of commercially available automated methods for medical imaging diagnosis, including those based on artificial intelligence, to diagnose acute brain pathology on computed tomography or magnetic resonance imaging in patients with stroke.

Automated ASPECTS in acute ischemic stroke: comparison of the overall scores and Hounsfield unit values of two software packages and radiologists with different levels of experience

BACKGROUND

ASPECTS is a simple, rapid, and semi-quantitative method for detecting early ischemic changes (EIC). However, the agreement between software applications and neuroradiologists varies greatly.

PURPOSE

To compare ASPECTS calculated by using automated software tools to neuroradiologists evaluation in patients with acute ischemic stroke (AIS).

MATERIAL AND METHODS

Retrospectively, 61 patients with large vessel occlusion (LVO) who underwent multimodal stroke computed tomography (CT) were evaluated using two automated ASPECTS software tools (NSK and RAPID) and three neuroradiologists with different experiences (two senior neuroradiologists and one junior neuroradiologist). Four weeks later, the same three neuroradiologists re-evaluated the ASPECTS in consensus using the baseline CT and follow-up non-contrast CT (NCCT). Interclass correlation coefficients (ICCs) and Pearson correlation coefficients were applied for statistical analysis.

RESULTS

The HU value exhibited the greatest correlation in the insular lobe (r = 0.81; P < 0.001) and the lowest correlation in the internal capsule (r = 0.65; P < 0.001) between NSK and RAPID. Software analysis and human readers showed excellent agreement with the consensus reading. Compared with the consensus reading, the correlation of the two senior radiologists (ICC = 0.975 and 0.969, respectively) were higher than that of junior radiologist (ICC = 0.869), and the consistency values of the NSK and RAPID software tools after 6 h of onset to imaging (ICC = 0.894 and 0.874, respectively) were greater than those within 6 h of onset (ICC = 0.746 and 0.828, respectively).

CONCLUSION

For patients experiencing AIS due to LVO, the ASPECTS calculated with automated software agrees well with the predefined consensus score but is inferior to that of senior radiologists.

Distinguishing nontuberculous mycobacteria from Mycobacterium tuberculosis lung disease from CT images using a deep learning framework

Purpose

To develop and evaluate the effectiveness of a deep learning framework (3D-ResNet) based on CT images to distinguish nontuberculous mycobacterium lung disease (NTM-LD) from Mycobacterium tuberculosis lung disease (MTB-LD).

Method

Chest CT images of 301 with NTM-LD and 804 with MTB-LD confirmed by pathogenic microbiological examination were retrospectively collected. The differences between the clinical manifestations of the two diseases were analysed. 3D-ResNet was developed to randomly extract data in an 8:1:1 ratio for training, validating, and testing. We also collected external test data (40 with NTM-LD and 40 with MTB-LD) for external validation of the model. The activated region of interest was evaluated using a class activation map. The model was compared with three radiologists in the test set.

Result

Patients with NTM-LD were older than those with MTB-LD, patients with MTB-LD had more cough, and those with NTM-LD had more dyspnoea, and the results were statistically significant (p < 0.05). The AUCs of our model on training, validating, and testing datasets were 0.90, 0.88, and 0.86, respectively, while the AUC on the external test set was 0.78. Additionally, the performance of the model was higher than that of the radiologist, and without manual labelling, the model automatically identified lung areas with abnormalities on CT > 1000 times more effectively than the radiologists.

Conclusion

This study shows the efficacy of 3D-ResNet as a rapid auxiliary diagnostic tool for NTB-LD and MTB-LD. Its use can help provide timely and accurate treatment strategies to patients with these diseases.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05432-x.

Alberta Stroke Program Early CT Score Calculation Using the Deep Learning-Based Brain Hemisphere Comparison Algorithm

OBJECTIVES

The Alberta Stroke Program Early Computed Tomography Score (ASPECTS) is a promising tool for the evaluation of stroke expansion to determine suitability for reperfusion therapy. The aim of this study was to validate deep learning-based ASPECTS calculation software that utilizes a three-dimensional fully convolutional network-based brain hemisphere comparison algorithm (3D-BHCA).

MATERIALS AND METHODS

We retrospectively collected head non-contrast computed tomography (CT) data from 71 patients with acute ischemic stroke and 80 non-stroke patients. The results for ASPECTS on CT assessed by 5 stroke neurologists and by the 3D-BHCA model were compared with the ground truth by means of region-based and score-based analyses.

RESULTS

In total, 151 patients and 3020 (151 × 20) ASPECTS regions were investigated. Median time from onset to CT was 195 min in the stroke patients. In region-based analysis, the sensitivity (0.80), specificity (0.97), and accuracy (0.96) of the 3D-BHCA model were superior to those of stroke neurologists. The sensitivity (0.98), specificity (0.92), and accuracy (0.97) of dichotomized ASPECTS > 5 analysis and the intraclass correlation coefficient (0.90) in total score-based analysis of the 3D-BHCA model were superior to those of stroke neurologists overall. When patients with stroke were stratified by onset-to-CT time, the 3D-BHCA model exhibited the highest performance to calculate ASPECTS, even in the earliest time period.

CONCLUSIONS

The automated ASPECTS calculation software we developed using a deep learning-based algorithm was superior or equal to stroke neurologists in performing ASPECTS calculation in patients with acute stroke and non-stroke patients.