Interobserver variability in quality assessment of magnetic resonance images

How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

OBJECTIVES

The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans.

METHODS

Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC).

RESULTS

Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396-0.814) to 0.785 (95%CI 0.676-0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034-0.500) to 0.562 (95%CI 0.337-0.729). Intraobserver agreement increased for four out of six observers.

CONCLUSION

The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement.

CLINICAL RELEVANCE STATEMENT

This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms.

KEY POINTS

• Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies.

Assessment of image quality and establishment of local acceptable quality dose for computed tomography based on patient effective diameter

Purpose

We aim to develop modified clinical indication (CI)-based image quality scoring criteria (IQSC) for assessing image quality (IQ) and establishing acceptable quality doses (AQDs) in adult computed tomography (CT) examinations, based on CIs and patient sizes.

Approach

CT images, volume CT dose index (CTDI vol ), and dose length product (DLP) were collected retrospectively between September 2020 and September 2021 for eight common CIs from two CT scanners at a central hospital in the Kingdom of Bahrain. Using the modified CI-based IQSC and a Likert scale (0 to 4), three radiologists assessed the IQ of each examination. AQDs were then established as the median value ofCTDI vol and DLP for images with an average score of 3 and compared to national diagnostic reference levels (NDRLs).

Results

Out of 581 examinations, 60 were excluded from the study due to average scores above or below 3. The established AQDs were lower than the NDRLs for all CIs, except AQDs / CTDI vol for oncologic follow-up for large patients (28 versus 26 mGy) in scanner A, besides abdominal pain for medium patients (16 versus 15 mGy) and large patients (34 versus 27 mGy), and diverticulitis/appendicitis for medium patients (15 versus 12 mGy) and large patients (33 versus 30 mGy) in scanner B, indicating the need for optimization.

Conclusions

CI-based IQSC is crucial for IQ assessment and establishing AQDs according to patient size. It identifies stations requiring optimization of patient radiation exposure.

CT radiomics to differentiate neuroendocrine neoplasm from adenocarcinoma in patients with a peripheral solid pulmonary nodule: a multicenter study

Purpose

To construct and validate a computed tomography (CT) radiomics model for differentiating lung neuroendocrine neoplasm (LNEN) from lung adenocarcinoma (LADC) manifesting as a peripheral solid nodule (PSN) to aid in early clinical decision-making.

Methods

A total of 445 patients with pathologically confirmed LNEN and LADC from June 2016 to July 2023 were retrospectively included from five medical centers. Those patients were split into the training set (n = 316; 158 LNEN) and external test set (n = 129; 43 LNEN), the former including the cross-validation (CV) training set and CV test set using ten-fold CV. The support vector machine (SVM) classifier was used to develop the semantic, radiomics and merged models. The diagnostic performances were evaluated by the area under the receiver operating characteristic curve (AUC) and compared by Delong test. Preoperative neuron-specific enolase (NSE) levels were collected as a clinical predictor.

Results

In the training set, the AUCs of the radiomics model (0.878 [95% CI: 0.836, 0.915]) and merged model (0.884 [95% CI: 0.844, 0.919]) significantly outperformed the semantic model (0.718 [95% CI: 0.663, 0.769], p both<.001). In the external test set, the AUCs of the radiomics model (0.787 [95% CI: 0.696, 0.871]), merged model (0.807 [95%CI: 0.720, 0.889]) and semantic model (0.729 [95% CI: 0.631, 0.811]) did not exhibit statistical differences. The radiomics model outperformed NSE in sensitivity in the training set (85.3% vs 20.0%; p <.001) and external test set (88.9% vs 40.7%; p = .002).

Conclusion

The CT radiomics model could non-invasively, effectively and sensitively predict LNEN and LADC presenting as a PSN to assist in treatment strategy selection.

Deep Learning-based Approach for Brainstem and Ventricular MR Planimetry: Application in Patients with Progressive Supranuclear Palsy

Purpose To develop a fast and fully automated deep learning (DL)-based method for the MRI planimetric segmentation and measurement of the brainstem and ventricular structures most affected in patients with progressive supranuclear palsy (PSP). Materials and Methods In this retrospective study, T1-weighted MR images in healthy controls (n = 84) were used to train DL models for segmenting the midbrain, pons, middle cerebellar peduncle (MCP), superior cerebellar peduncle (SCP), third ventricle, and frontal horns (FHs). Internal, external, and clinical test datasets (n = 305) were used to assess segmentation model reliability. DL masks from test datasets were used to automatically extract midbrain and pons areas and the width of MCP, SCP, third ventricle, and FHs. Automated measurements were compared with those manually performed by an expert radiologist. Finally, these measures were combined to calculate the midbrain to pons area ratio, MR parkinsonism index (MRPI), and MRPI 2.0, which were used to differentiate patients with PSP (n = 71) from those with Parkinson disease (PD) (n = 129). Results Dice coefficients above 0.85 were found for all brain regions when comparing manual and DL-based segmentations. A strong correlation was observed between automated and manual measurements (Spearman ρ > 0.80, P < .001). DL-based measurements showed excellent performance in differentiating patients with PSP from those with PD, with an area under the receiver operating characteristic curve above 0.92. Conclusion The automated approach successfully segmented and measured the brainstem and ventricular structures. DL-based models may represent a useful approach to support the diagnosis of PSP and potentially other conditions associated with brainstem and ventricular alterations. Keywords: MR Imaging, Brain/Brain Stem, Segmentation, Quantification, Diagnosis, Convolutional Neural Network Supplemental material is available for this article. © RSNA, 2024 See also the commentary by Mohajer in this issue.

Lightweight preprocessing and template matching facilitate streamlined ischemic myocardial scar classification

Purpose

Ischemic myocardial scarring (IMS) is a common outcome of coronary artery disease that potentially leads to lethal arrythmias and heart failure. Late-gadolinium-enhanced cardiac magnetic resonance (CMR) imaging scans have served as the diagnostic bedrock for IMS, with recent advancements in machine learning enabling enhanced scar classification. However, the trade-off for these improvements is intensive computational and time demands. As a solution, we propose a combination of lightweight preprocessing (LWP) and template matching (TM) to streamline IMS classification.

Approach

CMR images from 279 patients (151 IMS, 128 control) were classified for IMS presence using two convolutional neural networks (CNNs) and TM, both with and without LWP. Evaluation metrics included accuracy, sensitivity, specificity, F1-score, area under the receiver operating characteristic curve (AUROC), and processing time. External testing dataset analysis encompassed patient-level classifications (PLCs) and a CNN versus TM classification comparison (CVTCC).

Results

LWP enhanced the speed of both CNNs (4.9x) and TM (21.9x). Furthermore, in the absence of LWP, TM outpaced CNNs by over 10x, while with LWP, TM was more than 100x faster. Additionally, TM performed similarly to the CNNs in accuracy, sensitivity, specificity, F1-score, and AUROC, with PLCs demonstrating improvements across all five metrics. Moreover, the CVTCC revealed a substantial 90.9% agreement.

Conclusions

Our results highlight the effectiveness of LWP and TM in streamlining IMS classification. Anticipated enhancements to LWP's region of interest (ROI) isolation and TM's ROI targeting are expected to boost accuracy, positioning them as a potential alternative to CNNs for IMS classification, supporting the need for further research.

Deep learning-based high-accuracy detection for lumbar and cervical degenerative disease on T2-weighted MR images

PURPOSE

To develop and validate a deep learning (DL) model for detecting lumbar degenerative disease in both sagittal and axial views of T2-weighted MRI and evaluate its generalized performance in detecting cervical degenerative disease.

METHODS

T2-weighted MRI scans of 804 patients with symptoms of lumbar degenerative disease were retrospectively collected from three hospitals. The training dataset (n = 456) and internal validation dataset (n = 134) were randomly selected from the center I. Two external validation datasets comprising 100 and 114 patients were from center II and center III, respectively. A DL model based on 3D ResNet18 and transformer architecture was proposed to detect lumbar degenerative disease. In addition, a cervical MR image dataset comprising 200 patients from an independent hospital was used to evaluate the generalized performance of the DL model. The diagnostic performance was assessed by the free-response receiver operating characteristic (fROC) curve and precision-recall (PR) curve. Precision, recall, and F1-score were used to measure the DL model.

RESULTS

A total of 2497 three-dimension retrogression annotations were labeled for training (n = 1157) and multicenter validation (n = 1340). The DL model showed excellent detection efficiency in the internal validation dataset, with F1-score achieving 0.971 and 0.903 on the sagittal and axial MR images, respectively. Good performance was also observed in the external validation dataset I (F1-score, 0.768 on sagittal MR images and 0.837 on axial MR images) and external validation dataset II (F1-score, 0.787 on sagittal MR images and 0.770 on axial MR images). Furthermore, the robustness of the DL model was demonstrated via transfer learning and generalized performance evaluation on the external cervical dataset, with the F1-score yielding 0.931 and 0.919 on the sagittal and axial MR images, respectively.

CONCLUSION

The proposed DL model can automatically detect lumbar and cervical degenerative disease on T2-weighted MR images with good performance, robustness, and feasibility in clinical practice.

Generalizability of Machine Learning Models: Quantitative Evaluation of Three Methodological Pitfalls

Purpose

To investigate the impact of the following three methodological pitfalls on model generalizability: (a) violation of the independence assumption, (b) model evaluation with an inappropriate performance indicator or baseline for comparison, and (c) batch effect.

Materials and Methods

The authors used retrospective CT, histopathologic analysis, and radiography datasets to develop machine learning models with and without the three methodological pitfalls to quantitatively illustrate their effect on model performance and generalizability. F1 score was used to measure performance, and differences in performance between models developed with and without errors were assessed using the Wilcoxon rank sum test when applicable.

Results

Violation of the independence assumption by applying oversampling, feature selection, and data augmentation before splitting data into training, validation, and test sets seemingly improved model F1 scores by 71.2% for predicting local recurrence and 5.0% for predicting 3-year overall survival in head and neck cancer and by 46.0% for distinguishing histopathologic patterns in lung cancer. Randomly distributing data points for a patient across datasets superficially improved the F1 score by 21.8%. High model performance metrics did not indicate high-quality lung segmentation. In the presence of a batch effect, a model built for pneumonia detection had an F1 score of 98.7% but correctly classified only 3.86% of samples from a new dataset of healthy patients.

Conclusion

Machine learning models developed with these methodological pitfalls, which are undetectable during internal evaluation, produce inaccurate predictions; thus, understanding and avoiding these pitfalls is necessary for developing generalizable models.Keywords: Random Forest, Diagnosis, Prognosis, Convolutional Neural Network (CNN), Medical Image Analysis, Generalizability, Machine Learning, Deep Learning, Model Evaluation Supplemental material is available for this article. Published under a CC BY 4.0 license.

GAN augmentation for multiclass image classification using hemorrhage detection as a case-study

Purpose: In recent years, the development and exploration of deeper and more complex deep learning models has been on the rise. However, the availability of large heterogeneous datasets to support efficient training of deep learning models is lacking. While linear image transformations for augmentation have been used traditionally, the recent development of generative adversarial networks (GANs) could theoretically allow us to generate an infinite amount of data from the real distribution to support deep learning model training. Recently, the Radiological Society of North America (RSNA) curated a multiclass hemorrhage detection challenge dataset that includes over 800,000 images for hemorrhage detection, but all high-performing models were trained using traditional data augmentation techniques. Given a wide variety of selections, the augmentation for image classification often follows a trial-and-error policy. Approach: We designed conditional DCGAN (cDCGAN) and in parallel trained multiple popular GAN models to use as online augmentations and compared them to traditional augmentation methods for the hemorrhage case study. Results: Our experimentations show that the super-minority, epidural hemorrhages with cDCGAN augmentation presented a minimum of 2 × improvement in their performance against the traditionally augmented model using the same classifier configuration. Conclusion: This shows that for complex and imbalanced datasets, traditional data imbalancing solutions may not be sufficient and require more complex and diverse data augmentation methods such as GANs to solve.

Deep Learning in Large and Multi-Site Structural Brain MR Imaging Datasets

Large, multi-site, heterogeneous brain imaging datasets are increasingly required for the training, validation, and testing of advanced deep learning (DL)-based automated tools, including structural magnetic resonance (MR) image-based diagnostic and treatment monitoring approaches. When assembling a number of smaller datasets to form a larger dataset, understanding the underlying variability between different acquisition and processing protocols across the aggregated dataset (termed “batch effects”) is critical. The presence of variation in the training dataset is important as it more closely reflects the true underlying data distribution and, thus, may enhance the overall generalizability of the tool. However, the impact of batch effects must be carefully evaluated in order to avoid undesirable effects that, for example, may reduce performance measures. Batch effects can result from many sources, including differences in acquisition equipment, imaging technique and parameters, as well as applied processing methodologies. Their impact, both beneficial and adversarial, must be considered when developing tools to ensure that their outputs are related to the proposed clinical or research question (i.e., actual disease-related or pathological changes) and are not simply due to the peculiarities of underlying batch effects in the aggregated dataset. We reviewed applications of DL in structural brain MR imaging that aggregated images from neuroimaging datasets, typically acquired at multiple sites. We examined datasets containing both healthy control participants and patients that were acquired using varying acquisition protocols. First, we discussed issues around Data Access and enumerated the key characteristics of some commonly used publicly available brain datasets. Then we reviewed methods for correcting batch effects by exploring the two main classes of approaches: Data Harmonization that uses data standardization, quality control protocols or other similar algorithms and procedures to explicitly understand and minimize unwanted batch effects; and Domain Adaptation that develops DL tools that implicitly handle the batch effects by using approaches to achieve reliable and robust results. In this narrative review, we highlighted the advantages and disadvantages of both classes of DL approaches, and described key challenges to be addressed in future studies.

Deep learning-accelerated T2-weighted imaging of the prostate: Impact of further acceleration with lower spatial resolution on image quality

PURPOSE

To compare image quality in prostate MRI among standard T2-weighted imaging (T2-std), accelerated T2-weighted imaging (T2WI) with high resolution (T2-HR) and more accelerated T2WI with lower resolution (T2-LR) using both conventional reconstruction (C) and deep learning reconstruction (DL).

MATERIALS AND METHODS

In 46 consecutive patients, T2-std, T2-HR and T2-LR were acquired in 3:32 min, 1:06 min and 0.52 min, respectively. Both reconstruction techniques (C and DL) were applied to T2-HR and T2-LR. Five sets of images (T2-std, T2-HR_C, T2-LR_C, T2-HR_DL, and T2-LR_DL) for each patient were independently evaluated by two radiologists. Quantitative analysis including the signal-to-noise ratio (SNR) and contrast ratio (CR) and qualitative analysis with a 5-point scale for the sharpness of structures, ghosting or other artifacts, noise and overall image quality were performed.

RESULTS

The SNR was not different in either the peripheral zone (PZ) or transition zone (TZ) between T2-LR_DL and T2-std with the median value of 21.7 versus 22.6 in PZ and 16.5 versus 17.3 in TZ, respectively. The CR between the prostate gland and muscle was significantly lower on T2-HR_C and T2-LR_C than on T2-std. Most of the evaluated factors showed significantly lower scores on T2-HR_C and T2-LR_C than on T2-std. Although noise and overall image quality on T2-HR_DL and other artifacts on T2-LR_DL were rated significantly lower than on T2-std (median value 4.0 versus 4.5, P < 0.001; 4.5 versus 5.0, P = 0.001; 4.5 versus 5.0, P = 0.006, respectively), other factors did not differ between T2-std and T2-HR_DL or T2-LR_DL.

CONCLUSION

DL is useful to improve image quality in accelerated T2WI of the prostate gland. Using DL, accelerated T2WI with lower spatial resolution than T2-std can be achieved with similar image quality in much shorter scan time (75.5% reduction in the acquisition time).