Validation of algorithmic CT image quality metrics with preferences of radiologists

Technology Characterization Through Diverse Evaluation Methodologies: Application to Thoracic Imaging in Photon-Counting Computed Tomography

OBJECTIVE

Different methods can be used to condition imaging systems for clinical use. The purpose of this study was to assess how these methods complement one another in evaluating a system for clinical integration of an emerging technology, photon-counting computed tomography (PCCT), for thoracic imaging.

METHODS

Four methods were used to assess a clinical PCCT system (NAEOTOM Alpha; Siemens Healthineers, Forchheim, Germany) across 3 reconstruction kernels (Br40f, Br48f, and Br56f). First, a phantom evaluation was performed using a computed tomography quality control phantom to characterize noise magnitude, spatial resolution, and detectability. Second, clinical images acquired using conventional and PCCT systems were used for a multi-institutional reader study where readers from 2 institutions were asked to rank their preference of images. Third, the clinical images were assessed in terms of in vivo image quality characterization of global noise index and detectability. Fourth, a virtual imaging trial was conducted using a validated simulation platform (DukeSim) that models PCCT and a virtual patient model (XCAT) with embedded lung lesions imaged under differing conditions of respiratory phase and positional displacement. Using known ground truth of the patient model, images were evaluated for quantitative biomarkers of lung intensity histograms and lesion morphology metrics.

RESULTS

For the physical phantom study, the Br56f kernel was shown to have the highest resolution despite having the highest noise and lowest detectability. Readers across both institutions preferred the Br56f kernel (71% first rank) with a high interclass correlation (0.990). In vivo assessments found superior detectability for PCCT compared with conventional computed tomography but higher noise and reduced detectability with increased kernel sharpness. For the virtual imaging trial, Br40f was shown to have the best performance for histogram measures, whereas Br56f was shown to have the most precise and accurate morphology metrics.

CONCLUSION

The 4 evaluation methods each have their strengths and limitations and bring complementary insight to the evaluation of PCCT. Although no method offers a complete answer, concordant findings between methods offer affirmatory confidence in a decision, whereas discordant ones offer insight for added perspective. Aggregating our findings, we concluded the Br56f kernel best for high-resolution tasks and Br40f for contrast-dependent tasks.

How subjective CT image quality assessment becomes surprisingly reliable: pairwise comparisons instead of Likert scale

OBJECTIVES

The aim of this study is to improve the reliability of subjective IQ assessment using a pairwise comparison (PC) method instead of a Likert scale method in abdominal CT scans.

METHODS

Abdominal CT scans (single-center) were retrospectively selected between September 2019 and February 2020 in a prior study. Sample variance in IQ was obtained by adding artificial noise using dedicated reconstruction software, including reconstructions with filtered backprojection and varying iterative reconstruction strengths. Two datasets (each n = 50) were composed with either higher or lower IQ variation with the 25 original scans being part of both datasets. Using in-house developed software, six observers (five radiologists, one resident) rated both datasets via both the PC method (forcing observers to choose preferred scans out of pairs of scans resulting in a ranking) and a 5-point Likert scale. The PC method was optimized using a sorting algorithm to minimize necessary comparisons. The inter- and intraobserver agreements were assessed for both methods with the intraclass correlation coefficient (ICC).

RESULTS

Twenty-five patients (mean age 61 years ± 15.5; 56% men) were evaluated. The ICC for interobserver agreement for the high-variation dataset increased from 0.665 (95%CI 0.396-0.814) to 0.785 (95%CI 0.676-0.867) when the PC method was used instead of a Likert scale. For the low-variation dataset, the ICC increased from 0.276 (95%CI 0.034-0.500) to 0.562 (95%CI 0.337-0.729). Intraobserver agreement increased for four out of six observers.

CONCLUSION

The PC method is more reliable for subjective IQ assessment indicated by improved inter- and intraobserver agreement.

CLINICAL RELEVANCE STATEMENT

This study shows that the pairwise comparison method is a more reliable method for subjective image quality assessment. Improved reliability is of key importance for optimization studies, validation of automatic image quality assessment algorithms, and training of AI algorithms.

KEY POINTS

• Subjective assessment of diagnostic image quality via Likert scale has limited reliability. • A pairwise comparison method improves the inter- and intraobserver agreement. • The pairwise comparison method is more reliable for CT optimization studies.

Automatic chest computed tomography image noise quantification using deep learning

PURPOSE

This study aimed to develop a deep learning (DL) method for noise quantification for clinical chest computed tomography (CT) images without the need for repeated scanning or homogeneous tissue regions.

METHODS

A comprehensive phantom CT dataset (three dose levels, six reconstruction methods, amounting to 9240 slices) was acquired and used to train a convolutional neural network (CNN) to output an estimate of local image noise standard deviations (SD) from a single CT scan input. The CNN model consisting of seven convolutional layers was trained on the phantom image dataset representing a range of scan parameters and was tested with phantom images acquired in a variety of different scan conditions, as well as publicly available chest CT images to produce clinical noise SD maps.

RESULTS

Noise SD maps predicted by the CNN agreed well with the ground truth both visually and numerically in the phantom dataset (errors of < 5 HU for most scan parameter combinations). In addition, the noise SD estimates obtained from clinical chest CT images were similar to running-average based reference estimates in areas without prominent tissue interfaces.

CONCLUSIONS

Predicting local noise magnitudes without the need for repeated scans is feasible using DL. Our implementation trained with phantom data was successfully applied to open-source clinical data with heterogeneous tissue borders and textures. We suggest that automatic DL noise mapping from clinical patient images could be used as a tool for objective CT image quality estimation and protocol optimization.

Evaluation of Image Quality and Detectability of Deep Learning Image Reconstruction (DLIR) Algorithm in Single- and Dual-energy CT

This study is aimed to evaluate effects of deep learning image reconstruction (DLIR) on image quality in single-energy CT (SECT) and dual-energy CT (DECT), in reference to adaptive statistical iterative reconstruction-V (ASIR-V). The Gammex 464 phantom was scanned in SECT and DECT modes at three dose levels (5, 10, and 20 mGy). Raw data were reconstructed using six algorithms: filtered back-projection (FBP), ASIR-V at 40% (AV-40) and 100% (AV-100) strength, and DLIR at low (DLIR-L), medium (DLIR-M), and high strength (DLIR-H), to generate SECT 120kVp images and DECT 120kVp-like images. Objective image quality metrics were computed, including noise power spectrum (NPS), task transfer function (TTF), and detectability index (d'). Subjective image quality evaluation, including image noise, texture, sharpness, overall quality, and low- and high-contrast detectability, was performed by six readers. DLIR-H reduced overall noise magnitudes from FBP by 55.2% in a more balanced way of low and high frequency ranges comparing to AV-40, and improved the TTF values at 50% for acrylic inserts by average percentages of 18.32%. Comparing to SECT 20 mGy AV-40 images, the DECT 10 mGy DLIR-H images showed 20.90% and 7.75% improvement in d' for the small-object high-contrast and large-object low-contrast tasks, respectively. Subjective evaluation showed higher image quality and better detectability. At 50% of the radiation dose level, DECT with DLIR-H yields a gain in objective detectability index compared to full-dose AV-40 SECT images used in daily practice.

A New Algorithm for Automatically Calculating Noise, Spatial Resolution, and Contrast Image Quality Metrics: Proof-of-Concept and Agreement With Subjective Scores in Phantom and Clinical Abdominal CT

OBJECTIVES

The aims of this study were to develop a proof-of-concept computer algorithm to automatically determine noise, spatial resolution, and contrast-related image quality (IQ) metrics in abdominal portal venous phase computed tomography (CT) imaging and to assess agreement between resulting objective IQ metrics and subjective radiologist IQ ratings.

MATERIALS AND METHODS

An algorithm was developed to calculate noise, spatial resolution, and contrast IQ parameters. The algorithm was subsequently used on 2 datasets of anthropomorphic phantom CT scans, acquired on 2 different scanners (n = 57 each), and on 1 dataset of patient abdominal CT scans (n = 510). These datasets include a range of high to low IQ: in the phantom dataset, this was achieved through varying scanner settings (tube voltage, tube current, reconstruction algorithm); in the patient dataset, lower IQ images were obtained by reconstructing 30 consecutive portal venous phase scans as if they had been acquired at lower mAs. Five noise, 1 spatial, and 13 contrast parameters were computed for the phantom datasets; for the patient dataset, 5 noise, 1 spatial, and 18 contrast parameters were computed. Subjective IQ rating was done using a 5-point Likert scale: 2 radiologists rated a single phantom dataset each, and another 2 radiologists rated the patient dataset in consensus. General agreement between IQ metrics and subjective IQ scores was assessed using Pearson correlation analysis. Likert scores were grouped into 2 categories, "insufficient" (scores 1-2) and "sufficient" (scores 3-5), and differences in computed IQ metrics between these categories were assessed using the Mann-Whitney U test.

RESULTS

The algorithm was able to automatically calculate all IQ metrics for 100% of the included scans. Significant correlations with subjective radiologist ratings were found for 4 of 5 noise (R2 range = 0.55-0.70), 1 of 1 spatial resolution (R2 = 0.21 and 0.26), and 10 of 13 contrast (R2 range = 0.11-0.73) parameters in the phantom datasets and for 4 of 5 noise (R2 range = 0.019-0.096), 1 of 1 spatial resolution (R2 = 0.11), and 16 of 18 contrast (R2 range = 0.008-0.116) parameters in the patient dataset. Computed metrics that significantly differed between "insufficient" and "sufficient" categories were 4 of 5 noise, 1 of 1 spatial resolution, 9 and 10 of 13 contrast parameters for phantom the datasets and 3 of 5 noise, 1 of 1 spatial resolution, and 10 of 18 contrast parameters for the patient dataset.

CONCLUSION

The developed algorithm was able to successfully calculate objective noise, spatial resolution, and contrast IQ metrics of both phantom and clinical abdominal CT scans. Furthermore, multiple calculated IQ metrics of all 3 categories were in agreement with subjective radiologist IQ ratings and significantly differed between "insufficient" and "sufficient" IQ scans. These results demonstrate the feasibility and potential of algorithm-determined objective IQ. Such an algorithm should be applicable to any scan and may help in optimization and quality control through automatic IQ assessment in daily clinical practice.

Oncology-specific radiation dose and image noise reference levels in adult abdominal-pelvic CT

OBJECTIVES

To provide our oncology-specific adult abdominal-pelvic CT reference levels for image noise and radiation dose from a high-volume, oncologic, tertiary referral center.

METHODS

The portal venous phase abdomen-pelvis acquisition was assessed for image noise and radiation dose in 13,320 contrast-enhanced CT examinations. Patient size (effective diameter) and radiation dose (CTDIvol) were recorded using a commercial software system, and image noise (Global Noise metric) was quantified using a custom processing system. The reference level and range for dose and noise were calculated for the full dataset, and for examinations grouped by CT scanner model. Dose and noise reference levels were also calculated for exams grouped by five different patient size categories.

RESULTS

The noise reference level was 11.25 HU with a reference range of 10.25-12.25 HU. The dose reference level at a median effective diameter of 30.7 cm was 26.7 mGy with a reference range of 19.6-37.0 mGy. Dose increased with patient size; however, image noise remained approximately constant within the noise reference range. The doses were 2.1-2.5 times than the doses in the ACR DIR registry for corresponding patient sizes. The image noise was 0.63-0.75 times the previously published reference level in abdominal-pelvic CT examinations.

CONCLUSIONS

Our oncology-specific abdominal-pelvic CT dose reference levels are higher than in the ACR dose index registry and our oncology-specific image noise reference levels are lower than previously proposed image noise reference levels.

ADVANCES IN KNOWLEDGE

This study reports reference image noise and radiation dose levels appropriate for the indication of abdomen-pelvis CT examination for cancer diagnosis and staging. The difference in these reference levels from non-oncology-specific CT examinations highlight a need for indication-specific, dose index and image quality reference registries.

Task-based validation and application of a scanner-specific CT simulator using an anthropomorphic phantom

BACKGROUND

Quantitative analysis of computed tomography (CT) images traditionally utilizes real patient data that can pose challenges with replicability, efficiency, and radiation exposure. Instead, virtual imaging trials (VITs) can overcome these hurdles through computer simulations of models of patients and imaging systems. DukeSim is a scanner-specific CT imaging simulator that has previously been validated with simple cylindrical phantoms, but not with anthropomorphic conditions and clinically relevant measurements.

PURPOSE

To validate a scanner-specific CT simulator (DukeSim) for the assessment of lung imaging biomarkers under clinically relevant conditions across multiple scanners using an anthropomorphic chest phantom, and to demonstrate the utility of virtual trials by studying the effects or radiation dose and reconstruction kernels on the lung imaging quantifications.

METHODS

An anthropomorphic chest phantom with customized tube inserts was imaged with two commercial scanners (Siemens Force and Siemens Flash) at 28 dose and reconstruction conditions. A computational version of the chest phantom was used with a scanner-specific CT simulator (DukeSim) to simulate virtual images corresponding to the settings of the real acquisitions. Lung imaging biomarkers were computed from both real and simulated CT images and quantitatively compared across all imaging conditions. The VIT framework was further utilized to investigate the effects of radiation dose (20-300 mAs) and reconstruction settings (Qr32f, Qr40f, and Qr69f reconstruction kernels using ADMIRE strength 3) on the accuracy of lung imaging biomarkers, compared against the ground-truth values modeled in the computational chest phantom.

RESULTS

The simulated CT images matched closely the real images for both scanners and all imaging conditions qualitatively and quantitatively, with the average biomarker percent error of 3.51% (range 0.002%-18.91%). The VIT study showed that sharper reconstruction kernels had lower accuracy with errors in mean lung HU of 84-94 HU, lung volume of 797-3785 cm3 , and lung mass of -800 to 1751 g. Lower tube currents had the lower accuracy with errors in mean lung HU of 6-84 HU, lung volume of 66-3785 cm3 , and lung mass of 170-1751 g. Other imaging biomarkers were consistent under the studied reconstruction settings and tube currents.

CONCLUSION

We comprehensively evaluated the realism of DukeSim in an anthropomorphic setup across a diverse range of imaging conditions. This study paves the way toward utilizing VITs more reliably for conducting medical imaging experiments that are not practical using actual patient images.

A patient-informed approach to predict iodinated-contrast media enhancement in the liver

OBJECTIVE

To devise a patient-informed time series model that predicts liver contrast enhancement, by integrating clinical data and pharmacokinetics models, and to assess its feasibility to improve enhancement consistency in contrast-enhanced liver CT scans.

METHODS

The study included 1577 Chest/Abdomen/Pelvis CT scans, with 70-30% training/validation-testing split. A Gaussian function was used to approximate the early arterial, late arterial, and the portal venous phases of the contrast perfusion curve of each patient using their respective bolus tracking and diagnostic scan data. Machine learning models were built to predict the Gaussian parameters of each patient using the patient attributes (weight, height, age, sex, BMI). Pearson's coefficient, mean absolute error, and root mean squared error were used to assess the prediction accuracy.

RESULTS

The integration of the pharmacokinetics model with a two-layered neural network achieved the highest prediction accuracy on the test data (R2 = 0.61), significantly exceeding the performance of the pharmacokinetics model alone (R2 = 0.11). Applying the model demonstrated that adjusting the contrast administration directed by the model may reduce clinical enhancement inconsistency by up to 40 %.

CONCLUSIONS

A new model using a Gaussian function and supervised machine learning can be used to build liver parenchyma contrast enhancement prediction model. The model can have utility in clinical settings to optimize and improve consistency in contrast-enhanced liver imaging.

Automatic head computed tomography image noise quantification with deep learning

PURPOSE

Computed tomography (CT) image noise is usually determined by standard deviation (SD) of pixel values from uniform image regions. This study investigates how deep learning (DL) could be applied in head CT image noise estimation.

METHODS

Two approaches were investigated for noise image estimation of a single acquisition image: direct noise image estimation using supervised DnCNN convolutional neural network (CNN) architecture, and subtraction of a denoised image estimated with denoising UNet-CNN experimented with supervised and unsupervised noise2noise training approaches. Noise was assessed with local SD maps using 3D- and 2D-CNN architectures. Anthropomorphic phantom CT image dataset (N = 9 scans, 3 repetitions) was used for DL-model comparisons. Mean square error (MSE) and mean absolute percentage errors (MAPE) of SD values were determined using the SD values of subtraction images as ground truth. Open-source clinical head CT low-dose dataset (N_train = 37, N_test = 10 subjects) were used to demonstrate DL applicability in noise estimation from manually labeled uniform regions and in automated noise and contrast assessment.

RESULTS

The direct SD estimation using 3D-CNN was the most accurate assessment method when comparing in phantom dataset (MAPE = 15.5%, MSE = 6.3HU). Unsupervised noise2noise approach provided only slightly inferior results (MAPE = 20.2%, MSE = 13.7HU). 2DCNN and unsupervised UNet models provided the smallest MSE on clinical labeled uniform regions.

CONCLUSIONS

DL-based clinical image assessment is feasible and provides acceptable accuracy as compared to true image noise. Noise2noise approach may be feasible in clinical use where no ground truth data is available. Noise estimation combined with tissue segmentation may enable more comprehensive image quality characterization.

Comparison of two versions of a deep learning image reconstruction algorithm on CT image quality and dose reduction: A phantom study

PURPOSE

To compare the impact on CT image quality and dose reduction of two versions of a Deep Learning Image Reconstruction algorithm.

MATERIAL AND METHODS

Acquisitions on the CT ACR 464 phantom were performed at five dose levels (CTDI_vol : 10/7.5/5/2.5/1 mGy) using chest or abdomen pelvis protocol parameters. Raw data were reconstructed using the filtered-back projection (FBP), the enhanced level of AIDR 3D (AIDR 3De), and the three levels of AiCE (Mild, Standard, and Strong) for the two versions (AiCE V8 vs AiCE V10). The noise power spectrum (NPS) and task-based transfer function (TTF) for bone (high-contrast insert) and acrylic (low-contrast insert) inserts were computed. To quantify the changes of noise magnitude and texture, the square root of the area under the NPS curve and the average spatial frequency (f_av ) of the NPS curve were measured. The detectability index (d') was computed to model the detectability of either a large mass in the liver or lung, or a small calcification or high contrast tissue boundaries.

RESULTS

The noise magnitude was lower with both AiCE versions than with AIDR 3De. The noise magnitude was lower with AiCE V10 than with AiCE V8 (-4 ± 6% for Mild, -13 ± 3% for Standard, and -48 ± 0% for Strong levels). f_av and TTF_50% values for both inserts shifted towards higher frequencies with AiCE than with AIDR 3De. Compared to AiCE V08, f_av shifted towards higher frequencies with AiCE V10 (45 ± 4%, 36 ± 3%, and 5 ± 4% for all levels, respectively). The TTF_50% values shifted towards higher frequencies with AiCE V10 as compared with AiCE V8 for both inserts, except for the Strong level for the acrylic insert. Whatever the dose and AiCE levels, d' values were higher with AiCE V10 than with AiCE V8 for the small object/calcification and for the large object/lesion.

CONCLUSION

As compared to AIDR 3De, lower noise magnitude and higher spatial resolution and detectability index were found with both versions of AiCE. As compared to AiCE V8, AiCE V10 reduced noise and improved spatial resolution and detectability without changing the noise texture in a simple geometric phantom, except for the Strong level. AiCE V10 seems to have a greater potential for dose reduction than AiCE V8.

Radiation dose reduction considerations and imaging patterns of ground glass opacities in coronavirus: risk of over exposure in computed tomography

This article aims to summarize the available data on the severe acute respiratory syndrome coronavirus 2 (SAR-CoV-2) imaging patterns as well as reducing radiation dose exposure in chest computed tomography (CT) protocols. First, the general aspects of radiation dose in CT and radiation risk are discussed, followed by the effect of changing parameters on image quality. This article attempts to highlight some of the common chest CT signs that radiologists and emergency physicians are likely to encounter. With the increasing trend of using chest CT scans as an imaging tool to diagnose and monitor SAR-CoV-2, we emphasize that pattern recognition is the key, and this pictorial essay should serve as a guide to help establish correct diagnosis coupled with correct scanner parameters to reduce radiation dose without affecting imaging quality in this tragic pandemic the world is facing.

Evaluation of a Motion Correction Algorithm for C-Arm Computed Tomography Acquired During Transarterial Chemoembolization

Purpose

The aim of this retrospective study was to evaluate the feasibility of a motion correction 3D reconstruction prototype technique for C-arm computed tomography (CACT).

Material and Methods

We included 65 consecutive CACTs acquired during transarterial chemoembolization of 54 patients (47 m,7f; 67 ± 11.3 years). All original raw datasets (CACT_Org) underwent reconstruction with and without volume punching of high-contrast objects using a 3D image reconstruction software to compensate for motion (CACT_{MC_bone};CACT_{MC_no bone}). Subsequently, the effect on image quality (IQ) was evaluated using objective (image sharpness metric) and subjective criteria. Subjective criteria were defined by vessel geometry, overall IQ, delineation of tumor feeders, the presence of foreign material-induced artifacts and need for additional imaging, assessed by two independent readers on a 3-(vessel geometry and overall IQ) or 2-point scale, respectively. Friedman rank-sum test and post hoc analysis in form of pairwise Wilcoxon signed-rank test were computed and inter-observer agreement analyzed using kappa test.

Results

Objective IQ as defined by an image sharpness metric, increased from 273.5 ± 28 (CACT_Org) to 328.5 ± 55.1 (CACT_{MC_bone}) and 331 ± 57.8 (CACT_{MC_no bone}; all p < 0.0001). These results could largely be confirmed by the subjective analysis, which demonstrated predominantly good and moderate inter-observer agreement, with best agreement for CACT_{MC_no bone} in all categories (e.g., vessel geometry: CACT_Org: κ = 0.51, CACT_{MC_bone}: κ = 0.42, CACT_{MC_no bone}: κ = 0.69).

Conclusion

The application of a motion correction algorithm was feasible for all data sets and led to an increase in both objective and subjective IQ parameters.

Level of Evidence

3

Modeling Patient-Informed Liver Contrast Perfusion in Contrast-enhanced Computed Tomography

OBJECTIVE

To determine the correlation between patient attributes and contrast enhancement in liver parenchyma and demonstrate the potential for patient-informed prediction and optimization of contrast enhancement in liver imaging.

METHODS

The study included 418 chest/abdomen/pelvis computed tomography scans, with 75% to 25% training-testing split. Two regression models were built to predict liver parenchyma contrast enhancement over time: first model (model A) utilized patient attributes (height, weight, sex, age, bolus volume, injection rate, scan times, body mass index, lean body mass) and bolus-tracking data. A second model (model B) only used the patient attributes. Pearson coefficient was used to assess predictive accuracy.

RESULTS

Weight- and height-related features were found to be statistically significant predictors (P < 0.05), weight being the strongest. Of the 2 models, model A (r = 0.75) showed greater accuracy than model B (r = 0.42).

CONCLUSIONS

Patient attributes can be used to build prediction model for liver parenchyma contrast enhancement. The model can have utility in optimization and improved consistency in contrast-enhanced liver imaging.

Achieving CT Regulatory Compliance: A Comprehensive and Continuous Quality Improvement Approach

Computed tomography (CT) represents one of the largest sources of radiation exposure to the public in the United States. Regulatory requirements now mandate dose tracking for all exams and investigation of dose events that exceed set dose thresholds. Radiology practices are tasked with ensuring quality control and optimizing patient CT exam doses while maintaining diagnostic efficacy. Meeting regulatory requirements necessitates the development of an effective quality program in CT. This review provides a template for accreditation compliant quality control and CT dose optimization. The following paper summarizes a large health system approach for establishing a quality program in CT and discusses successes, challenges, and future needs.