A comparison of resampling schemes for estimating model observer performance with small ensembles

Need for objective task-based evaluation of deep learning-based denoising methods: A study in the context of myocardial perfusion SPECT

BACKGROUND

Artificial intelligence-based methods have generated substantial interest in nuclear medicine. An area of significant interest has been the use of deep-learning (DL)-based approaches for denoising images acquired with lower doses, shorter acquisition times, or both. Objective evaluation of these approaches is essential for clinical application.

PURPOSE

DL-based approaches for denoising nuclear-medicine images have typically been evaluated using fidelity-based figures of merit (FoMs) such as root mean squared error (RMSE) and structural similarity index measure (SSIM). However, these images are acquired for clinical tasks and thus should be evaluated based on their performance in these tasks. Our objectives were to: (1) investigate whether evaluation with these FoMs is consistent with objective clinical-task-based evaluation; (2) provide a theoretical analysis for determining the impact of denoising on signal-detection tasks; and (3) demonstrate the utility of virtual imaging trials (VITs) to evaluate DL-based methods.

METHODS

A VIT to evaluate a DL-based method for denoising myocardial perfusion SPECT (MPS) images was conducted. To conduct this evaluation study, we followed the recently published best practices for the evaluation of AI algorithms for nuclear medicine (the RELAINCE guidelines). An anthropomorphic patient population modeling clinically relevant variability was simulated. Projection data for this patient population at normal and low-dose count levels (20%, 15%, 10%, 5%) were generated using well-validated Monte Carlo-based simulations. The images were reconstructed using a 3-D ordered-subsets expectation maximization-based approach. Next, the low-dose images were denoised using a commonly used convolutional neural network-based approach. The impact of DL-based denoising was evaluated using both fidelity-based FoMs and area under the receiver operating characteristic curve (AUC), which quantified performance on the clinical task of detecting perfusion defects in MPS images as obtained using a model observer with anthropomorphic channels. We then provide a mathematical treatment to probe the impact of post-processing operations on signal-detection tasks and use this treatment to analyze the findings of this study.

RESULTS

Based on fidelity-based FoMs, denoising using the considered DL-based method led to significantly superior performance. However, based on ROC analysis, denoising did not improve, and in fact, often degraded detection-task performance. This discordance between fidelity-based FoMs and task-based evaluation was observed at all the low-dose levels and for different cardiac-defect types. Our theoretical analysis revealed that the major reason for this degraded performance was that the denoising method reduced the difference in the means of the reconstructed images and of the channel operator-extracted feature vectors between the defect-absent and defect-present cases.

CONCLUSIONS

The results show the discrepancy between the evaluation of DL-based methods with fidelity-based metrics versus the evaluation on clinical tasks. This motivates the need for objective task-based evaluation of DL-based denoising approaches. Further, this study shows how VITs provide a mechanism to conduct such evaluations computationally, in a time and resource-efficient setting, and avoid risks such as radiation dose to the patient. Finally, our theoretical treatment reveals insights into the reasons for the limited performance of the denoising approach and may be used to probe the effect of other post-processing operations on signal-detection tasks.

Optimizing constrained reconstruction in magnetic resonance imaging for signal detection

Constrained reconstruction in magnetic resonance imaging (MRI) allows the use of prior information through constraints to improve reconstructed images. These constraints often take the form of regularization terms in the objective function used for reconstruction. Constrained reconstruction leads to images which appear to have fewer artifacts than reconstructions without constraints but because the methods are typically nonlinear, the reconstructed images have artifacts whose structure is hard to predict. In this work, we compared different methods of optimizing the regularization parameter using a total variation (TV) constraint in the spatial domain and sparsity in the wavelet domain for one-dimensional (2.56×) undersampling using variable density undersampling. We compared the mean squared error (MSE), structural similarity (SSIM), L-curve and the area under the receiver operating characteristic (AUC) using a linear discriminant for detecting a small and a large signal. We used a signal-known-exactly task with varying backgrounds in a simulation where the anatomical variation was the major source of clutter for the detection task. Our results show that the AUC dependence on regularization parameters varies with the imaging task (i.e. the signal being detected). The choice of regularization parameters for MSE, SSIM, L-curve and AUC were similar. We also found that a model-based reconstruction including TV and wavelet sparsity did slightly better in terms of AUC than just enforcing data consistency but using these constraints resulted in much better MSE and SSIM. These results suggest that the increased performance in MSE and SSIM over-estimate the improvement in detection performance for the tasks in this paper. The MSE and SSIM metrics show a big difference in performance where the difference in AUC is small. To our knowledge, this is the first time that signal detection with varying backgrounds has been used to optimize constrained reconstruction in MRI.

A projection image database to investigate factors affecting image quality in weight-based dosing: application to pediatric renal SPECT

Balancing the tradeoff between radiation dose, acquisition duration and diagnostic image quality is essential for medical imaging modalities involving ionizing radiation. Lower administered activities to the patient can reduce absorbed dose, but can result in reduced diagnostic image quality or require longer acquisition durations. In pediatric nuclear medicine, it is desirable to use the lowest amount of administered radiopharmaceutical activity and the shortest acquisition duration that gives sufficient image quality for clinical diagnosis. However, diagnostic image quality is a complex function of patient factors including body morphometry. In this study, we present a digital population of 90 computational anatomic phantoms that model realistic variations in body morphometry and internal anatomy. These phantoms were used to generate a large database of projection images modeling pediatric SPECT imaging using a ^99mTc-DMSA tracer. We used an analytic projection code that models attenuation, spatially varying collimator-detector response, and object-dependent scatter to generate the projections. The projections for each organ were generated separately and can be subsequently scaled by parameters extracted from a pharmacokinetics model to simulate realistic tracer biodistribution, including variations in uptake, inside each relevant organ or tissue structure for a given tracer. Noise-free projection images can be obtained by summing these individual organ projections and scaling by the system sensitivity and acquisition duration. We applied this database in the context of ^99mTc-DMSA renal SPECT, the most common nuclear medicine imaging procedure in pediatric patients. Organ uptake fractions based on literature values and patient studies were used. Patient SPECT images were used to verify that the sum of counts in the simulated projection images was clinically realistic. For each phantom, 384 uptake realizations, modeling random variations in the uptakes of organs of interest, were generated, producing 34 560 noise-free projection datasets (384 uptake realizations times 90 phantoms). Noisy images modeling various count levels (corresponding to different products of acquisition duration and administered activity) were generated by appropriately scaling these images and simulating Poisson noise. Acquisition duration was fixed; six count levels were simulated corresponding to projection images acquired using 25%, 50%, 75%, 100%, 125%, and 150% of the original weight-based administrated activity as computed using the North American Guidelines (Gelfand et al 2011 J. Nucl. Med. 52 318-22). Combined, a total number of 207 360 noisy projection images were generated, creating a realistic projection database for use in renal pediatric SPECT imaging research. The phantoms and projection datasets were used to calculate three surrogate indices for factors affecting image quality: renal count density, average radius of rotation, and scatter-to-primary ratio. Differences in these indices were seen across the phantoms for dosing based on current guidelines, and especially for the phantom modeling the newborn. We also performed an image quality study using an anthropomorphic model observer that demonstrates that the weight-based dose scaling does not equalize image quality as measured by the area under the receiver-operating characteristics curve. These studies suggest that a dosing procedure beyond weight-based scaling of administered activities is required to equalize image quality in pediatric renal SPECT.