Reproducibility of Left Ventricular Function Derived From Cardiac Magnetic Resonance and Gated 13N-Ammonia Positron Emission Tomography Myocardial Perfusion Imaging: A Head-to-Head Comparison Using Hybrid Positron Emission Tomography/Magnetic Resonance

RATIONALE AND OBJECTIVES

Cardiac magnetic resonance (CMR) and gated 13N-ammonia positron emission tomography myocardial perfusion imaging (PET-MPI) offer accurate and highly comparable global left ventricular ejection fraction (LVEF) measurements. In addition to accuracy, however, reproducibility is crucial to avoid variations in LVEF assessment potentially negatively impacting treatment decisions. We performed a head-to-head comparison of the reproducibility of LVEF measurements derived from simultaneously acquired CMR and PET-MPI using different state-of-the-art commercially available software.

MATERIALS AND METHODS

93 patients undergoing hybrid PET/MR were retrospectively included. LVEF was derived from CMR and PET-MPI at two separate core labs, using two state-of-the-art software packages for CMR (cvi42 and Medis Suite MR) and PET (QPET and CardIQ Physio). Intra- and inter-reader agreement was assessed using correlation and Bland-Altman (BA) analyses.

RESULTS

While intra- and inter-reader reproducibility of LVEF was high among both modalities and all software packages (r ≥ 0.87 and ICC≥0.91, all significant at p < 0.0001), LVEF derived from PET-MPI and analyzed with QPET outperformed all other analyses (intra-reader reproducibility: r = 0.99, ICC=0.99; inter-reader reproducibility: r = 0.98, ICC=1.00; Pearson correlations significantly higher than all others at p ≤ 0.0001). BA analyses showed smaller biases for LVEF derived from PET-MPI (-0.1% and +0.9% for intra-reader, -0.4% and -0.8% for inter-reader agreement) than those derived from CMR (+0.7% and +2.8% for intra-reader, -0.9% and -2.2% for inter-reader agreement) with similar results for BA limits of agreement.

CONCLUSION

Gated 13N-ammonia PET-MPI provides equivalent reproducibility of LVEF compared to CMR. It may offer a valid alternative to CMR for patients requiring LV functional assessment.

Reference standard for the evaluation of automatic segmentation algorithms: Quantification of inter observer variability of manual delineation of prostate contour on MRI

PURPOSE

The purpose of this study was to investigate the relationship between inter-reader variability in manual prostate contour segmentation on magnetic resonance imaging (MRI) examinations and determine the optimal number of readers required to establish a reliable reference standard.

MATERIALS AND METHODS

Seven radiologists with various experiences independently performed manual segmentation of the prostate contour (whole-gland [WG] and transition zone [TZ]) on 40 prostate MRI examinations obtained in 40 patients. Inter-reader variability in prostate contour delineations was estimated using standard metrics (Dice similarity coefficient [DSC], Hausdorff distance and volume-based metrics). The impact of the number of readers (from two to seven) on segmentation variability was assessed using pairwise metrics (consistency) and metrics with respect to a reference segmentation (conformity), obtained either with majority voting or simultaneous truth and performance level estimation (STAPLE) algorithm.

RESULTS

The average segmentation DSC for two readers in pairwise comparison was 0.919 for WG and 0.876 for TZ. Variability decreased with the number of readers: the interquartile ranges of the DSC were 0.076 (WG) / 0.021 (TZ) for configurations with two readers, 0.005 (WG) / 0.012 (TZ) for configurations with three readers, and 0.002 (WG) / 0.0037 (TZ) for configurations with six readers. The interquartile range decreased slightly faster between two and three readers than between three and six readers. When using consensus methods, variability often reached its minimum with three readers (with STAPLE, DSC = 0.96 [range: 0.945-0.971] for WG and DSC = 0.94 [range: 0.912-0.957] for TZ, and interquartile range was minimal for configurations with three readers.

CONCLUSION

The number of readers affects the inter-reader variability, in terms of inter-reader consistency and conformity to a reference. Variability is minimal for three readers, or three readers represent a tipping point in the variability evolution, with both pairwise-based metrics or metrics with respect to a reference. Accordingly, three readers may represent an optimal number to determine references for artificial intelligence applications.

Robustness and performance of radiomic features in diagnosing cystic renal masses

PURPOSE

We study the inter-reader variability in manual delineation of cystic renal masses (CRMs) presented in computerized tomography (CT) images and its effect on the classification performance of a machine learning algorithm in distinguishing benign from potentially malignant CRMs. In addition, we assessed whether the inclusion of higher-order robust radiomic features improves the classification performance over the use of first-order features.

METHODS

230 CRMs were independently delineated by two radiologists. Through a combination of random fluctuations, dilation, and erosion operations over the original region of interests (ROIs), we generated four additional sets of synthetic ROIs to capture the inter-reader variability realistically, as confirmed by dice coefficient measurements and visual assessment. We then identified the robust features based on the intra-class coefficient (ICC > 0.85) across these datasets. We applied a tenfold stratified cross-validation (CV) to train and test the performance of the random forest model for the classification of CRMs into benign and potentially malignant.

RESULTS

The mean area under the curve (AUC), sensitivity, specificity, positive predictive value, and negative predictive value were 0.87, 0.82, 0.90, 0.85, and 0.93, respectively. With the usage of first-order features alone, the corresponding values were nearly identical.

CONCLUSION

AUC ranged for the robust and uncorrelated features from 0.83 ± 0.09 to 0.93 ± 0.04 and for the first-order features from 0.84 ± 0.09 to 0.91 ± 0.04. Our study indicates that the first-order features alone are sufficient for the classification of CRMs, and that inclusion of higher-order features does not necessarily improve performance.

A self-directed learning intervention for radiographers rating mammographic breast density

PURPOSE

Subjective methods of mammographic breast density (MBD) assessment are prone to inter-reader variability. This work aims to assess the impact of a short self-directed, experiential learning intervention on radiographers' reproducibility of MBD assessment.

METHOD

The study used two sets of images (test and learning intervention) containing left craniocaudal and left mediolateral oblique views. The test set had MBD ratings from Volpara™ and radiologists using the fourth edition Breast Imaging and Data Systems (BI-RADS^®). Seven radiographers rated the MBD of the test set before and after a self-directed learning intervention using the percentage descriptors in the fourth edition BI-RADS^® Atlas. The inter-reader agreement, the agreement between radiographers and Volpara™ as well as radiologists, was assessed using a Weighted Kappa (к_w).

RESULTS

Overall, radiographers' inter-reader agreement (к_w) was substantial (0.79; 95% CI: 0.70-0.87) before the intervention and almost perfect (0.84; 95% CI: 0.77-0.90) after the intervention. Before the intervention, radiographers demonstrated fair agreement with radiologists (0.24; 95% CI: -0.46-0.61) and Volpara™ (0.24; 95% CI: -0.41-0.59). A fair but slightly improved agreement was also observed between radiographers and radiologists (0.31; 95% CI: -0.33-0.64) as well as Volpara™ (0.28; 95% CI: -0.34-0.61) after the intervention.

CONCLUSION

Findings demonstrate that a short duration self-directed, experiential learning intervention reduces inter-reader differences in MBD classification, but has a negligible impact on improving the agreement between inexperienced and expert readers.