Nonsupervised ranking of different segmentation approaches: application to the estimation of the left ventricular ejection fraction from cardiac cine MRI sequences

How accurately can quantitative imaging methods be ranked without ground truth: An upper bound on no-gold-standard evaluation

Objective evaluation of quantitative imaging (QI) methods with patient data, while important, is typically hindered by the lack of gold standards. To address this challenge, no-gold-standard evaluation (NGSE) techniques have been proposed. These techniques have demonstrated efficacy in accurately ranking QI methods without access to gold standards. The development of NGSE methods has raised an important question: how accurately can QI methods be ranked without ground truth. To answer this question, we propose a Cramér-Rao bound (CRB)-based framework that quantifies the upper bound in ranking QI methods without any ground truth. We present the application of this framework in guiding the use of a well-known NGSE technique, namely the regression-without-truth (RWT) technique. Our results show the utility of this framework in quantifying the performance of this NGSE technique for different patient numbers. These results provide motivation towards studying other applications of this upper bound.

No-gold-standard evaluation of quantitative imaging methods in the presence of correlated noise

Objective evaluation of quantitative imaging (QI) methods with patient data is highly desirable, but is hindered by the lack or unreliability of an available gold standard. To address this issue, techniques that can evaluate QI methods without access to a gold standard are being actively developed. These techniques assume that the true and measured values are linearly related by a slope, bias, and Gaussian-distributed noise term, where the noise between measurements made by different methods is independent of each other. However, this noise arises in the process of measuring the same quantitative value, and thus can be correlated. To address this limitation, we propose a no-gold-standard evaluation (NGSE) technique that models this correlated noise by a multi-variate Gaussian distribution parameterized by a covariance matrix. We derive a maximum-likelihood-based approach to estimate the parameters that describe the relationship between the true and measured values, without any knowledge of the true values. We then use the estimated slopes and diagonal elements of the covariance matrix to compute the noise-to-slope ratio (NSR) to rank the QI methods on the basis of precision. The proposed NGSE technique was evaluated with multiple numerical experiments. Our results showed that the technique reliably estimated the NSR values and yielded accurate rankings of the considered methods for 83% of 160 trials. In particular, the technique correctly identified the most precise method for ∼ 97% of the trials. Overall, this study demonstrates the efficacy of the NGSE technique to accurately rank different QI methods when correlated noise is present, and without access to any knowledge of the ground truth. The results motivate further validation of this technique with realistic simulation studies and patient data.

Objective Task-Based Evaluation of Artificial Intelligence-Based Medical Imaging Methods:: Framework, Strategies, and Role of the Physician

Artificial intelligence-based methods are showing promise in medical imaging applications. There is substantial interest in clinical translation of these methods, requiring that they be evaluated rigorously. We lay out a framework for objective task-based evaluation of artificial intelligence methods. We provide a list of available tools to conduct this evaluation. We outline the important role of physicians in conducting these evaluation studies. The examples in this article are proposed in the context of PET scans with a focus on evaluating neural network-based methods. However, the framework is also applicable to evaluate other medical imaging modalities and other types of artificial intelligence methods.

Toward High-Throughput Artificial Intelligence-Based Segmentation in Oncological PET Imaging

Artificial intelligence (AI) techniques for image-based segmentation have garnered much attention in recent years. Convolutional neural networks have shown impressive results and potential toward fully automated segmentation in medical imaging, and particularly PET imaging. To cope with the limited access to annotated data needed in supervised AI methods, given tedious and prone-to-error manual delineations, semi-supervised and unsupervised AI techniques have also been explored for segmentation of tumors or normal organs in single- and bimodality scans. This work reviews existing AI techniques for segmentation tasks and the evaluation criteria for translational AI-based segmentation efforts toward routine adoption in clinical workflows.

Comparative analysis of active contour and convolutional neural network in rapid left-ventricle volume quantification using echocardiographic imaging

In cardiology, ultrasound is often used to diagnose heart disease associated with myocardial infarction. This study aims to develop robust segmentation techniques for segmenting the left ventricle (LV) in ultrasound images to check myocardium movement during heartbeat. The proposed technique utilizes machine learning (ML) techniques such as the active contour (AC) and convolutional neural networks (CNNs) for segmentation. Medical experts determine the consistency between the proposed ML approach, which is a state-of-the-art deep learning method, and the manual segmentation approach. These methods are compared in terms of performance indicators such as the ventricular area (VA), ventricular maximum diameter (VMXD), ventricular minimum diameter (VMID), and ventricular long axis angle (AVLA) measurements. Furthermore, the Dice similarity coefficient, Jaccard index, and Hausdorff distance are measured to estimate the agreement of the LV segmented results between the automatic and visual approaches. The obtained results indicate that the proposed techniques for LV segmentation are useful and practical. There is no significant difference between the use of AC and CNN in image segmentation; however, the AC method could obtain comparable accuracy as the CNN method using less training data and less run-time.

A novel U-Net approach to segment the cardiac chamber in magnetic resonance images with ghost artifacts

OBJECTIVE

We propose a robust technique for segmenting magnetic resonance images of post-atrial septal occlusion intervention in the cardiac chamber.

METHODS

A variant of the U-Net architecture is used to perform atrial segmentation via a deep convolutional neural network, and we compare performance with the Kass snake model. It can be used to determine the surgical success of atrial septal occlusion (ASO) pre- and post- the implantation of the septal occluder, which is based on the volume restoration of the right atria (RA) and left atria (LA).

RESULTS

The method was evaluated on a test dataset containing 550 two-dimensional image slices, outperforming conventional active contouring regarding the Dice similarity coefficient, Jaccard index, and Hausdorff distance, and achieving segmentation in the presence of ghost artifacts that occlude the atrium outline. This problem has been unsolvable using traditional machine learning algorithm pertaining to active contouring via the Kass snake algorithm. Moreover, the proposed technique is closer to manual segmentation than the snakes active contour model in mean of atrial area (M-AA), mean of atrial maximum diameter (M-AMXD), mean atrial minimum diameter (M-AMID), and mean angle of the atrial long axis (M-AALA).

CONCLUSION

After segmentation, we compute the volume ratio of right to left atria, obtaining a smaller ratio that indicates better restoration. Hence, the proposed technique allows to evaluate the surgical success of atrial septal occlusion and may support diagnosis regarding the accurate evaluation of atrial septal defects before and after occlusion procedures.

Cardiac Chamber Segmentation Using Deep Learning on Magnetic Resonance Images from Patients Before and After Atrial Septal Occlusion Surgery

We propose a robust technique for segmenting magnetic resonance images of post-atrial septal occlusion intervention in the cardiac chamber. The technique can be used to determine the surgical outcomes of atrial septal defects before and after implantation of a septal occluder, which intends to provide volume restoration of the right and left atria. A variant of the U-Net architecture is used to perform atrial segmentation via a deep convolutional neural network. The method was evaluated on a dataset containing 550 two-dimensional image slices, outperforming conventional active contouring regarding the Dice similarity coefficient, Jaccard index, and Hausdorff distance, and achieving segmentation in the presence of ghost artifacts that occlude the atrium outline. Moreover, the proposed technique is closer to manual segmentation than the snakes active contour model. After segmentation, we computed the volume ratio of right to left atria, obtaining a smaller ratio that indicates better restoration. Hence, the proposed technique allows to evaluate the surgical success of atrial septal occlusion and may support diagnosis regarding the accurate evaluation of atrial septal defects before and after occlusion procedures.

Reference-free error estimation for multiple measurement methods

We present a computational framework to select the most accurate and precise method of measurement of a certain quantity, when there is no access to the true value of the measurand. A typical use case is when several image analysis methods are applied to measure the value of a particular quantitative imaging biomarker from the same images. The accuracy of each measurement method is characterized by systematic error (bias), which is modeled as a polynomial in true values of measurand, and the precision as random error modeled with a Gaussian random variable. In contrast to previous works, the random errors are modeled jointly across all methods, thereby enabling the framework to analyze measurement methods based on similar principles, which may have correlated random errors. Furthermore, the posterior distribution of the error model parameters is estimated from samples obtained by Markov chain Monte-Carlo and analyzed to estimate the parameter values and the unknown true values of the measurand. The framework was validated on six synthetic and one clinical dataset containing measurements of total lesion load, a biomarker of neurodegenerative diseases, which was obtained with four automatic methods by analyzing brain magnetic resonance images. The estimates of bias and random error were in a good agreement with the corresponding least squares regression estimates against a reference.

Practical no-gold-standard evaluation framework for quantitative imaging methods: application to lesion segmentation in positron emission tomography

Recently, a class of no-gold-standard (NGS) techniques have been proposed to evaluate quantitative imaging methods using patient data. These techniques provide figures of merit (FoMs) quantifying the precision of the estimated quantitative value without requiring repeated measurements and without requiring a gold standard. However, applying these techniques to patient data presents several practical difficulties including assessing the underlying assumptions, accounting for patient-sampling-related uncertainty, and assessing the reliability of the estimated FoMs. To address these issues, we propose statistical tests that provide confidence in the underlying assumptions and in the reliability of the estimated FoMs. Furthermore, the NGS technique is integrated within a bootstrap-based methodology to account for patient-sampling-related uncertainty. The developed NGS framework was applied to evaluate four methods for segmenting lesions from F-Fluoro-2-deoxyglucose positron emission tomography images of patients with head-and-neck cancer on the task of precisely measuring the metabolic tumor volume. The NGS technique consistently predicted the same segmentation method as the most precise method. The proposed framework provided confidence in these results, even when gold-standard data were not available. The bootstrap-based methodology indicated improved performance of the NGS technique with larger numbers of patient studies, as was expected, and yielded consistent results as long as data from more than 80 lesions were available for the analysis.

No-gold-standard evaluation of image-acquisition methods using patient data

Several new and improved modalities, scanners, and protocols, together referred to as image-acquisition methods (IAMs), are being developed to provide reliable quantitative imaging. Objective evaluation of these IAMs on the clinically relevant quantitative tasks is highly desirable. Such evaluation is most reliable and clinically decisive when performed with patient data, but that requires the availability of a gold standard, which is often rare. While no-gold-standard (NGS) techniques have been developed to clinically evaluate quantitative imaging methods, these techniques require that each of the patients be scanned using all the IAMs, which is expensive, time consuming, and could lead to increased radiation dose. A more clinically practical scenario is where different set of patients are scanned using different IAMs. We have developed an NGS technique that uses patient data where different patient sets are imaged using different IAMs to compare the different IAMs. The technique posits a linear relationship, characterized by a slope, bias, and noise standard-deviation term, between the true and measured quantitative values. Under the assumption that the true quantitative values have been sampled from a unimodal distribution, a maximum-likelihood procedure was developed that estimates these linear relationship parameters for the different IAMs. Figures of merit can be estimated using these linear relationship parameters to evaluate the IAMs on the basis of accuracy, precision, and overall reliability. The proposed technique has several potential applications such as in protocol optimization, quantifying difference in system performance, and system harmonization using patient data.

An improved FSL-FIRST pipeline for subcortical gray matter segmentation to study abnormal brain anatomy using quantitative susceptibility mapping (QSM)

Accurate and robust segmentation of subcortical gray matter (SGM) nuclei is required in many neuroimaging applications. FMRIB's Integrated Registration and Segmentation Tool (FIRST) is one of the most popular software tools for automated subcortical segmentation based on T₁-weighted (T1w) images. In this work, we demonstrate that FIRST tends to produce inaccurate SGM segmentation results in the case of abnormal brain anatomy, such as present in atrophied brains, due to a poor spatial match of the subcortical structures with the training data in the MNI space as well as due to insufficient contrast of SGM structures on T1w images. Consequently, such deviations from the average brain anatomy may introduce analysis bias in clinical studies, which may not always be obvious and potentially remain unidentified. To improve the segmentation of subcortical nuclei, we propose to use FIRST in combination with a special Hybrid image Contrast (HC) and Non-Linear (nl) registration module (HC-nlFIRST), where the hybrid image contrast is derived from T1w images and magnetic susceptibility maps to create subcortical contrast that is similar to that in the Montreal Neurological Institute (MNI) template. In our approach, a nonlinear registration replaces FIRST's default linear registration, yielding a more accurate alignment of the input data to the MNI template. We evaluated our method on 82 subjects with particularly abnormal brain anatomy, selected from a database of >2000 clinical cases. Qualitative and quantitative analyses revealed that HC-nlFIRST provides improved segmentation compared to the default FIRST method.

An SPCNN-GVF-based approach for the automatic segmentation of left ventricle in cardiac cine MR images

PURPOSE

Accurate segmentation of left ventricle (LV) is essential for the cardiac function analysis. However, it is labor intensive and time consuming for radiologists to delineate LV boundary manually. In this paper, we present a novel self-correcting framework for the fully automatic LV segmentation.

METHODS

Firstly, a time-domain method is designed to extract a rectangular region of interest around the heart. Then, the simplified pulse-coupled neural network (SPCNN) is employed to locate the LV cavity. Different from the existing approaches, SPCNN can realize the self-correcting segmentation due to its parameter controllability. Subsequently, the post-processing based on the maximum gradient searching is proposed to obtain the accurate endocardium. Finally, a new external force based on the shape similarity is defined and integrated into the gradient vector flow (GVF) snake with the balloon force to segment the epicardium.

RESULTS

We obtain encouraging segmentation results tested on the database provided by MICCAI 2009. The average percentage of good contours is 92.26 %, the average perpendicular distance is 2.38 mm, and the overlapping dice metric is 0.89. Besides, the experiment results show good correlations between the automatic segmentation and the manual delineation (for the LV ejection fraction and the LV myocardial mass, the correlation coefficients R are 0.9683 and 0.9278, respectively).

CONCLUSION

We propose an effective and fast method combing the SPCNN and the improved GVF for the automatic segmentation of LV.

A no-gold-standard technique for objective assessment of quantitative nuclear-medicine imaging methods

The objective optimization and evaluation of nuclear-medicine quantitative imaging methods using patient data is highly desirable but often hindered by the lack of a gold standard. Previously, a regression-without-truth (RWT) approach has been proposed for evaluating quantitative imaging methods in the absence of a gold standard, but this approach implicitly assumes that bounds on the distribution of true values are known. Several quantitative imaging methods in nuclear-medicine imaging measure parameters where these bounds are not known, such as the activity concentration in an organ or the volume of a tumor. We extended upon the RWT approach to develop a no-gold-standard (NGS) technique for objectively evaluating such quantitative nuclear-medicine imaging methods with patient data in the absence of any ground truth. Using the parameters estimated with the NGS technique, a figure of merit, the noise-to-slope ratio (NSR), can be computed, which can rank the methods on the basis of precision. An issue with NGS evaluation techniques is the requirement of a large number of patient studies. To reduce this requirement, the proposed method explored the use of multiple quantitative measurements from the same patient, such as the activity concentration values from different organs in the same patient. The proposed technique was evaluated using rigorous numerical experiments and using data from realistic simulation studies. The numerical experiments demonstrated that the NSR was estimated accurately using the proposed NGS technique when the bounds on the distribution of true values were not precisely known, thus serving as a very reliable metric for ranking the methods on the basis of precision. In the realistic simulation study, the NGS technique was used to rank reconstruction methods for quantitative single-photon emission computed tomography (SPECT) based on their performance on the task of estimating the mean activity concentration within a known volume of interest. Results showed that the proposed technique provided accurate ranking of the reconstruction methods for 97.5% of the 50 noise realizations. Further, the technique was robust to the choice of evaluated reconstruction methods. The simulation study pointed to possible violations of the assumptions made in the NGS technique under clinical scenarios. However, numerical experiments indicated that the NGS technique was robust in ranking methods even when there was some degree of such violation.

Is STAPLE algorithm confident to assess segmentation methods in PET imaging?

Accurate tumor segmentation in [18F]-fluorodeoxyglucose positron emission tomography is crucial for tumor response assessment and target volume definition in radiation therapy. Evaluation of segmentation methods from clinical data without ground truth is usually based on physicians' manual delineations. In this context, the simultaneous truth and performance level estimation (STAPLE) algorithm could be useful to manage the multi-observers variability. In this paper, we evaluated how this algorithm could accurately estimate the ground truth in PET imaging. Complete evaluation study using different criteria was performed on simulated data. The STAPLE algorithm was applied to manual and automatic segmentation results. A specific configuration of the implementation provided by the Computational Radiology Laboratory was used. Consensus obtained by the STAPLE algorithm from manual delineations appeared to be more accurate than manual delineations themselves (80% of overlap). An improvement of the accuracy was also observed when applying the STAPLE algorithm to automatic segmentations results. The STAPLE algorithm, with the configuration used in this paper, is more appropriate than manual delineations alone or automatic segmentations results alone to estimate the ground truth in PET imaging. Therefore, it might be preferred to assess the accuracy of tumor segmentation methods in PET imaging.

Brain extraction in pediatric ADC maps, toward characterizing neuro-development in multi-platform and multi-institution clinical images

Apparent Diffusion Coefficient (ADC) maps can be used to characterize myelination and to detect abnormalities in the developing brain. However, given the normal variation in regional ADC with myelination, detection of abnormalities is difficult when based on visual assessment. Quantitative and automated analysis of pediatric ADC maps is thus desired but requires accurate brain extraction as the first step. Currently, most existing brain extraction methods are optimized for structural T1-weighted MR images of fully myelinated brains. Due to differences in age and image contrast, these approaches do not translate well to pediatric ADC maps. To address this problem, we present a multi-atlas brain extraction framework that has 1) specificity: designed and optimized specifically for pediatric ADC maps; 2) generality: applicable to multi-platform and multi-institution data, and to subjects at various neuro-developmental stages across the first 6 years of life; 3) accuracy: highly accurate compared to expert annotations; and 4) consistency: consistently accurate regardless of sources of data and ages of subjects. We show how we achieve these goals, via optimizing major components in a multi-atlas brain extraction framework, and via developing and evaluating new criteria for its atlas ranking component. Moreover, we demonstrate that these goals can be achieved with a fixed set of atlases and a fixed set of parameters, which opens doors for our optimized framework to be used in large-scale and multi-institution neuro-developmental and clinical studies. In a pilot study, we use this framework in a dataset containing scanner-generated ADC maps from 308 pediatric patients collected during the course of routine clinical care. Our framework leads to successful quantifications of the changes in whole-brain volumes and mean ADC values across the first 6 years of life.

Improved Estimation of Cardiac Function Parameters Using a Combination of Independent Automated Segmentation Results in Cardiovascular Magnetic Resonance Imaging

This work aimed at combining different segmentation approaches to produce a robust and accurate segmentation result. Three to five segmentation results of the left ventricle were combined using the STAPLE algorithm and the reliability of the resulting segmentation was evaluated in comparison with the result of each individual segmentation method. This comparison was performed using a supervised approach based on a reference method. Then, we used an unsupervised statistical evaluation, the extended Regression Without Truth (eRWT) that ranks different methods according to their accuracy in estimating a specific biomarker in a population. The segmentation accuracy was evaluated by estimating six cardiac function parameters resulting from the left ventricle contour delineation using a public cardiac cine MRI database. Eight different segmentation methods, including three expert delineations and five automated methods, were considered, and sixteen combinations of the automated methods using STAPLE were investigated. The supervised and unsupervised evaluations demonstrated that in most cases, STAPLE results provided better estimates than individual automated segmentation methods. Overall, combining different automated segmentation methods improved the reliability of the segmentation result compared to that obtained using an individual method and could achieve the accuracy of an expert.

Fully automated segmentation of the left ventricle applied to cine MR images: description and results on a database of 45 subjects

A fully automated segmentation method of the left ventricle from short-axis cardiac MR images is proposed and evaluated. The segmentation is based on morphological filtering and gradient vector flow snake for which an automatic setting of parameters has already been proposed. The present work focuses on the automatic detection of a region of interest (ROI) surrounding the left ventricle, prior to the segmentation step. The whole process was applied to the MICCAI 2009 Left Ventricle Challenge database containing 45 subjects (9 healthy subjects and 36 with pathology). The automatic detection of the ROI was judged accurate in 86% of the cases. It failed in 2% of the slices and provided an overestimation in 9% of the slices. Furthermore, the endocardial segmentation was accurate in 80% of the slices and the epicardial was judged satisfactory in 71% of the slices. This fully automated procedure can thus be used as a first step in a user controlled approach, in order to reduce the total number of interactions.