Globally minimal surfaces by continuous maximal flows

Integrated platform for multiscale molecular imaging and phenotyping of the human brain

Understanding cellular architectures and their connectivity is essential for interrogating system function and dysfunction. However, we lack technologies for mapping the multiscale details of individual cells and their connectivity in the human organ-scale system. We developed a platform that simultaneously extracts spatial, molecular, morphological, and connectivity information of individual cells from the same human brain. The platform includes three core elements: a vibrating microtome for ultraprecision slicing of large-scale tissues without losing cellular connectivity (MEGAtome), a polymer hydrogel-based tissue processing technology for multiplexed multiscale imaging of human organ-scale tissues (mELAST), and a computational pipeline for reconstructing three-dimensional connectivity across multiple brain slabs (UNSLICE). We applied this platform for analyzing human Alzheimer's disease pathology at multiple scales and demonstrating scalable neural connectivity mapping in the human brain.

Consistent Estimation of the Max-Flow Problem: Towards Unsupervised Image Segmentation

Advances in the image-based diagnostics of complex biological and manufacturing processes have brought unsupervised image segmentation to the forefront of enabling automated, on the fly decision making. However, most existing unsupervised segmentation approaches are either computationally complex or require manual parameter selection (e.g., flow capacities in max-flow/min-cut segmentation). In this work, we present a fully unsupervised segmentation approach using a continuous max-flow formulation over the image domain while optimally estimating the flow parameters from the image characteristics. More specifically, we show that the maximum a posteriori estimate of the image labels can be formulated as a continuous max-flow problem given the flow capacities are known. The flow capacities are then iteratively obtained by employing a novel Markov random field prior over the image domain. We present theoretical results to establish the posterior consistency of the flow capacities. We compare the performance of our approach using brain tumor image segmentation, defect identification in additively manufactured components using electron microscopic images, and segmentation of multiple real-world images. Comparative results with several state-of-the-art supervised as well as unsupervised methods suggest that the present method performs statistically similar to the supervised methods, but results in more than 90 percent improvement in the Dice score when compared to the state-of-the-art unsupervised methods.

A Continuous Random Walk Model With Explicit Coherence Regularization for Image Segmentation

Random walk is a popular and efficient algorithm for image segmentation, especially for extracting regions of interest (ROIs). One difficulty with the random walk algorithm is the requirement for solving a huge sparse linear system when applied to large images. Another limitation is its sensitivity to seeds distribution, i.e., the segmentation result depends on the number of seeds as well as their placement, which puts a burden on users. In this paper, we first propose a continuous random walk model with explicit coherence regularization (CRWCR) for the extracted ROI, which helps to reduce the seeds sensitivity, so as to reduce the user interactions. Then, a very efficient algorithm to solve the CRWCR model will be developed, which helps to remove the difficulty of solving huge linear systems. Our algorithm consists of two stages: initialization by performing one-dimensional random walk sweeping based on user-provided seeds, followed by the alternating direction scheme, i.e., Peaceman-Rachford scheme for further correction. The first stage aims to provide a good initial guess for the ROI, and it is very fast since we just solve a limited number of one-dimensional random walk problems. Then, this initial guess is evolved to the ideal solution by applying the second stage, which should also be very efficient since it fits well for GPU computing, and 10 iterations are usually sufficient for convergence. Numerical experiments are provided to validate the proposed model as well as the efficiency of the two-stage algorithm.

ACTIVE MEAN FIELDS FOR PROBABILISTIC IMAGE SEGMENTATION: CONNECTIONS WITH CHAN-VESE AND RUDIN-OSHER-FATEMI MODELS

Segmentation is a fundamental task for extracting semantically meaningful regions from an image. The goal of segmentation algorithms is to accurately assign object labels to each image location. However, image-noise, shortcomings of algorithms, and image ambiguities cause uncertainty in label assignment. Estimating the uncertainty in label assignment is important in multiple application domains, such as segmenting tumors from medical images for radiation treatment planning. One way to estimate these uncertainties is through the computation of posteriors of Bayesian models, which is computationally prohibitive for many practical applications. On the other hand, most computationally efficient methods fail to estimate label uncertainty. We therefore propose in this paper the Active Mean Fields (AMF) approach, a technique based on Bayesian modeling that uses a mean-field approximation to efficiently compute a segmentation and its corresponding uncertainty. Based on a variational formulation, the resulting convex model combines any label-likelihood measure with a prior on the length of the segmentation boundary. A specific implementation of that model is the Chan-Vese segmentation model (CV), in which the binary segmentation task is defined by a Gaussian likelihood and a prior regularizing the length of the segmentation boundary. Furthermore, the Euler-Lagrange equations derived from the AMF model are equivalent to those of the popular Rudin-Osher-Fatemi (ROF) model for image denoising. Solutions to the AMF model can thus be implemented by directly utilizing highly-efficient ROF solvers on log-likelihood ratio fields. We qualitatively assess the approach on synthetic data as well as on real natural and medical images. For a quantitative evaluation, we apply our approach to the icgbench dataset.

A Framework for Classification and Segmentation of Branch Retinal Artery Occlusion in SD-OCT

Branch retinal artery occlusion (BRAO) is an ocular emergency, which could lead to blindness. Quantitative analysis of the BRAO region in the retina is necessary for the assessment of the severity of retinal ischemia. In this paper, a fully automatic framework was proposed to segment BRAO regions based on 3D spectral-domain optical coherence tomography (SD-OCT) images. To the best of our knowledge, this is the first automatic 3D BRAO segmentation framework. First, the input 3D image is automatically classified into BRAO of acute phase and BRAO of chronic phase or normal retina using an AdaBoost classifier based on combining local structural, intensity, textural features with our new feature distribution analyzing strategy. Then, BRAO regions of acute phase and chronic phase are segmented separately. A thickness model is built to segment BRAO in the chronic phase. While for segmenting BRAO in the acute phase, a two-step segmentation strategy is performed: rough initialization and refine segmentation. The proposed method was tested on SD-OCT images of 35 patients (12 BRAO acute phase, 11 BRAO chronic phase, and 12 normal eyes) using the leave-one-out strategy. The classification accuracy for BRAO acute phase, BRAO chronic phase, and normal retina were 100%, 90.9%, and 91.7%, respectively. The overall true positive volume fraction (TPVF) and false positive volume fraction (FPVF) for the acute phase were 91.1% and 5.5% and for the chronic phase were 92.7% and 8.4%, respectively.

Dynamic Programming Using Polar Variance for Image Segmentation

When using polar dynamic programming (PDP) for image segmentation, the object size is one of the main features used. This is because if size is left unconstrained the final segmentation may include high-gradient regions that are not associated with the object. In this paper, we propose a new feature, polar variance, which allows the algorithm to segment the objects of different sizes without the need for training data. The polar variance is the variance in a polar region between a user-selected origin and a pixel we want to analyze. We also incorporate a new technique that allows PDP to segment complex shapes by finding low-gradient regions and growing them. The experimental analysis consisted on comparing our technique with different active contour segmentation techniques on a series of tests. The tests consisted on robustness to additive Gaussian noise, segmentation accuracy with different grayscale images and finally robustness to algorithm-specific parameters. Experimental results show that our technique performs favorably when compared with other segmentation techniques.

Energy minimization in medical image analysis: Methodologies and applications

Energy minimization is of particular interest in medical image analysis. In the past two decades, a variety of optimization schemes have been developed. In this paper, we present a comprehensive survey of the state-of-the-art optimization approaches. These algorithms are mainly classified into two categories: continuous method and discrete method. The former includes Newton-Raphson method, gradient descent method, conjugate gradient method, proximal gradient method, coordinate descent method, and genetic algorithm-based method, while the latter covers graph cuts method, belief propagation method, tree-reweighted message passing method, linear programming method, maximum margin learning method, simulated annealing method, and iterated conditional modes method. We also discuss the minimal surface method, primal-dual method, and the multi-objective optimization method. In addition, we review several comparative studies that evaluate the performance of different minimization techniques in terms of accuracy, efficiency, or complexity. These optimization techniques are widely used in many medical applications, for example, image segmentation, registration, reconstruction, motion tracking, and compressed sensing. We thus give an overview on those applications as well.

Automatic atlas-based three-label cartilage segmentation from MR knee images

Osteoarthritis (OA) is the most common form of joint disease and often characterized by cartilage changes. Accurate quantitative methods are needed to rapidly screen large image databases to assess changes in cartilage morphology. We therefore propose a new automatic atlas-based cartilage segmentation method for future automatic OA studies. Atlas-based segmentation methods have been demonstrated to be robust and accurate in brain imaging and therefore also hold high promise to allow for reliable and high-quality segmentations of cartilage. Nevertheless, atlas-based methods have not been well explored for cartilage segmentation. A particular challenge is the thinness of cartilage, its relatively small volume in comparison to surrounding tissue and the difficulty to locate cartilage interfaces - for example the interface between femoral and tibial cartilage. This paper focuses on the segmentation of femoral and tibial cartilage, proposing a multi-atlas segmentation strategy with non-local patch-based label fusion which can robustly identify candidate regions of cartilage. This method is combined with a novel three-label segmentation method which guarantees the spatial separation of femoral and tibial cartilage, and ensures spatial regularity while preserving the thin cartilage shape through anisotropic regularization. Our segmentation energy is convex and therefore guarantees globally optimal solutions. We perform an extensive validation of the proposed method on 706 images of the Pfizer Longitudinal Study. Our validation includes comparisons of different atlas segmentation strategies, different local classifiers, and different types of regularizers. To compare to other cartilage segmentation approaches we validate based on the 50 images of the SKI10 dataset.

An efficient optimization framework for multi-region segmentation based on Lagrangian duality

We introduce a multi-region model for simultaneous segmentation of medical images. In contrast to many other models, geometric constraints such as inclusion and exclusion between the regions are enforced, which makes it possible to correctly segment different regions even if the intensity distributions are identical. We efficiently optimize the model using a combination of graph cuts and Lagrangian duality which is faster and more memory efficient than current state of the art. As the method is based on global optimization techniques, the resulting segmentations are independent of initialization. We apply our framework to the segmentation of the left and right ventricles, myocardium and the left ventricular papillary muscles in magnetic resonance imaging and to lung segmentation in full-body X-ray computed tomography. We evaluate our approach on a publicly available benchmark with competitive results.

Segmentation with area constraints

Image segmentation approaches typically incorporate weak regularity conditions such as boundary length or curvature terms, or use shape information. High-level information such as a desired area or volume, or a particular topology are only implicitly specified. In this paper we develop a segmentation method with explicit bounds on the segmented area. Area constraints allow for the soft selection of meaningful solutions, and can counteract the shrinking bias of length-based regularization. We analyze the intrinsic problems of convex relaxations proposed in the literature for segmentation with size constraints. Hence, we formulate the area-constrained segmentation task as a mixed integer program, propose a branch and bound method for exact minimization, and use convex relaxations to obtain the required lower energy bounds on candidate solutions. We also provide a numerical scheme to solve the convex subproblems. We demonstrate the method for segmentations of vesicles from electron tomography images.

Three-dimensional segmentation of fluid-associated abnormalities in retinal OCT: probability constrained graph-search-graph-cut

An automated method is reported for segmenting 3-D fluid-associated abnormalities in the retina, so-called symptomatic exudate-associated derangements (SEAD), from 3-D OCT retinal images of subjects suffering from exudative age-related macular degeneration. In the first stage of a two-stage approach, retinal layers are segmented, candidate SEAD regions identified, and the retinal OCT image is flattened using a candidate-SEAD aware approach. In the second stage, a probability constrained combined graph search-graph cut method refines the candidate SEADs by integrating the candidate volumes into the graph cut cost function as probability constraints. The proposed method was evaluated on 15 spectral domain OCT images from 15 subjects undergoing intravitreal anti-VEGF injection treatment. Leave-one-out evaluation resulted in a true positive volume fraction (TPVF), false positive volume fraction (FPVF) and relative volume difference ratio (RVDR) of 86.5%, 1.7%, and 12.8%, respectively. The new graph cut-graph search method significantly outperformed both the traditional graph cut and traditional graph search approaches (p < 0.01, p < 0.04) and has the potential to improve clinical management of patients with choroidal neovascularization due to exudative age-related macular degeneration.

Power Watershed: A Unifying Graph-Based Optimization Framework

In this work, we extend a common framework for graph-based image segmentation that includes the graph cuts, random walker, and shortest path optimization algorithms. Viewing an image as a weighted graph, these algorithms can be expressed by means of a common energy function with differing choices of a parameter q acting as an exponent on the differences between neighboring nodes. Introducing a new parameter p that fixes a power for the edge weights allows us to also include the optimal spanning forest algorithm for watershed in this same framework. We then propose a new family of segmentation algorithms that fixes p to produce an optimal spanning forest but varies the power q beyond the usual watershed algorithm, which we term the power watershed. In particular, when q=2, the power watershed leads to a multilabel, scale and contrast invariant, unique global optimum obtained in practice in quasi-linear time. Placing the watershed algorithm in this energy minimization framework also opens new possibilities for using unary terms in traditional watershed segmentation and using watershed to optimize more general models of use in applications beyond image segmentation.

Region detection by minimizing intraclass variance with geometric constraints, global optimality, and efficient approximation

Efficient segmentation of globally optimal surfaces in volumetric images is a central problem in many medical image analysis applications. Intraclass variance has been successfully utilized for object segmentation, for instance, in the Chan-Vese model, especially for images without prominent edges. In this paper, we study the optimization problem of detecting a region (volume) between two coupled smooth surfaces by minimizing the intraclass variance using an efficient polynomial-time algorithm. Our algorithm is based on the shape probing technique in computational geometry and computes a sequence of minimum-cost closed sets in a derived parametric graph. The method has been validated on computer-synthetic volumetric images and in X-ray CT-scanned datasets of plexiglas tubes of known sizes. Its applicability to clinical data sets was also demonstrated. In all cases, the approach yielded highly accurate results. We believe that the developed technique is of interest on its own. We expect that it can shed some light on solving other important optimization problems arising in medical imaging. Furthermore, we report an approximation algorithm which runs much faster than the exact algorithm while yielding highly comparable segmentation accuracy.

Minimal surfaces extend shortest path segmentation methods to 3D

Shortest paths have been used to segment object boundaries with both continuous and discrete image models. Although these techniques are well defined in 2D, the character of the path as an object boundary is not preserved in 3D. An object boundary in three dimensions is a 2D surface. However, many different extensions of the shortest path techniques to 3D have been previously proposed in which the 3D object is segmented via a collection of shortest paths rather than a minimal surface, leading to a solution which bears an uncertain relationship to the true minimal surface. Specifically, there is no guarantee that a minimal path between points on two closed contours will lie on the minimal surface joining these contours. We observe that an elegant solution to the computation of a minimal surface on a cellular complex (e.g., a 3D lattice) was given by Sullivan [47]. Sullivan showed that the discrete minimal surface connecting one or more closed contours may be found efficiently by solving a Minimum-cost Circulation Network Flow (MCNF) problem. In this work, we detail why a minimal surface properly extends a shortest path (in the context of a boundary) to three dimensions, present Sullivan's solution to this minimal surface problem via an MCNF calculation, and demonstrate the use of these minimal surfaces on the segmentation of image data.

Continuous Maximal Flows and Wulff Shapes: Application to MRFs

Convex and continuous energy formulations for low level vision problems enable efficient search procedures for the corresponding globally optimal solutions. In this work we extend the well-established continuous, isotropic capacity-based maximal flow framework to the anisotropic setting. By using powerful results from convex analysis, a very simple and efficient minimization procedure is derived. Further, we show that many important properties carry over to the new anisotropic framework, e.g. globally optimal binary results can be achieved simply by thresholding the continuous solution. In addition, we unify the anisotropic continuous maximal flow approach with a recently proposed convex and continuous formulation for Markov random fields, thereby allowing more general smoothness priors to be incorporated. Dense stereo results are included to illustrate the capabilities of the proposed approach.

Globally Optimal Finsler Active Contours

We present a continuous and convex formulation for Finsler active contours using seed regions or utilizing a regional bias term. The utilization of general Finsler metrics instead of Riemannian metrics allows the segmentation boundary to favor appropriate locations (e.g. with strong image discontinuities) and suitable directions (e.g. aligned with dark to bright image gradients). Strong edges are not required everywhere along the desired segmentation boundary due to incorporation of a regional bias. The resulting optimization procedure is simple and efficient, and leads to binary segmentation results regardless of the underlying continuous formulation. We demonstrate the proposed method in several examples.

A high-throughput system for segmenting nuclei using multiscale techniques

Automatic segmentation of cell nuclei is critical in several high-throughput cytometry applications whereas manual segmentation is laborious and irreproducible. One such emerging application is measuring the spatial organization (radial and relative distances) of fluorescence in situ hybridization (FISH) DNA sequences, where recent investigations strongly suggest a correlation between nonrandom arrangement of genes to carcinogenesis. Current automatic segmentation methods have varying performance in the presence of nonuniform illumination and clustering, and boundary accuracy is seldom assessed, which makes them suboptimal for this application. The authors propose a modular and model-based algorithm for extracting individual nuclei. It uses multiscale edge reconstruction for contrast stretching and edge enhancement as well as a multiscale entropy-based thresholding for handling nonuniform intensity variations. Nuclei are initially oversegmented and then merged based on area followed by automatic multistage classification into single nuclei and clustered nuclei. Estimation of input parameters and training of the classifiers is automatic. The algorithm was tested on 4,181 lymphoblast nuclei with varying degree of background nonuniformity and clustering. It extracted 3,515 individual nuclei and identified single nuclei and individual nuclei in clusters with 99.8 +/- 0.3% and 95.5 +/- 5.1% accuracy, respectively. Segmented boundaries of the individual nuclei were accurate when compared with manual segmentation with an average RMS deviation of 0.26 microm (approximately 2 pixels). The proposed segmentation method is efficient, robust, and accurate for segmenting individual nuclei from fluorescence images containing clustered and isolated nuclei. The algorithm allows complete automation and facilitates reproducible and unbiased spatial analysis of DNA sequences.

Segmentation of whole cells and cell nuclei from 3-D optical microscope images using dynamic programming

Communications between cells in large part drive tissue development and function, as well as disease-related processes such as tumorigenesis. Understanding the mechanistic bases of these processes necessitates quantifying specific molecules in adjacent cells or cell nuclei of intact tissue. However, a major restriction on such analyses is the lack of an efficient method that correctly segments each object (cell or nucleus) from 3-D images of an intact tissue specimen. We report a highly reliable and accurate semi-automatic algorithmic method for segmenting fluorescence-labeled cells or nuclei from 3-D tissue images. Segmentation begins with semi-automatic, 2-D object delineation in a user-selected plane, using dynamic programming (DP) to locate the border with an accumulated intensity per unit length greater that any other possible border around the same object. Then the two surfaces of the object in planes above and below the selected plane are found using an algorithm that combines DP and combinatorial searching. Following segmentation, any perceived errors can be interactively corrected. Segmentation accuracy is not significantly affected by intermittent labeling of object surfaces, diffuse surfaces, or spurious signals away from surfaces. The unique strength of the segmentation method was demonstrated on a variety of biological tissue samples where all cells, including irregularly shaped cells, were accurately segmented based on visual inspection.

Interactive image segmentation via adaptive weighted distances

An interactive algorithm for soft segmentation of natural images is presented in this paper. The user first roughly scribbles different regions of interest, and from them, the whole image is automatically segmented. This soft segmentation is obtained via fast, linear complexity computation of weighted distances to the user-provided scribbles. The adaptive weights are obtained from a series of Gabor filters, and are automatically computed according to the ability of each single filter to discriminate between the selected regions of interest. We present the underlying framework and examples showing the capability of the algorithm to segment diverse images.

Automatic segmentation of left and right cerebral hemispheres from MRI brain volumes using the graph cuts algorithm

An automated algorithm has been developed to segment stripped (non-brain tissue excluded) T1-weighted MRI brain volumes into left and right cerebral hemispheres and cerebellum+brainstem. The algorithm, which uses the Graph Cuts technique, performs a fully automated segmentation in approximately 30 s following pre-processing. It is robust and accurate and has been tested on datasets from two scanners using different field strengths and pulse sequences. We describe the Graph Cuts algorithm and compare the results of Graph Cuts segmentations against "gold standard" manual segmentations and segmentations produced by three popular software packages used by neuroimagers: BrainVisa, CLASP, and SurfRelax.

An energy-based three-dimensional segmentation approach for the quantitative interpretation of electron tomograms

Electron tomography allows for the determination of the three-dimensional structures of cells and tissues at resolutions significantly higher than that which is possible with optical microscopy. Electron tomograms contain, in principle, vast amounts of information on the locations and architectures of large numbers of subcellular assemblies and organelles. The development of reliable quantitative approaches for the analysis of features in tomograms is an important problem, and a challenging prospect due to the low signal-to-noise ratios that are inherent to biological electron microscopic images. This is, in part, a consequence of the tremendous complexity of biological specimens. We report on a new method for the automated segmentation of HIV particles and selected cellular compartments in electron tomograms recorded from fixed, plastic-embedded sections derived from HIV-infected human macrophages. Individual features in the tomogram are segmented using a novel robust algorithm that finds their boundaries as global minimal surfaces in a metric space defined by image features. The optimization is carried out in a transformed spherical domain with the center an interior point of the particle of interest, providing a proper setting for the fast and accurate minimization of the segmentation energy. This method provides tools for the semi-automated detection and statistical evaluation of HIV particles at different stages of assembly in the cells and presents opportunities for correlation with biochemical markers of HIV infection. The segmentation algorithm developed here forms the basis of the automated analysis of electron tomograms and will be especially useful given the rapid increases in the rate of data acquisition. It could also enable studies of much larger data sets, such as those which might be obtained from the tomographic analysis of HIV-infected cells from studies of large populations.