A common framework for image segmentation

Improved Reptile Search Algorithm by Salp Swarm Algorithm for Medical Image Segmentation

This study proposes a novel nature-inspired meta-heuristic optimizer based on the Reptile Search Algorithm combed with Salp Swarm Algorithm for image segmentation using gray-scale multi-level thresholding, called RSA-SSA. The proposed method introduces a better search space to find the optimal solution at each iteration. However, we proposed RSA-SSA to avoid the searching problem in the same area and determine the optimal multi-level thresholds. The obtained solutions by the proposed method are represented using the image histogram. The proposed RSA-SSA employed Otsu's variance class function to get the best threshold values at each level. The performance measure for the proposed method is valid by detecting fitness function, structural similarity index, peak signal-to-noise ratio, and Friedman ranking test. Several benchmark images of COVID-19 validate the performance of the proposed RSA-SSA. The results showed that the proposed RSA-SSA outperformed other metaheuristics optimization algorithms published in the literature.

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

In this work we address the task of semantic image segmentation with Deep Learning and make three main contributions that are experimentally shown to have substantial practical merit. First, we highlight convolution with upsampled filters, or 'atrous convolution', as a powerful tool in dense prediction tasks. Atrous convolution allows us to explicitly control the resolution at which feature responses are computed within Deep Convolutional Neural Networks. It also allows us to effectively enlarge the field of view of filters to incorporate larger context without increasing the number of parameters or the amount of computation. Second, we propose atrous spatial pyramid pooling (ASPP) to robustly segment objects at multiple scales. ASPP probes an incoming convolutional feature layer with filters at multiple sampling rates and effective fields-of-views, thus capturing objects as well as image context at multiple scales. Third, we improve the localization of object boundaries by combining methods from DCNNs and probabilistic graphical models. The commonly deployed combination of max-pooling and downsampling in DCNNs achieves invariance but has a toll on localization accuracy. We overcome this by combining the responses at the final DCNN layer with a fully connected Conditional Random Field (CRF), which is shown both qualitatively and quantitatively to improve localization performance. Our proposed "DeepLab" system sets the new state-of-art at the PASCAL VOC-2012 semantic image segmentation task, reaching 79.7 percent mIOU in the test set, and advances the results on three other datasets: PASCAL-Context, PASCAL-Person-Part, and Cityscapes. All of our code is made publicly available online.

Robust point matching via vector field consensus

In this paper, we propose an efficient algorithm, called vector field consensus, for establishing robust point correspondences between two sets of points. Our algorithm starts by creating a set of putative correspondences which can contain a very large number of false correspondences, or outliers, in addition to a limited number of true correspondences (inliers). Next, we solve for correspondence by interpolating a vector field between the two point sets, which involves estimating a consensus of inlier points whose matching follows a nonparametric geometrical constraint. We formulate this a maximum a posteriori (MAP) estimation of a Bayesian model with hidden/latent variables indicating whether matches in the putative set are outliers or inliers. We impose nonparametric geometrical constraints on the correspondence, as a prior distribution, using Tikhonov regularizers in a reproducing kernel Hilbert space. MAP estimation is performed by the EM algorithm which by also estimating the variance of the prior model (initialized to a large value) is able to obtain good estimates very quickly (e.g., avoiding many of the local minima inherent in this formulation). We illustrate this method on data sets in 2D and 3D and demonstrate that it is robust to a very large number of outliers (even up to 90%). We also show that in the special case where there is an underlying parametric geometrical model (e.g., the epipolar line constraint) that we obtain better results than standard alternatives like RANSAC if a large number of outliers are present. This suggests a two-stage strategy, where we use our nonparametric model to reduce the size of the putative set and then apply a parametric variant of our approach to estimate the geometric parameters. Our algorithm is computationally efficient and we provide code for others to use it. In addition, our approach is general and can be applied to other problems, such as learning with a badly corrupted training data set.

Coupling Image Restoration and Segmentation: A Generalized Linear Model/Bregman Perspective

We introduce a new class of data-fitting energies that couple image segmentation with image restoration. These functionals model the image intensity using the statistical framework of generalized linear models. By duality, we establish an information-theoretic interpretation using Bregman divergences. We demonstrate how this formulation couples in a principled way image restoration tasks such as denoising, deblurring (deconvolution), and inpainting with segmentation. We present an alternating minimization algorithm to solve the resulting composite photometric/geometric inverse problem. We use Fisher scoring to solve the photometric problem and to provide asymptotic uncertainty estimates. We derive the shape gradient of our data-fitting energy and investigate convex relaxation for the geometric problem. We introduce a new alternating split-Bregman strategy to solve the resulting convex problem and present experiments and comparisons on both synthetic and real-world images.

Robust principal component analysis by self-organizing rules based on statistical physics approach

This paper applies statistical physics to the problem of robust principal component analysis (PCA). The commonly used PCA learning rules are first related to energy functions. These functions are generalized by adding a binary decision field with a given prior distribution so that outliers in the data are dealt with explicitly in order to make PCA robust. Each of the generalized energy functions is then used to define a Gibbs distribution from which a marginal distribution is obtained by summing over the binary decision field. The marginal distribution defines an effective energy function, from which self-organizing rules have been developed for robust PCA. Under the presence of outliers, both the standard PCA methods and the existing self-organizing PCA rules studied in the literature of neural networks perform quite poorly. By contrast, the robust rules proposed here resist outliers well and perform excellently for fulfilling various PCA-like tasks such as obtaining the first principal component vector, the first k principal component vectors, and directly finding the subspace spanned by the first k vector principal component vectors without solving for each vector individually. Comparative experiments have been made, and the results show that the authors' robust rules improve the performances of the existing PCA algorithms significantly when outliers are present.

Recursive segmentation and recognition templates for image parsing

In this paper, we propose a Hierarchical Image Model (HIM) which parses images to perform segmentation and object recognition. The HIM represents the image recursively by segmentation and recognition templates at multiple levels of the hierarchy. This has advantages for representation, inference, and learning. First, the HIM has a coarse-to-fine representation which is capable of capturing long-range dependency and exploiting different levels of contextual information (similar to how natural language models represent sentence structure in terms of hierarchical representations such as verb and noun phrases). Second, the structure of the HIM allows us to design a rapid inference algorithm, based on dynamic programming, which yields the first polynomial time algorithm for image labeling. Third, we learn the HIM efficiently using machine learning methods from a labeled data set. We demonstrate that the HIM is comparable with the state-of-the-art methods by evaluation on the challenging public MSRC and PASCAL VOC 2007 image data sets.

STILL IMAGE SEGMENTATION TOOLS FOR OBJECT-BASED MULTIMEDIA APPLICATIONS

In this paper, a color image segmentation algorithm and an approach to large-format image segmentation are presented, both focused on breaking down images to semantic objects for object-based multimedia applications. The proposed color image segmentation algorithm performs the segmentation in the combined intensity–texture–position feature space in order to produce connected regions that correspond to the real-life objects shown in the image. A preprocessing stage of conditional image filtering and a modified K-Means-with-connectivity-constraint pixel classification algorithm are used to allow for seamless integration of the different pixel features. Unsupervised operation of the segmentation algorithm is enabled by means of an initial clustering procedure. The large-format image segmentation scheme employs the aforementioned segmentation algorithm, providing an elegant framework for the fast segmentation of relatively large images. In this framework, the segmentation algorithm is applied to reduced versions of the original images, in order to speed-up the completion of the segmentation, resulting in a coarse-grained segmentation mask. The final fine-grained segmentation mask is produced with partial reclassification of the pixels of the original image to the already formed regions, using a Bayes classifier. As shown by experimental evaluation, this novel scheme provides fast segmentation with high perceptual segmentation quality.

An edge embedded marker-based watershed algorithm for high spatial resolution remote sensing image segmentation

This correspondence proposes an edge embedded marker-based watershed algorithm for high spatial resolution remote sensing image segmentation. Two improvement techniques are proposed for the two key steps of maker extraction and pixel labeling, respectively, to make it more effective and efficient for high spatial resolution image segmentation. Moreover, the edge information, detected by the edge detector embedded with confidence, is used to direct the two key steps for detecting objects with weak boundary and improving the positional accuracy of the objects boundary. Experiments on different images show that the proposed method has a good generality in producing good segmentation results. It performs well both in retaining the weak boundary and reducing the undesired over-segmentation.

Shape Matching and Registration by Data-driven EM

In this paper, we present an efficient and robust algorithm for shape matching, registration, and detection. The task is to geometrically transform a source shape to fit a target shape. The measure of similarity is defined in terms of the amount of transformation required. The shapes are represented by sparse-point or continuous-contour representations depending on the form of the data. We formulate the problem as probabilistic inference using a generative model and the EM algorithm. But this algorithm has problems with initialization and computing the E-step. To address these problems, we define a discriminative model which makes use of shape features. This gives a hybrid algorithm which combines the generative and discriminative models. The resulting algorithm is very fast, due to the effectiveness of shape-features for solving correspondence requiring typically only four iterations. The convergence time of the algorithm is under a second. We demonstrate the effectiveness of the algorithm by testing it on standard datasets, such as MPEG7, for shape matching and by applying it to a range of matching, registration, and foreground/background segmentation problems.

A generic shape/texture descriptor over multiscale edge field: 2-D walking ant histogram

A novel shape descriptor, which can be extracted from the major object edges automatically and used for the multimedia content-based retrieval in multimedia databases, is presented. By adopting a multiscale approach over the edge field where the scale represents the amount of simplification, the most relevant edge segments, referred to as subsegments, which eventually represent the major object boundaries, are extracted from a scale-map. Similar to the process of a walking ant with a limited line of sight over the boundary of a particular object, we traverse through each subsegment and describe a certain line of sight, whether it is a continuous branch or a corner, using individual 2-D histograms. Furthermore, the proposed method can also be tuned to be an efficient texture descriptor, which achieves a superior performance especially for directional textures. Finally, integrating the whole process as feature extraction module into MUVIS framework allows us to test the mutual performance of the proposed shape descriptor in the context of multimedia indexing and retrieval.

Automatic object extraction over multiscale edge field for multimedia retrieval

In this work, we focus on automatic extraction of object boundaries from Canny edge field for the purpose of content-based indexing and retrieval over image and video databases. A multiscale approach is adopted where each successive scale provides further simplification of the image by removing more details, such as texture and noise, while keeping major edges. At each stage of the simplification, edges are extracted from the image and gathered in a scale-map, over which a perceptual subsegment analysis is performed in order to extract true object boundaries. The analysis is mainly motivated by Gestalt laws and our experimental results suggest a promising performance for main objects extraction, even for images with crowded textural edges and objects with color, texture, and illumination variations. Finally, integrating the whole process as feature extraction module into MUVIS framework allows us to test the mutual performance of the proposed object extraction method and subsequent shape description in the context of multimedia indexing and retrieval. A promising retrieval performance is achieved, and especially in some particular examples, the experimental results show that the proposed method presents such a retrieval performance that cannot be achieved by using other features such as color or texture.

A multiresolution diffused expectation-maximization algorithm for medical image segmentation

In this paper a new method for segmenting medical images is presented, the multiresolution diffused expectation-maximization (MDEM) algorithm. The algorithm operates within a multiscale framework, thus taking advantage of the fact that objects/regions to be segmented usually reside at different scales. At each scale segmentation is carried out via the expectation-maximization algorithm, coupled with anisotropic diffusion on classes, in order to account for the spatial dependencies among pixels. This new approach is validated via experiments on a variety of medical images and its performance is compared with more standard methods.

Optimization via Intermittency with a Self-Organizing Neural Network

One of the major obstacles in using neural networks to solve combinatorial optimization problems is the convergence toward one of the many local minima instead of the global minima. In this letter, we propose a technique that enables a self-organizing neural network to escape from local minima by virtue of the intermittency phenomenon. It gives rise to novel search dynamics that allow the system to visit multiple global minima as meta-stable states. Numerical experiments performed suggest that the phenomenon is a combined effect of Kohonen-type competitive learning and the iterated softmax function operating near bifurcation. The resultant intermittent search exhibits fractal characteristics when the optimization performance is at its peak in the form of 1/f signals in the time evolution of the cost, as well as power law distributions in the meta-stable solution states. The N-Queens problem is used as an example to illustrate the meta-stable convergence process that sequentially generates, in a single run, 92 solutions to the 8-Queens problem and 4024 solutions to the 17-Queens problem.

Higher-order nonlinear priors for surface reconstruction

For surface reconstruction problems with noisy and incomplete range data, a Bayesian estimation approach can improve the overall quality of the surfaces. The Bayesian approach to surface estimation relies on a likelihood term, which ties the surface estimate to the input data, and the prior, which ensures surface smoothness or continuity. This paper introduces a new high-order, nonlinear prior for surface reconstruction. The proposed prior can smooth complex, noisy surfaces, while preserving sharp, geometric features, and it is a natural generalization of edge-preserving methods in image processing, such as anisotropic diffusion. An exact solution would require solving a fourth-order partial differential equation (PDE), which can be difficult with conventional numerical techniques. Our approach is to solve a cascade system of two second-order PDEs, which resembles the original fourth-order system. This strategy is based on the observation that the generalization of image processing to surfaces entails filtering the surface normals. We solve one PDE for processing the normals and one for refitting the surface to the normals. Furthermore, we implement the associated surface deformations using level sets. Hence, the algorithm can accommodate very complex shapes with arbitrary and changing topologies. This paper gives the mathematical formulation and describes the numerical algorithms. We also show results using range and medical data.

Robust anisotropic diffusion

Relations between anisotropic diffusion and robust statistics are described in this paper. Specifically, we show that anisotropic diffusion can be seen as a robust estimation procedure that estimates a piecewise smooth image from a noisy input image. The "edge-stopping" function in the anisotropic diffusion equation is closely related to the error norm and influence function in the robust estimation framework. This connection leads to a new "edge-stopping" function based on Tukey's biweight robust estimator that preserves sharper boundaries than previous formulations and improves the automatic stopping of the diffusion. The robust statistical interpretation also provides a means for detecting the boundaries (edges) between the piecewise smooth regions in an image that has been smoothed with anisotropic diffusion. Additionally, we derive a relationship between anisotropic diffusion and regularization with line processes. Adding constraints on the spatial organization of the line processes allows us to develop new anisotropic diffusion equations that result in a qualitative improvement in the continuity of edges.

Inversion of large-support ill-posed linear operators using a piecewise Gaussian MRF

We propose a method for the reconstruction of signals and images observed partially through a linear operator with a large support (e.g., a Fourier transform on a sparse set). This inverse problem is ill-posed and we resolve it by incorporating the prior information that the reconstructed objects are composed of smooth regions separated by sharp transitions. This feature is modeled by a piecewise Gaussian (PG) Markov random field (MRF), known also as the weak-string in one dimension and the weak-membrane in two dimensions. The reconstruction is defined as the maximum a posteriori estimate. The prerequisite for the use of such a prior is the success of the optimization stage. The posterior energy corresponding to a PG MRF is generally multimodal and its minimization is particularly problematic. In this context, general forms of simulated annealing rapidly become intractable when the observation operator extends over a large support. In this paper, global optimization is dealt with by extending the graduated nonconvexity (GNC) algorithm to ill-posed linear inverse problems. GNC has been pioneered by Blake and Zisserman in the field of image segmentation. The resulting algorithm is mathematically suboptimal but it is seen to be very efficient in practice. We show that the original GNC does not correctly apply to ill-posed problems. Our extension is based on a proper theoretical analysis, which provides further insight into the GNC. The performance of the proposed algorithm is corroborated by a synthetic example in the area of diffraction tomography.

Deformable boundary finding in medical images by integrating gradient and region information

Accurately segmenting and quantifying structures is a key issue in biomedical image analysis. The two conventional methods of image segmentation, region-based segmentation, and boundary finding, often suffer from a variety of limitations. Here the authors propose a method which endeavors to integrate the two approaches in an effort to form a unified approach that is robust to noise and poor initialization. The authors' approach uses Green's theorem to derive the boundary of a homogeneous region-classified area in the image and integrates this with a gray level gradient-based boundary finder. This combines the perceptual notions of edge/shape information with gray level homogeneity. A number of experiments were performed both on synthetic and real medical images of the brain and heart to evaluate the new approach, and it is shown that the integrated method typically performs better when compared to conventional gradient-based deformable boundary finding. Further, this method yields these improvements with little increase in computational overhead, an advantage derived from the application of the Green's theorem.

Temporal coherence theory for the detection and measurement of visual motion

A recent challenge to the completeness of some influential models of local-motion detection has come from experiments in which subjects had to detect a single dot moving along a trajectory amidst noise dots undergoing Brownian motion. We propose and test a new theory of the detection and measurement of visual motion, which can account for these signal-in-Brownian-noise experiments. The theory postulates that the signals from local-motion detectors are made coherent in space and time by a special purpose network, and that this coherence boosts signals of features moving along non-random trajectories over time. Two experiments were performed to estimate parameters and test the theory. These experiments showed that detection is impaired with increasing eccentricity, an effect that varies inversely with step size. They also showed that detection improves over durations extending to at least 600 msec. An implementation of the theory accounts for these psychophysical detection measurements.

A Bayesian framework for understanding texture segmentation in the primary visual cortex

This paper presents a mathematical theory for understanding the computations involved in texture segmentation in the primary visual cortex. We propose that texture segmentation is a part of the early visual system's overall strategy to infer surfaces of objects in a visual scene. Based on this insight, we use the Bayesian inference paradigm to formulate the texture segmentation problem into a maximum a posteriori surface inference problem. The dynamical system for finding the optimal solution of this problem can be characterized by two concurrent and interactive processes: a gradual sharpening of the boundary signals and a simultaneous smoothing of the surface signals. The behavior of these dynamical processes was studied using both analytical and computational methods. We present some computational results and mathematical predictions. This theory suggests a novel framework for understanding the functional roles of the complex cells in the primary visual cortex.

A nonlinear regularization approach to early vision

We propose a new class of approaches to smooth visual data while preserving significant transitions of these data as clues for segmentation. Formally, the given visual data are represented as a noisy (image) function g, and we present a class of continuously formulated global minimization problems to smooth g. The resulting function u can be characterized as the minimizer of a specific nonquadratic functional or, equivalently, as the result of an associated nonlinear diffusion process. Our approach generalizes the well-known quadratic regularization principle while retaining its attractive properties: For any given g, the solution u to the proposed minimization problem is unique and depends continuously on the data g. Furthermore, convergence of approximate solutions obtained by finite element discretization holds true. We show that the nodal variables of any chosen finite element subspace can be interpreted as computational units whose activation dynamics due to the nonlinear smoothing process evolve like a globally asymptotically stable network. A corresponding analogue implementation is thus feasible and would provide a real time processing stage for the transition preserving smoothing of visual data. Using artificial as well as real data we illustrate our approach by numerical examples. We demonstrate that solutions to our approach improve those obtained by quadratic minimization and show the influence of global parameters which allow for a continuous, scale-dependent, and selective control of the smoothing process.