Structure-Sensitive Superpixels via Geodesic Distance

An Efficient Hybrid Linear Clustering Superpixel Decomposition Framework for Traffic Scene Semantic Segmentation

Superpixel decomposition could reconstruct an image through meaningful fragments to extract regional features, thus boosting the performance of advanced computer vision tasks. To further optimize the computational efficiency as well as segmentation quality, a novel framework is proposed to generate superpixels from the perspective of hybridizing two existing linear clustering frameworks. Instead of conventional grid sampling seeds for region clustering, a fast convergence strategy is first introduced to center the final superpixel clusters, which is based on an accelerated convergence strategy. Superpixels are then generated from a center-fixed online average clustering, which adopts region growing to label all pixels in an efficient one-pass manner. The experiments verify that the integration of this two-step implementation could generate a synergistic effect and that it becomes more well-rounded than each single method. Compared with other state-of-the-art superpixel algorithms, the proposed framework achieves a comparable overall performance in terms of segmentation accuracy, spatial compactness and running efficiency; moreover, an application on image segmentation verifies its facilitation for traffic scene analysis.

Fast Generation of Superpixels With Lattice Topology

Serving as an essential step for many applications of image processing, superpixel generation has attracted a lot of attentions. Most existing superpixel generation algorithms focus on the boundary adherence and compactness of the superpixels, but ignore the topological consistency between the superpixels, which severely limites their applications in the subsequent tasks, especially in the CNN based image processing tasks. In this paper, we present a fast lattice superpixel generation algorithm, which can generate superpixels with lattice topology like the original pixels. We also propose a local similarity loss function to improve the segmentation accuracy of the generated lattice superpixels. The whole algorithm is parallelly implemented on GPU. We perform extensive experiments on three datasets (i.e., BSDS500, NYUv2 and VOC) to verify the efficacy of our algorithm. The experimental results show that our method achieves competitive results compared to the state-of-the-art methods.

Content-Aware SLIC Super-Pixels for Semi-Dark Images (SLIC++)

Super-pixels represent perceptually similar visual feature vectors of the image. Super-pixels are the meaningful group of pixels of the image, bunched together based on the color and proximity of singular pixel. Computation of super-pixels is highly affected in terms of accuracy if the image has high pixel intensities, i.e., a semi-dark image is observed. For computation of super-pixels, a widely used method is SLIC (Simple Linear Iterative Clustering), due to its simplistic approach. The SLIC is considerably faster than other state-of-the-art methods. However, it lacks in functionality to retain the content-aware information of the image due to constrained underlying clustering technique. Moreover, the efficiency of SLIC on semi-dark images is lower than bright images. We extend the functionality of SLIC to several computational distance measures to identify potential substitutes resulting in regular and accurate image segments. We propose a novel SLIC extension, namely, SLIC++ based on hybrid distance measure to retain content-aware information (lacking in SLIC). This makes SLIC++ more efficient than SLIC. The proposed SLIC++ does not only hold efficiency for normal images but also for semi-dark images. The hybrid content-aware distance measure effectively integrates the Euclidean super-pixel calculation features with Geodesic distance calculations to retain the angular movements of the components present in the visual image exclusively targeting semi-dark images. The proposed method is quantitively and qualitatively analyzed using the Berkeley dataset. We not only visually illustrate the benchmarking results, but also report on the associated accuracies against the ground-truth image segments in terms of boundary precision. SLIC++ attains high accuracy and creates content-aware super-pixels even if the images are semi-dark in nature. Our findings show that SLIC++ achieves precision of 39.7%, outperforming the precision of SLIC by a substantial margin of up to 8.1%.

Infrared Image Superpixel Segmentation Based on Seed Strategy of Contour Encoding

Superpixel segmentation has become a crucial pre-processing tool to reduce computation in many computer vision applications. In this paper, a superpixel extraction algorithm based on a seed strategy of contour encoding (SSCE) for infrared images is presented, which can generate superpixels with high boundary adherence and compactness. Specifically, SSCE can solve the problem of superpixels being unable to self-adapt to the image content. First, a contour encoding map is obtained by ray scanning the binary edge map, which ensures that each connected domain belongs to the same homogeneous region. Second, according to the seed sampling strategy, each seed point can be extracted from the contour encoding map. The initial seed set, which is adaptively scattered based on the local structure, is capable of improving the capability of boundary adherence, especially for small regions. Finally, the initial superpixels limited by the image contour are generated by clustering and refined by merging similar adjacent superpixels in the region adjacency graph (RAG) to reduce redundant superpixels. Experimental results on a self-built infrared dataset and the public datasets BSD500 and 3Dircadb demonstrate the generalization ability in grayscale and medical images, and the superiority of the proposed method over several state-of-the-art methods in terms of accuracy and compactness.

GPU-Based Supervoxel Generation With a Novel Anisotropic Metric

Video over-segmentation into supervoxels is an important pre-processing technique for many computer vision tasks. Videos are an order of magnitude larger than images. Most existing methods for generating supervovels are either memory- or time-inefficient, which limits their application in subsequent video processing tasks. In this paper, we present an anisotropic supervoxel method, which is memory-efficient and can be executed on the graphics processing unit (GPU). Therefore, our algorithm achieves good balance among segmentation quality, memory usage and processing time. In order to provide accurate segmentation for moving objects in video, we use the optical flow information to design a brand new non-Euclidean metric to calculate the anisotropic distances between seeds and voxels. To efficiently compute the anisotropic metric, we adjust the classic jump flooding algorithm (which is designed for parallel execution on the GPU) to generate anisotropic Voronoi tessellation in the combined color and spatio-temporal space. We evaluate our method and the representative supervoxel algorithms for their capability on segmentation performance, computation speed and memory efficiency. We also apply supervoxel results to the application of foreground propagation in videos to test the performance on solving practical problems. Experiments show that our algorithm is much faster than the existing methods, and achieves good balance on segmentation quality and efficiency.

Feature-Aware Uniform Tessellations on Video Manifold for Content-Sensitive Supervoxels

Over-segmenting a video into supervoxels has strong potential to reduce the complexity of downstream computer vision applications. Content-sensitive supervoxels (CSSs) are typically smaller in content-dense regions (i.e., with high variation of appearance and/or motion) and larger in content-sparse regions. In this paper, we propose to compute feature-aware CSSs (FCSSs) that are regularly shaped 3D primitive volumes well aligned with local object/region/motion boundaries in video. To compute FCSSs, we map a video to a 3D manifold embedded in a combined color and spatiotemporal space, in which the volume elements of video manifold give a good measure of the video content density. Then any uniform tessellation on video manifold can induce CSS in the video. Our idea is that among all possible uniform tessellations on the video manifold, FCSS finds one whose cell boundaries well align with local video boundaries. To achieve this goal, we propose a novel restricted centroidal Voronoi tessellation method that simultaneously minimizes the tessellation energy (leading to uniform cells in the tessellation) and maximizes the average boundary distance (leading to good local feature alignment). Theoretically our method has an optimal competitive ratio O(1), and its time and space complexities are O(NK) and O(N+K) for computing K supervoxels in an N-voxel video. We also present a simple extension of FCSS to streaming FCSS for processing long videos that cannot be loaded into main memory at once. We evaluate FCSS, streaming FCSS and ten representative supervoxel methods on four video datasets and two novel video applications. The results show that our method simultaneously achieves state-of-the-art performance with respect to various evaluation criteria.

Rooted Spanning Superpixels

This paper proposes a new approach for superpixel segmentation. It is formulated as finding a rooted spanning forest of a graph with respect to some roots and a path-cost function. The underlying graph represents an image, the roots serve as seeds for segmentation, each pixel is connected to one seed via a path, the path-cost function measures both the color similarity and spatial closeness between two pixels via a path, and each tree in the spanning forest represents one superpixel. Originating from the evenly distributed seeds, the superpixels are guided by a path-cost function to grow uniformly and adaptively, the pixel-by-pixel growing continues until they cover the whole image. The number of superpixels is controlled by the number of seeds. The connectivity is maintained by region growing. Good performances are assured by connecting each pixel to the similar seed, which are dominated by the path-cost function. It is evaluated by both the superpixel benchmark and supervoxel benchmark. Its performance is ranked as the second among top performing state-of-the-art methods. Moreover, it is much faster than the other superpixel and supervoxel methods.

Content-Sensitive Superpixel Generation with Boundary Adjustment

Superpixel segmentation has become a crucial tool in many image processing and computer vision applications. In this paper, a novel content-sensitive superpixel generation algorithm with boundary adjustment is proposed. First, the image local entropy was used to measure the amount of information in the image, and the amount of information was evenly distributed to each seed. It placed more seeds to achieve the lower under-segmentation in content-dense regions, and placed the fewer seeds to increase computational efficiency in content-sparse regions. Second, the Prim algorithm was adopted to generate uniform superpixels efficiently. Third, a boundary adjustment strategy with the adaptive distance further optimized the superpixels to improve the performance of the superpixel. Experimental results on the Berkeley Segmentation Database show that our method outperforms competing methods under evaluation metrics.

An Iterative Spanning Forest Framework for Superpixel Segmentation

Superpixel segmentation has emerged as an important research problem in the areas of image processing and computer vision. In this paper, we propose a framework, namely Iterative Spanning Forest (ISF), in which improved sets of connected superpixels (supervoxels in 3D) can be generated by a sequence of image foresting transforms. In this framework, one can choose the most suitable combination of ISF components for a given application-i.e., 1) a seed sampling strategy; 2) a connectivity function; 3) an adjacency relation; and 4) a seed pixel recomputation procedure. The superpixels in ISF structurally correspond to spanning trees rooted at those seeds. We present five ISF-based methods to illustrate different choices for those components. These methods are compared with a number of state-of-the-art approaches with respect to effectiveness and efficiency. Experiments are carried out on several datasets containing 2D and 3D objects with distinct texture and shape properties, including a high-level application, named sky image segmentation. The theoretical properties of ISF are demonstrated in the supplementary material and the results show ISF-based methods rank consistently among the best for all datasets.

A Two-Stage Gradient Ascent-Based Superpixel Framework for Adaptive Segmentation

Superpixel segmentation usually over-segments an image into fragments to extract regional features, thus linking up advanced computer vision tasks. In this work, a novel coarse-to-fine gradient ascent framework is proposed for superpixel-based color image adaptive segmentation. In the first stage, a speeded-up Simple Linear Iterative Clustering (sSLIC) method is adopted to generate uniform superpixels efficiently, which assumes that homogeneous regions preserve high consistence during clustering, consequently, much redundant computation for updating can be avoided. Then a simple criterion is introduced to evaluate the uniformity in each superpixel region, once a superpixel region is under-segmented, an adaptive marker-controlled watershed algorithm processes a finer subdivision. Experimental results show that the framework achieves better performance on detail-rich regions than previous superpixel approaches with satisfactory efficiency.

An overview of state-of-the-art image restoration in electron microscopy

In Life Science research, electron microscopy (EM) is an essential tool for morphological analysis at the subcellular level as it allows for visualization at nanometer resolution. However, electron micrographs contain image degradations such as noise and blur caused by electromagnetic interference, electron counting errors, magnetic lens imperfections, electron diffraction, etc. These imperfections in raw image quality are inevitable and hamper subsequent image analysis and visualization. In an effort to mitigate these artefacts, many electron microscopy image restoration algorithms have been proposed in the last years. Most of these methods rely on generic assumptions on the image or degradations and are therefore outperformed by advanced methods that are based on more accurate models. Ideally, a method will accurately model the specific degradations that fit the physical acquisition settings. In this overview paper, we discuss different electron microscopy image degradation solutions and demonstrate that dedicated artefact regularisation results in higher quality restoration and is applicable through recently developed probabilistic methods.

A topo-graph model for indistinct target boundary definition from anatomical images

BACKGROUND AND OBJECTIVE

It can be challenging to delineate the target object in anatomical imaging when the object boundaries are difficult to discern due to the low contrast or overlapping intensity distributions from adjacent tissues.

METHODS

We propose a topo-graph model to address this issue. The first step is to extract a topographic representation that reflects multiple levels of topographic information in an input image. We then define two types of node connections - nesting branches (NBs) and geodesic edges (GEs). NBs connect nodes corresponding to initial topographic regions and GEs link the nodes at a detailed level. The weights for NBs are defined to measure the similarity of regional appearance, and weights for GEs are defined with geodesic and local constraints. NBs contribute to the separation of topographic regions and the GEs assist the delineation of uncertain boundaries. Final segmentation is achieved by calculating the relevance of the unlabeled nodes to the labels by the optimization of a graph-based energy function. We test our model on 47 low contrast CT studies of patients with non-small cell lung cancer (NSCLC), 10 contrast-enhanced CT liver cases and 50 breast and abdominal ultrasound images. The validation criteria are the Dice's similarity coefficient and the Hausdorff distance.

RESULTS

Student's t-test show that our model outperformed the graph models with pixel-only, pixel and regional, neighboring and radial connections (p-values <0.05).

CONCLUSIONS

Our findings show that the topographic representation and topo-graph model provides improved delineation and separation of objects from adjacent tissues compared to the tested models.

Superpixel Segmentation Using Gaussian Mixture Model

Superpixel segmentation partitions an image into perceptually coherent segments of similar size, namely, superpixels. It is becoming a fundamental preprocessing step for various computer vision tasks because superpixels significantly reduce the number of inputs and provide a meaningful representation for feature extraction. We present a pixel-related Gaussian mixture model (GMM) to segment images into superpixels. GMM is a weighted sum of Gaussian functions, each one corresponding to a superpixel, to describe the density of each pixel represented by a random variable. Different from previously proposed GMMs, our weights are constant, and Gaussian functions in the sums are subsets of all the Gaussian functions, resulting in segments of similar size and an algorithm of linear complexity with respect to the number of pixels. In addition to the linear complexity, our algorithm is inherently parallel and allows fast execution on multicore systems. During the expectation-maximization iterations of estimating the unknown parameters in the Gaussian functions, we impose two lower bounds to truncate the eigenvalues of the covariance matrices, which enables the proposed algorithm to control the regularity of superpixels. Experiments on a wellknown segmentation dataset show that our method can efficiently produce superpixels that adhere to object boundaries better than the current state-of-the-art methods.

Occlusion-aware Method for Temporally Consistent Superpixels

A wide variety of computer vision applications rely on superpixel or supervoxel algorithms as a preprocessing step. This underlines the overall importance that these approaches have gained in recent years. However, most methods show a lack of temporal consistency or fail in producing temporally stable superpixels. In this paper, we present an approach to generate temporally consistent superpixels for video content. Our method is formulated as a contour-evolving expectation-maximization framework, which utilizes an efficient label propagation scheme to encourage the preservation of superpixel shapes and their relative positioning over time. By explicitly detecting the occlusion of superpixels and the disocclusion of new image regions, our framework is able to terminate and create superpixels whose corresponding image region becomes hidden or newly appears. Additionally, the occluded parts of superpixels are incorporated in the further optimization. This increases the compliance of the superpixel flow with the optical flow present in the scene. Using established benchmark suites, we show the performance of our approach in comparison to state-of-the-art supervoxel and superpixel algorithms for video content.

Intrinsic Manifold SLIC: A Simple and Efficient Method for Computing Content-Sensitive Superpixels

Superpixels are perceptually meaningful atomic regions that can effectively capture image features. Among various methods for computing uniform superpixels, simple linear iterative clustering (SLIC) is popular due to its simplicity and high performance. In this paper, we extend SLIC to compute content-sensitive superpixels, i.e., small superpixels in content-dense regions with high intensity or colour variation and large superpixels in content-sparse regions. Rather than using the conventional SLIC method that clusters pixels in , we map the input image to a 2-dimensional manifold , whose area elements are a good measure of the content density in . We propose a simple method, called intrinsic manifold SLIC (IMSLIC), for computing a geodesic centroidal Voronoi tessellation (GCVT)-a uniform tessellation-on , which induces the content-sensitive superpixels in . In contrast to the existing algorithms, IMSLIC characterizes the content sensitivity by measuring areas of Voronoi cells on . Using a simple and fast approximation to a closed-form solution, the method can compute the GCVT at a very low cost and guarantees that all Voronoi cells are simply connected. We thoroughly evaluate IMSLIC and compare it with eleven representative methods on the BSDS500 dataset and seven representative methods on the NYUV2 dataset. Computational results show that IMSLIC outperforms existing methods in terms of commonly used quality measures pertaining to superpixels such as compactness, adherence to boundaries, and achievable segmentation accuracy. We also evaluate IMSLIC and seven representative methods in an image contour closure application, and the results on two datasets, WHD and WSD, show that IMSLIC achieves the best foreground segmentation performance.

Superpixels of RGB-D Images for Indoor Scenes Based on Weighted Geodesic Driven Metric

Serving as a key step for applications of image processing, superpixel generation has been attracting increasing attention. RGB-D images are used pervasively in scenes reconstruction and representation, benefiting from their contained depth data. In this paper, we present a novel framework for generating superpixels focus on RGB-D images of indoor scenes, based on a weighted geodesic driven metric that combines both color and geometric information. In particular, taking into account the unique structures of indoor scenarios, we first denoise the given RGB-D image, and construct the corresponding triangular mesh. A new weighted geodesic driven metric is defined by introducing a weight function constrained with normal vectors and colors. Under this metric, an energy function is defined to measure our over-segmentation of the triangular mesh, by optimizing which, we can acquire an optimal over-segmentation of the triangular mesh with object boundaries respected, such that vertices in each sub-region have similar geometric structures and color intensities. Re-mapping the over-segmentation of the triangular mesh to the RGB-D image results in desired superpixels. We perform extensive experiments on a large-scale database of RGB-D images to verify the efficacy of our algorithm. The results show that our algorithm has considerable advantages over the existing state-of-the-art methods.

Improved Recursive Geodesic Distance Computation for Edge Preserving Filter

All known recursive filters based on the geodesic distance affinity are realized by two 1D recursions applied in two orthogonal directions of the image plane. The 2D extension of the filter is not valid and has theoretically drawbacks, which lead to known artifacts. In this paper, a maximum influence propagation method is proposed to approximate the 2D extension for the geodesic distance-based recursive filter. The method allows to partially overcome the drawbacks of the 1D recursion approach. We show that our improved recursion better approximates the true geodesic distance filter, and the application of this improved filter for image denoising outperforms the existing recursive implementation of the geodesic distance. As an application, we consider a geodesic distance-based filter for image denoising. Experimental evaluation of our denoising method demonstrates comparable and for several test images better results, than state-of-the-art approaches, while our algorithm is considerably faster with computational complexity O(8P).

Multifocus image fusion using superpixel segmentation and superpixel-based mean filtering

To achieve better performance in multifocus image fusion problems, a new regional approach based on superpixels and superpixel-based mean filtering is proposed in this paper. First, a fast and effective segmentation method is adopted to generate the superpixels over a clarity-enhanced average image. By averaging the clarity information in each superpixel, we make the initial decision map of fusion by regionally selecting sharper superpixels in different source images. Then a novel superpixel-based mean filtering technique is introduced to make full use of spatial consistency in images and the final post-processed decision map is produced. The fused image is constructed by selecting pixels from different source images according to the final decision map. Experimental results demonstrate the proposed method's competitive performance in comparison with state-of-the-art multifocus image fusion approaches.

Human Action Recognition Based on Multi-View Regularized Extreme Learning Machine

In this paper, we employ multiple Single-hidden Layer Feedforward Neural Networks for multi-view action recognition. We propose an extension of the Extreme Learning Machine algorithm that is able to exploit multiple action representations and scatter information in the corresponding ELM spaces for the calculation of the networks’ parameters and the determination of optimized network combination weights. The proposed algorithm is evaluated by using two state-of-the-art action video representation approaches on five publicly available action recognition databases designed for different application scenarios. Experimental comparison of the proposed approach with three commonly used video representation combination approaches and relating classification schemes illustrates that ELM networks employing a supervised view combination scheme generally outperform those exploiting unsupervised combination approaches, as well as that the exploitation of scatter information in ELM-based neural network training enhances the network’s performance.

Value-driven attentional priority signals in human basal ganglia and visual cortex

Goal-directed and stimulus-driven factors determine attentional priority through a well defined dorsal frontal-parietal and ventral temporal-parietal network of brain regions, respectively. Recent evidence demonstrates that reward-related stimuli also have high attentional priority, independent of their physical salience and goal-relevance. The neural mechanisms underlying such value-driven attentional control are unknown. Using human functional magnetic resonance imaging, we demonstrate that the tail of the caudate nucleus and extrastriate visual cortex respond preferentially to task-irrelevant but previously reward-associated objects, providing an attentional priority signal that is sensitive to reward history. The caudate tail has not been implicated in the control of goal-directed or stimulus-driven attention, but is well suited to mediate the value-driven control of attention. Our findings reveal the neural basis of value-based attentional priority.

Face super-resolution via multilayer locality-constrained iterative neighbor embedding and intermediate dictionary learning

Based on the assumption that low-resolution (LR) and high-resolution (HR) manifolds are locally isometric, the neighbor embedding super-resolution algorithms try to preserve the geometry (reconstruction weights) of the LR space for the reconstructed HR space, but neglect the geometry of the original HR space. Due to the degradation process of the LR image (e.g., noisy, blurred, and down-sampled), the neighborhood relationship of the LR space cannot reflect the truth. To this end, this paper proposes a coarse-to-fine face super-resolution approach via a multilayer locality-constrained iterative neighbor embedding technique, which intends to represent the input LR patch while preserving the geometry of original HR space. In particular, we iteratively update the LR patch representation and the estimated HR patch, and meanwhile an intermediate dictionary learning scheme is employed to bridge the LR manifold and original HR manifold. The proposed method can faithfully capture the intrinsic image degradation shift and enhance the consistency between the reconstructed HR manifold and the original HR manifold. Experiments with application to face super-resolution on the CAS-PEAL-R1 database and real-world images demonstrate the power of the proposed algorithm.