Multi-class constrained normalized cut with hard, soft, unary and pairwise priors and its applications to object segmentation

Normalized cut is a powerful method for image segmentation as well as data clustering. However, it does not perform well in challenging segmentation problems, such as segmenting objects in a complex background. Researchers have attempted to incorporate priors or constraints to handle such cases. Available priors in image segmentation problems may be hard or soft, unary or pairwise, but only hard must-link constraints and two-class settings are well studied. The main difficulties may lie in the following aspects: 1) the nontransitive nature of cannot-link constraints makes it hard to use such constraints in multi-class settings and 2) in multi-class or pairwise settings, the output labels have inconsistent representations with given priors, making soft priors difficult to use. In this paper, we propose novel algorithms, which can handle both hard and soft, both unary and pairwise priors in multi-class settings and provide closed form and efficient solutions. We also apply the proposed algorithms to the problem of object segmentation, producing good results by further introducing a spatial regularity term. Experiments show that the proposed algorithms outperform the state-of-the-art algorithms significantly in clustering accuracy. Other merits of the proposed algorithms are also demonstrated.

Shortest-path constraints for 3D multiobject semiautomatic segmentation via clustering and Graph Cut

We derive shortest-path constraints from graph models of structure adjacency relations and introduce them in a joint centroidal Voronoi image clustering and Graph Cut multiobject semiautomatic segmentation framework. The vicinity prior model thus defined is a piecewise-constant model incurring multiple levels of penalization capturing the spatial configuration of structures in multiobject segmentation. Qualitative and quantitative analyses and comparison with a Potts prior-based approach and our previous contribution on synthetic, simulated, and real medical images show that the vicinity prior allows for the correct segmentation of distinct structures having identical intensity profiles and improves the precision of segmentation boundary placement while being fairly robust to clustering resolution. The clustering approach we take to simplify images prior to segmentation strikes a good balance between boundary adaptivity and cluster compactness criteria furthermore allowing to control the trade-off. Compared with a direct application of segmentation on voxels, the clustering step improves the overall runtime and memory footprint of the segmentation process up to an order of magnitude without compromising the quality of the result.

Accuracy of surface tension measurement from drop shapes: the role of image analysis

Axisymmetric Drop Shape Analysis (ADSA) has been extensively used for surface tension measurement. In essence, ADSA works by matching a theoretical profile of the drop to the extracted experimental profile, taking surface tension as an adjustable parameter. Of the three main building blocks of ADSA, i.e. edge detection, the numerical integration of the Laplace equation for generating theoretical curves and the optimization procedure, only edge detection (that extracts the drop profile line from the drop image) needs extensive study. For the purpose of this article, the numerical integration of the Laplace equation for generating theoretical curves and the optimization procedure will only require a minor effort. It is the aim of this paper to investigate how far the surface tension accuracy of drop shape techniques can be pushed by fine tuning and optimizing edge detection strategies for a given drop image. Two different aspects of edge detection are pursued here: sub-pixel resolution and pixel resolution. The effect of two sub-pixel resolution strategies, i.e. spline and sigmoid, on the accuracy of surface tension measurement is investigated. It is found that the number of pixel points in the fitting procedure of the sub-pixel resolution techniques is crucial, and its value should be determined based on the contrast of the image, i.e. the gray level difference between the drop and the background. On the pixel resolution side, two suitable and reliable edge detectors, i.e. Canny and SUSAN, are explored, and the effect of user-specified parameters of the edge detector on the accuracy of surface tension measurement is scrutinized. Based on the contrast of the image, an optimum value of the user-specified parameter of the edge detector, SUSAN, is suggested. Overall, an accuracy of 0.01mJ/m(2) is achievable for the surface tension determination by careful fine tuning of edge detection algorithms.

A Gaussian process guided particle filter for tracking 3D human pose in video

In this paper, we propose a hybrid method that combines Gaussian process learning, a particle filter, and annealing to track the 3D pose of a human subject in video sequences. Our approach, which we refer to as annealed Gaussian process guided particle filter, comprises two steps. In the training step, we use a supervised learning method to train a Gaussian process regressor that takes the silhouette descriptor as an input and produces multiple output poses modeled by a mixture of Gaussian distributions. In the tracking step, the output pose distributions from the Gaussian process regression are combined with the annealed particle filter to track the 3D pose in each frame of the video sequence. Our experiments show that the proposed method does not require initialization and does not lose tracking of the pose. We compare our approach with a standard annealed particle filter using the HumanEva-I dataset and with other state of the art approaches using the HumanEva-II dataset. The evaluation results show that our approach can successfully track the 3D human pose over long video sequences and give more accurate pose tracking results than the annealed particle filter.

Interactive medical image segmentation using PDE control of active contours

Segmentation of injured or unusual anatomic structures in medical imagery is a problem that has continued to elude fully automated solutions. In this paper, the goal of easy-to-use and consistent interactive segmentation is transformed into a control synthesis problem. A nominal level set partial differential equation (PDE) is assumed to be given; this open-loop system achieves correct segmentation under ideal conditions, but does not agree with a human expert's ideal boundary for real image data. Perturbing the state and dynamics of a level set PDE via the accumulated user input and an observer-like system leads to desirable closed-loop behavior. The input structure is designed such that a user can stabilize the boundary in some desired state without needing to understand any mathematical parameters. Effectiveness of the technique is illustrated with applications to the challenging segmentations of a patellar tendon in magnetic resonance and a shattered femur in computed tomography.

Efficient Video Stream Monitoring for Near-Duplicate Detection and Localization in a Large-Scale Repository

In this article, we study the efficiency problem of video stream near-duplicate monitoring in a large-scale repository. Existing stream monitoring methods are mainly designed for a short video to scan over a query stream; they have difficulty being scalable for a large number of long videos. We present a simple but effective algorithm called incremental similarity update to address the problem. That is, a similarity upper bound between two videos can be calculated incrementally by leveraging the prior knowledge of the previous calculation. The similarity upper bound takes a lightweight computation to filter out unnecessary time-consuming computation for the actual similarity between two videos, making the search process more efficient. We integrate the algorithm with inverted indexing to obtain a candidate list from the repository for the given query stream. Meanwhile, the algorithm is applied to scan each candidate for locating exact near-duplicate subsequences. We implement several state-of-the-art methods for comparison in terms of accuracy, execution time, and memory consumption. Experimental results demonstrate the proposed algorithm yields comparable accuracy, compact memory size, and more efficient execution time.

Water reflection recognition based on motion blur invariant moments in curvelet space

Water reflection, a typical imperfect reflection symmetry problem, plays an important role in image content analysis. Existing techniques of symmetry recognition, however, cannot recognize water reflection images correctly because of the complex and various distortions caused by the water wave. Hence, we propose a novel water reflection recognition technique to solve the problem. First, we construct a novel feature space composed of motion blur invariant moments in low-frequency curvelet space and of curvelet coefficients in high-frequency curvelet space. Second, we propose an efficient algorithm including two sub-algorithms: low-frequency reflection cost minimization and high-frequency curvelet coefficients discrimination to classify water reflection images and to determine the reflection axis. Through experimenting on authentic images in a series of tasks, the proposed techniques prove effective and reliable in classifying water reflection images and detecting the reflection axis, as well as in retrieving images with water reflection.

Incorporating patch subspace model in Mumford-Shah type active contours

In this paper, we propose a unified energy minimization model for segmentation of non-smooth image structures, e.g., textures, based on Mumford-Shah functional and linear patch model. We consider that image patches of a non-smooth image structure can be modeled by a patch subspace, and image patches of different non-smooth image structures belong to different patch subspaces, which leads to a computational framework for segmentation of non-smooth image structures. Motivated by the Mumford-Shah model, we show that this segmentation framework is equivalent to minimizing a piecewise linear patch reconstruction energy. We also prove that the error of segmentation is bounded by the error of the linear patch reconstruction, meaning that improving the linear patch reconstruction for each region leads to reduction of the segmentation error. In addition, we derive an algorithm for the linear patch reconstruction with proven global optimality and linear rate of convergence. The segmentation in our method is achieved by minimizing a single energy functional without requiring predefined features. Hence, compared with the previous methods that require predefined texture features, our method can be more suitable for handling general textures in unsupervised segmentation. As a by-product, our method also produces a dictionary of optimized orthonormal descriptors for each segmented region. We mainly evaluate our method on the Brodatz textures. The experiments validate our theoretical claims and show the clear superior performance of our methods over other related methods for segmentation of the textures.

Efficient halftoning based on multiple look-up tables

Look-up table (LUT) halftoning is an efficient way to construct halftone images and approximately simulate the dot distribution of the learned halftone image set. In this paper, a general mechanism named multiple look-up table (MLUT) halftoning is proposed to generate the halftones of direct binary search (DBS), whereas the high efficient characteristic of the LUT is still preserved. In the MLUT, the standard deviation is adopted as an important feature to classify various tables. In addition, the proposed quick standard deviation evaluation is employed to yield an extremely low computational complexity in calculating the standard deviation. In the parameter optimization, the autocorrelation is adopted because it can fully characterize the periodicity of dot distribution. Experimental results demonstrate that the dot distribution generated by the proposed method approximates to that of the DBS, which enables the proposed scheme as a very competitive candidate in the copying and printing industry.

Augmented active surface model for the recovery of small structures in CT

This paper devises an augmented active surface model for the recovery of small structures in a low resolution and high noise setting, where the role of regularization is especially important. The emphasis here is on evaluating performance using real clinical computed tomography (CT) data with comparisons made to an objective ground truth acquired using micro-CT. In this paper, we show that the application of conventional active contour methods to small objects leads to non-optimal results because of the inherent properties of the energy terms and their interactions with one another. We show that the blind use of a gradient magnitude based energy performs poorly at these object scales and that the point spread function (PSF) is a critical factor that needs to be accounted for. We propose a new model that augments the external energy with prior knowledge by incorporating the PSF and the assumption of reasonably constant underlying CT numbers.

Detailed 3D representations for object recognition and modeling

Geometric 3D reasoning at the level of objects has received renewed attention recently in the context of visual scene understanding. The level of geometric detail, however, is typically limited to qualitative representations or coarse boxes. This is linked to the fact that today's object class detectors are tuned toward robust 2D matching rather than accurate 3D geometry, encouraged by bounding-box-based benchmarks such as Pascal VOC. In this paper, we revisit ideas from the early days of computer vision, namely, detailed, 3D geometric object class representations for recognition. These representations can recover geometrically far more accurate object hypotheses than just bounding boxes, including continuous estimates of object pose and 3D wireframes with relative 3D positions of object parts. In combination with robust techniques for shape description and inference, we outperform state-of-the-art results in monocular 3D pose estimation. In a series of experiments, we analyze our approach in detail and demonstrate novel applications enabled by such an object class representation, such as fine-grained categorization of cars and bicycles, according to their 3D geometry, and ultrawide baseline matching.

Sparse subspace clustering: algorithm, theory, and applications

Many real-world problems deal with collections of high-dimensional data, such as images, videos, text, and web documents, DNA microarray data, and more. Often, such high-dimensional data lie close to low-dimensional structures corresponding to several classes or categories to which the data belong. In this paper, we propose and study an algorithm, called sparse subspace clustering, to cluster data points that lie in a union of low-dimensional subspaces. The key idea is that, among the infinitely many possible representations of a data point in terms of other points, a sparse representation corresponds to selecting a few points from the same subspace. This motivates solving a sparse optimization program whose solution is used in a spectral clustering framework to infer the clustering of the data into subspaces. Since solving the sparse optimization program is in general NP-hard, we consider a convex relaxation and show that, under appropriate conditions on the arrangement of the subspaces and the distribution of the data, the proposed minimization program succeeds in recovering the desired sparse representations. The proposed algorithm is efficient and can handle data points near the intersections of subspaces. Another key advantage of the proposed algorithm with respect to the state of the art is that it can deal directly with data nuisances, such as noise, sparse outlying entries, and missing entries, by incorporating the model of the data into the sparse optimization program. We demonstrate the effectiveness of the proposed algorithm through experiments on synthetic data as well as the two real-world problems of motion segmentation and face clustering.

Mesenteric vasculature-guided small bowel segmentation on 3-D CT

Due to its importance and possible applications in visualization, tumor detection and preoperative planning, automatic small bowel segmentation is essential for computer-aided diagnosis of small bowel pathology. However, segmenting the small bowel directly on computed tomography (CT) scans is very difficult because of the low image contrast on CT scans and high tortuosity of the small bowel and its close proximity to other abdominal organs. Motivated by the intensity characteristics of abdominal CT images, the anatomic relationship between the mesenteric vasculature and the small bowel, and potential usefulness of the mesenteric vasculature for establishing the path of the small bowel, we propose a novel mesenteric vasculature map-guided method for small bowel segmentation on high-resolution CT angiography scans. The major mesenteric arteries are first segmented using a vessel tracing method based on multi-linear subspace vessel model and Bayesian inference. Second, multi-view, multi-scale vesselness enhancement filters are used to segment small vessels, and vessels directly or indirectly connecting to the superior mesenteric artery are classified as mesenteric vessels. Third, a mesenteric vasculature map is built by linking vessel bifurcation points, and the small bowel is segmented by employing the mesenteric vessel map and fuzzy connectness. The method was evaluated on 11 abdominal CT scans of patients suspected of having carcinoid tumors with manually labeled reference standard. The result, 82.5% volume overlap accuracy compared with the reference standard, shows it is feasible to segment the small bowel on CT scans using the mesenteric vasculature as a roadmap.

A Bayesian hierarchical factorization model for vector fields

Factorization-based techniques explain arrays of observations using a relatively small number of factors and provide an essential arsenal for multi-dimensional data analysis. Most factorization models are, however, developed on general arrays of scalar values. For a class of practical data arising from observing spatial signals including images, it is desirable for a model to consider general observations, e.g., handling a vector field and non-exchangeable factors, e.g., handling spatial connections between the columns and the rows of the data. In this paper, a probabilistic model for factorization is proposed. We adopt Bayesian hierarchical modeling and treat the factors as latent random variables. A Markov structure is imposed on the distribution of factors to account for the spatial connections. The model is designed to represent vector arrays sampled from fields of continuous domains. Therefore, a tailored observation model is developed to represent the link between the factor product and the data. The proposed technique has been shown effective in analyzing optical flow fields computed on both synthetic images and real-life videoclips.

SuperMatching: feature matching using supersymmetric geometric constraints

Feature matching is a challenging problem at the heart of numerous computer graphics and computer vision applications. We present the SuperMatching algorithm for finding correspondences between two sets of features. It does so by considering triples or higher order tuples of points, going beyond the pointwise and pairwise approaches typically used. SuperMatching is formulated using a supersymmetric tensor representing an affinity metric that takes into account feature similarity and geometric constraints between features: Feature matching is cast as a higher order graph matching problem. SuperMatching takes advantage of supersymmetry to devise an efficient sampling strategy to estimate the affinity tensor, as well as to store the estimated tensor compactly. Matching is performed by an efficient higher order power iteration approach that takes advantage of this compact representation. Experiments on both synthetic and real data show that SuperMatching provides more accurate feature matching than other state-of-the-art approaches for a wide range of 2D and 3D features, with competitive computational cost.

Semi-supervised video segmentation using tree structured graphical models

We present a novel patch-based probabilistic graphical model for semi-supervised video segmentation. At the heart of our model is a temporal tree structure that links patches in adjacent frames through the video sequence. This permits exact inference of pixel labels without resorting to traditional short time window-based video processing or instantaneous decision making. The input to our algorithm is labeled key frame(s) of a video sequence and the output is pixel-wise labels along with their confidences. We propose an efficient inference scheme that performs exact inference over the temporal tree, and optionally a per frame label smoothing step using loopy BP, to estimate pixel-wise labels and their posteriors. These posteriors are used to learn pixel unaries by training a Random Decision Forest in a semi-supervised manner. These unaries are used in a second iteration of label inference to improve the segmentation quality. We demonstrate the efficacy of our proposed algorithm using several qualitative and quantitative tests on both foreground/background and multiclass video segmentation problems using publicly available and our own datasets.

Recognising safety critical events: can automatic video processing improve naturalistic data analyses?

New trends in research on traffic accidents include Naturalistic Driving Studies (NDS). NDS are based on large scale data collection of driver, vehicle, and environment information in real world. NDS data sets have proven to be extremely valuable for the analysis of safety critical events such as crashes and near crashes. However, finding safety critical events in NDS data is often difficult and time consuming. Safety critical events are currently identified using kinematic triggers, for instance searching for deceleration below a certain threshold signifying harsh braking. Due to the low sensitivity and specificity of this filtering procedure, manual review of video data is currently necessary to decide whether the events identified by the triggers are actually safety critical. Such reviewing procedure is based on subjective decisions, is expensive and time consuming, and often tedious for the analysts. Furthermore, since NDS data is exponentially growing over time, this reviewing procedure may not be viable anymore in the very near future. This study tested the hypothesis that automatic processing of driver video information could increase the correct classification of safety critical events from kinematic triggers in naturalistic driving data. Review of about 400 video sequences recorded from the events, collected by 100 Volvo cars in the euroFOT project, suggested that drivers' individual reaction may be the key to recognize safety critical events. In fact, whether an event is safety critical or not often depends on the individual driver. A few algorithms, able to automatically classify driver reaction from video data, have been compared. The results presented in this paper show that the state of the art subjective review procedures to identify safety critical events from NDS can benefit from automated objective video processing. In addition, this paper discusses the major challenges in making such video analysis viable for future NDS and new potential applications for NDS video processing. As new NDS such as SHRP2 are now providing the equivalent of five years of one vehicle data each day, the development of new methods, such as the one proposed in this paper, seems necessary to guarantee that these data can actually be analysed.

A framework for automatic modeling from point cloud data

We propose a complete framework for the automatic modeling from point cloud data. Initially, the point cloud data are preprocessed into manageable datasets, which are then separated into clusters using a novel two-step, unsupervised clustering algorithm. The boundaries extracted for each cluster are then simplified and refined using a fast energy minimization process. Finally, three-dimensional models are generated based on the roof outlines. The proposed framework has been extensively tested, and the results are reported.

Efficient reconstruction of all-in-focus images through shifted pinholes from multi-focus images for dense light field synthesis and rendering

Scene refocusing beyond extended depth of field for users to observe objects effectively is aimed by researchers in computational photography, microscopic imaging, and so on. Ordinary all-in-focus image reconstruction from a sequence of multi-focus images achieves extended depth of field, where reconstructed images would be captured through a pinhole in the center on the lens. In this paper, we propose a novel method for reconstructing all-in-focus images through shifted pinholes on the lens based on 3D frequency analysis of multi-focus images. Such shifted pinhole images are obtained by a linear combination of multi-focus images with scene-independent 2D filters in the frequency domain. The proposed method enables us to efficiently synthesize dense 4D light field on the lens plane for image-based rendering, especially, robust scene refocusing with arbitrary bokeh. Our novel method using simple linear filters achieves not only reconstruction of all-in-focus images even for shifted pinholes more robustly than the conventional methods depending on scene/focus estimation, but also scene refocusing without suffering from limitation of resolution in comparison with recent approaches using special devices such as lens arrays in computational photography.

Automatic Characterization of the Visual Appearance of Industrial Materials through Colour and Texture Analysis: An Overview of Methods and Applications

We present an overview of methods and applications of automatic characterization of the appearance of materials through colour and texture analysis. We propose a taxonomy based on three classes of methods (spectral, spatial, and hybrid) and discuss their general advantages and disadvantages. For each class we present a set of methods that are computationally cheap and easy to implement and that was proved to be reliable in many applications. We put these methods in the context of typical industrial environments and provide examples of their application in the following tasks: surface grading, surface inspection, and content-based image retrieval. We emphasize the potential benefits that would come from a wide implementation of these methods, such as better product quality, new services, and higher customer satisfaction.

On the use of a low-cost thermal sensor to improve Kinect people detection in a mobile robot

Detecting people is a key capability for robots that operate in populated environments. In this paper, we have adopted a hierarchical approach that combines classifiers created using supervised learning in order to identify whether a person is in the view-scope of the robot or not. Our approach makes use of vision, depth and thermal sensors mounted on top of a mobile platform. The set of sensors is set up combining the rich data source offered by a Kinect sensor, which provides vision and depth at low cost, and a thermopile array sensor. Experimental results carried out with a mobile platform in a manufacturing shop floor and in a science museum have shown that the false positive rate achieved using any single cue is drastically reduced. The performance of our algorithm improves other well-known approaches, such as C⁴ and histogram of oriented gradients (HOG).

A review of data fusion techniques

The integration of data and knowledge from several sources is known as data fusion. This paper summarizes the state of the data fusion field and describes the most relevant studies. We first enumerate and explain different classification schemes for data fusion. Then, the most common algorithms are reviewed. These methods and algorithms are presented using three different categories: (i) data association, (ii) state estimation, and (iii) decision fusion.

3D Reconstruction of virtual colon structures from colonoscopy images

This paper presents the first fully automated reconstruction technique of 3D virtual colon segments from individual colonoscopy images. It is the basis of new software applications that may offer great benefits for improving quality of care for colonoscopy patients. For example, a 3D map of the areas inspected and uninspected during colonoscopy can be shown on request of the endoscopist during the procedure. The endoscopist may revisit the suggested uninspected areas to reduce the chance of missing polyps that reside in these areas. The percentage of the colon surface seen by the endoscopist can be used as a coarse objective indicator of the quality of the procedure. The derived virtual colon models can be stored for post-procedure training of new endoscopists to teach navigation techniques that result in a higher level of procedure quality. Our technique does not require a prior CT scan of the colon or any global positioning device. Our experiments on endoscopy images of an Olympus synthetic colon model reveal encouraging results with small average reconstruction errors (4.1 mm for the fold depths and 12.1 mm for the fold circumferences).

Internal limiting membrane contrast after staining with indocyanine green and brilliant blue G during macular surgery

PURPOSE

To evaluate the difference in color contrast by performing a color contrast ratio (CR) analysis and resulting visibility of the internal limiting membrane (ILM) when stained with indocyanine green and brilliant blue G (BBG) during macular surgery by performing a color CR analysis.

METHODS

The authors analyzed 40 consecutive cases in which vitrectomy with ILM removal was performed to treat a macular hole or an epiretinal membrane. The surgical procedure was performed in 21 patients (21 eyes) after staining with indocyanine green and in 19 patients (19 eyes) after staining with BBG. The color CRs were estimated based on digital analysis of the red, green, and blue data of the digital images captured, and the CRs obtained with the two dyes were compared.

RESULTS

Color contrast analysis was performed in all 40 eyes, in which the ILM was removed after staining with indocyanine green or BBG, and the CRs were estimated in every eye. The CR (mean ± SD) obtained with indocyanine green and BBG was 4.3 ± 0.3 and 2.4 ± 0.1, respectively. Indocyanine green provided a significantly higher CR than BBG (P = 0.015).

CONCLUSION

Digital color contrast analysis can be used to evaluate the visibility of digital images, and it may be useful when choosing the dye to use for staining the ILM better.