Layered motion segmentation and depth ordering by tracking edges

Automatic Discoid Lateral Meniscus Diagnosis from Radiographs Based on Image Processing Tools and Machine Learning

The aim of the present study is to build a software implementation of a previous study and to diagnose discoid lateral menisci on knee joint radiograph images. A total of 160 images from normal individuals and patients who were diagnosed with discoid lateral menisci were included. Our software implementation includes two parts: preprocessing and measurement. In the first phase, the whole radiograph image was analyzed to obtain basic information about the patient. Machine learning was used to segment the knee joint from the original radiograph image. Image enhancement and denoising tools were used to strengthen the image and remove noise. In the second phase, edge detection was used to quantify important features in the image. A specific algorithm was designed to build a model of the knee joint and measure the parameters. Of the test images, 99.65% were segmented correctly. Furthermore, 97.5% of the tested images were segmented correctly and their parameters were measured successfully. There was no significant difference between manual and automatic measurements in the discoid (P=0.28) and control groups (P=0.15). The mean and standard deviations of the ratio of lateral joint space distance to the height of the lateral tibial spine were compared with the results of manual measurement. The software performed well on raw radiographs, showing a satisfying success rate and robustness. Thus, it is possible to diagnose discoid lateral menisci on radiographs with the help of radiograph-image-analyzing software (BM3D, etc.) and artificial intelligence-related tools (YOLOv3). The results of this study can help build a joint database that contains data from patients and thus can play a role in the diagnosis of discoid lateral menisci and other knee joint diseases in the future.

Orthographic Perspective Mappings for Consistent Wide-Area Motion Feature Maps From Multiple Cameras

Spatiotemporal activity maps have been used to visualize where activity occurs over time, and are often displayed as pseudo-color heat maps. Our multi-dimensional activity map includes the following motion features: density, direction, bi-direction, velocity, and dwell. The primary contribution of this paper is to describe a set of mappings that will transform the activity maps captured from the cameras of different perspectives to the ones from a single orthographic perspective. The purpose of this is to be able to view and compare multiple activity maps from different camera views over a wide area with consistently comparable data. A second contribution is that the most mappings are based upon statistically learned camera perspectives to minimize manual camera calibration. We demonstrate mapping results with multiple video data sets and describe applications for visualization and wide-area spatial probability estimation.

Flexible synthesis of video frames based on motion hints

In this paper, we propose the use of "motion hints" to produce interframe predictions. A motion hint is a loose and global description of motion that can be communicated using metadata; it describes a continuous and invertible motion model over multiple frames, spatially overlapping other motion hints. A motion hint provides a reasonably accurate description of motion but only a loose description of where it is applicable; it is the task of the client to identify the exact locations where this motion model is applicable. The focus of this paper is a probabilistic multiscale approach to identifying these locations of applicability; the method is robust to noise, quantization, and contrast changes. The proposed approach employs the Laplacian pyramid; it generates motion hint probabilities from observations at each scale of the pyramid. These probabilities are then combined across the scales of the pyramid starting from the coarsest scale. The computational cost of the approach is reasonable, and only the neighborhood of a pixel is employed to determine a motion hint probability, which makes parallel implementation feasible. This paper also elaborates on how motion hint probabilities are exploited in generating interframe predictions. The scheme of this paper is applicable to closed-loop prediction, but it is more useful in open-loop prediction scenarios, such as using prediction in conjunction with remote browsing of surveillance footage, communicated by a JPEG2000 Interactive Protocol (JPIP) server. We show that the interframe predictions obtained using the proposed approach are good both visually and in terms of PSNR.

Detachable object detection: segmentation and depth ordering from short-baseline video

We describe an approach for segmenting a moving image into regions that correspond to surfaces in the scene that are partially surrounded by the medium. It integrates both appearance and motion statistics into a cost functional that is seeded with occluded regions and minimized efficiently by solving a linear programming problem. Where a short observation time is insufficient to determine whether the object is detachable, the results of the minimization can be used to seed a more costly optimization based on a longer sequence of video data. The result is an entirely unsupervised scheme to detect and segment an arbitrary and unknown number of objects. We test our scheme to highlight the potential, as well as limitations, of our approach.

Large disparity motion layer extraction via topological clustering

In this paper, we present a robust and efficient approach to extract motion layers from a pair of images with large disparity motion. First, motion models are established as: 1) initial SIFT matches are obtained and grouped into a set of clusters using our developed topological clustering algorithm; 2) for each cluster with no less than three matches, an affine transformation is estimated with least-square solution as tentative motion model; and 3) the tentative motion models are refined and the invalid models are pruned. Then, with the obtained motion models, a graph cuts based layer assignment algorithm is employed to segment the scene into several motion layers. Experimental results demonstrate that our method can successfully segment scenes containing objects with large interframe motion or even with significant interframe scale and pose changes. Furthermore, compared with the previous method invented by Wills and its modified version, our method is much faster and more robust.

Layered graph matching with composite cluster sampling

This paper presents a framework of layered graph matching for integrating graph partition and matching. The objective is to find an unknown number of corresponding graph structures in two images. We extract discriminative local primitives from both images and construct a candidacy graph whose vertices are matching candidates (i.e., a pair of primitives) and whose edges are either negative for mutual exclusion or positive for mutual consistence. Then we pose layered graph matching as a multicoloring problem on the candidacy graph and solve it using a composite cluster sampling algorithm. This algorithm assigns some vertices into a number of colors, each being a matched layer, and turns off all the remaining candidates. The algorithm iterates two steps: 1) Sampling the positive and negative edges probabilistically to form a composite cluster, which consists of a few mutually conflicting connected components (CCPs) in different colors and 2) assigning new colors to these CCPs with consistence and exclusion relations maintained, and the assignments are accepted by the Markov Chain Monte Carlo (MCMC) mechanism to preserve detailed balance. This framework demonstrates state-of-the-art performance on several applications, such as multi-object matching with large motion, shape matching and retrieval, and object localization in cluttered background.

Real-time motion segmentation of sparse feature points at any speed

We present a real-time incremental approach to motion segmentation operating on sparse feature points. In contrast to previous work, the algorithm allows for a variable number of image frames to affect the segmentation process, thus enabling an arbitrary number of objects traveling at different relative speeds to be detected. Feature points are detected and tracked throughout an image sequence, and the features are grouped using a spatially constrained expectation-maximization (EM) algorithm that models the interactions between neighboring features using the Markov assumption. The primary parameter used by the algorithm is the amount of evidence that must accumulate before features are grouped. A statistical goodness-of-fit test monitors the change in the motion parameters of a group over time in order to automatically update the reference frame. Experimental results on a number of challenging image sequences demonstrate the effectiveness and computational efficiency of the technique.

Entropy-controlled quadratic markov measure field models for efficient image segmentation

We present a new Markov random field (MRF) based model for parametric image segmentation. Instead of directly computing a label map, our method computes the probability that the observed data at each pixel is generated by a particular intensity model. Prior information about segmentation smoothness and low entropy of the probability distribution maps is codified in the form of a MRF with quadratic potentials so that the optimal estimator is obtained by solving a quadratic cost function with linear constraints. Although, for segmentation purposes, the mode of the probability distribution at each pixel is naturally used as an optimal estimator, our method permits the use of other estimators, such as the mean or the median, which may be more appropriate for certain applications. Numerical experiments and comparisons with other published schemes are performed, using both synthetic images and real data of brain MRI for which expert hand-made segmentations are available. Finally, we show that the proposed methodology may be easily extended to other problems, such as stereo disparity estimation.

Segmentation framework based on label field fusion

In this paper, we put forward a novel fusion framework that mixes together label fields instead of observation data as is usually the case. Our framework takes as input two label fields: a quickly estimated and to-be-refined segmentation map and a spatial region map that exhibits the shape of the main objects of the scene. These two label fields are fused together with a global energy function that is minimized with a deterministic iterative conditional mode algorithm. As explained in the paper, the energy function may implement a pure fusion strategy or a fusion-reaction function. In the latter case, a data-related term is used to make the optimization problem well posed. We believe that the conceptual simplicity, the small number of parameters, the use of a simple and fast deterministic optimizer that admits a natural implementation on a parallel architecture are among the main advantages of our approach. Our fusion framework is adapted to various computer vision applications among which are motion segmentation, motion estimation and occlusion detection.

Motion layer extraction in the presence of occlusion using graph cuts

Extracting layers from video is very important for video representation, analysis, compression, and synthesis. Assuming that a scene can be approximately described by multiple planar regions, this paper describes a robust and novel approach to automatically extract a set of affine or projective transformations induced by these regions, detect the occlusion pixels over multiple consecutive frames, and segment the scene into several motion layers. First, after determining a number of seed regions using correspondences in two frames, we expand the seed regions and reject the outliers employing the graph cuts method integrated with level set representation. Next, these initial regions are merged into several initial layers according to the motion similarity. Third, an occlusion order constraint on multiple frames is explored, which enforces that the occlusion area increases with the temporal order in a short period and effectively maintains segmentation consistency over multiple consecutive frames. Then, the correct layer segmentation is obtained by using a graph cuts algorithm and the occlusions between the overlapping layers are explicitly determined. Several experimental results are demonstrated to show that our approach is effective and robust.

A voting-based computational framework for visual motion analysis and interpretation

Most approaches for motion analysis and interpretation rely on restrictive parametric models and involve iterative methods which depend heavily on initial conditions and are subject to instability. Further difficulties are encountered in image regions where motion is not smooth-typically around motion boundaries. This work addresses the problem of visual motion analysis and interpretation by formulating it as an inference of motion layers from a noisy and possibly sparse point set in a 4D space. The core of the method is based on a layered 4D representation of data and a voting scheme for affinity propagation. The inherent problem caused by the ambiguity of 2D to 3D interpretation is usually handled by adding additional constraints, such as rigidity. However, enforcing such a global constraint has been problematic in the combined presence of noise and multiple independent motions. By decoupling the processes of matching, outlier rejection, segmentation, and interpretation, we extract accurate motion layers based on the smoothness of image motion, then locally enforce rigidity for each layer in order to infer its 3D structure and motion. The proposed framework is noniterative and consistently handles both smooth moving regions and motion discontinuities without using any prior knowledge of the motion model.