Computing optical flow across multiple scales: An adaptive coarse-to-fine strategy

Learned optical flow for intra-operative tracking of the retinal fundus

Purpose

Sustained delivery of regenerative retinal therapies by robotic systems requires intra-operative tracking of the retinal fundus. We propose a supervised deep convolutional neural network to densely predict semantic segmentation and optical flow of the retina as mutually supportive tasks, implicitly inpainting retinal flow information missing due to occlusion by surgical tools.

Methods

As manual annotation of optical flow is infeasible, we propose a flexible algorithm for generation of large synthetic training datasets on the basis of given intra-operative retinal images. We evaluate optical flow estimation by tracking a grid and sparsely annotated ground truth points on a benchmark of challenging real intra-operative clips obtained from an extensive internally acquired dataset encompassing representative vitreoretinal surgical cases.

Results

The U-Net-based network trained on the synthetic dataset is shown to generalise well to the benchmark of real surgical videos. When used to track retinal points of interest, our flow estimation outperforms variational baseline methods on clips containing tool motions which occlude the points of interest, as is routinely observed in intra-operatively recorded surgery videos.

Conclusions

The results indicate that complex synthetic training datasets can be used to specifically guide optical flow estimation. Our proposed algorithm therefore lays the foundation for a robust system which can assist with intra-operative tracking of moving surgical targets even when occluded.

Force estimation and prediction from time-varying density images

We present methods for estimating forces which drive motion observed in density image sequences. Using these forces, we also present methods for predicting velocity and density evolution. To do this, we formulate and apply a Minimum Energy Flow (MEF) method which is capable of estimating both incompressible and compressible flows from time-varying density images. Both the MEF and force-estimation techniques are applied to experimentally obtained density images, spanning spatial scales from micrometers to several kilometers. Using density image sequences describing cell splitting, for example, we show that cell division is driven by gradients in apparent pressure within a cell. Using density image sequences of fish shoals, we also quantify 1) intershoal dynamics such as coalescence of fish groups over tens of kilometers, 2) fish mass flow between different parts of a large shoal, and 3) the stresses acting on large fish shoals.

Optical flow estimation using temporally oversampled video

Recent advances in imaging sensor technology make high frame-rate video capture practical. As demonstrated in previous work, this capability can be used to enhance the performance of many image and video processing applications. The idea is to use the high frame-rate capability to temporally oversample the scene and, thus, to obtain more accurate information about scene motion and illumination. This information is then used to improve the performance of image and standard frame-rate video applications. This paper investigates the use of temporal oversampling to improve the accuracy of optical flow estimation (OFE). A method for obtaining high accuracy optical flow estimates at a conventional standard frame rate, e.g., 30 frames/s, by first capturing and processing a high frame-rate version of the video is presented. The method uses the Lucas-Kanade algorithm to obtain optical flow estimates at a high frame rate, which are then accumulated and refined to estimate the optical flow at the desired standard frame rate. The method demonstrates significant improvements in OFE accuracy both on synthetically generated video sequences and on a real video sequence captured using an experimental high-speed imaging system. It is then shown that a key benefit of using temporal oversampling to estimate optical flow is the reduction in motion aliasing. Using sinusoidal input sequences, the reduction in motion aliasing is identified and the desired minimum sampling rate as a function of the velocity and spatial bandwidth of the scene is determined. Using both synthetic and real video sequences, it is shown that temporal oversampling improves OFE accuracy by reducing motion aliasing not only for areas with large displacements but also for areas with small displacements and high spatial frequencies. The use of other OFE algorithms with temporally oversampled video is then discussed. In particular, the Haussecker algorithm is extended to work with high frame-rate sequences. This extension demonstrates yet another important benefit of temporal oversampling, which is improving OFE accuracy when brightness varies with time.

Hierarchical estimation of a dense deformation field for 3-D robust registration

A new method for medical image registration is formulated as a minimization problem involving robust estimators. We propose an efficient hierarchical optimization framework which is both multiresolution and multigrid. An anatomical segmentation of the cortex is introduced in the adaptive partitioning of the volume on which the multigrid minimization is based. This allows to limit the estimation to the areas of interest, to accelerate the algorithm, and to refine the estimation in specified areas. At each stage of the hierarchical estimation, we refine current estimate by seeking a piecewise affine model for the incremental deformation field. The performance of this method is numerically evaluated on simulated data and its benefits and robustness are shown on a database of 18 magnetic resonance imaging scans of the head.

Velocity estimation of spots in three-dimensional confocal image sequences of living cells

BACKGROUND

The analysis of three-dimensional (3D) motion is becoming increasingly important in life cell imaging. A simple description of sometimes complex patterns of movement in living cells gives insight in the underlying mechanisms governing these movements.

METHODS

We evaluate a velocity estimation method based on intensity derivatives in spatial and temporal domain from 3D confocal images of living cells. Cells of the sample contain intense spots throughout the cell nucleus. In simulations, we model these spots as Gaussian intensity profiles which are constant in intensity and shape. To quantify the quality of the estimated velocity, we introduce a reliability measure.

RESULTS

For constant linear velocity, the velocity estimation is unbiased. For accelerated motion paths or when a neighboring spot disturbs the intensity profile, the method results are biased. The influence of the point-spread function on the velocity estimation can be compensated for by introducing anisotropic derivative kernels. The insight gained in the simulations is confirmed by the results of the method applied on an image sequence of a living cell with fluorescently labeled chromatin.

CONCLUSIONS

With the velocity estimation method, a tool for estimating 3D velocity fields is described which is successfully applied to a living cell sequence. With the estimated velocity fields, motion patterns can be observed, which are a useful starting point for the analysis of dynamic processes in living cells.

Filtering requirements for gradient-based optical flow measurement

The accuracy of gradient-based optical flow algorithms depends on the ability to measure intensity gradients accurately. We show how the temporal gradient can be compromised by temporal aliasing arising from motion and how appropriate post-sampling spatial filtering improves the situation. We also demonstrate a benefit of using higher-order gradient estimators.

MRI monitoring of laser ablation using optical flow

The optical flow method is used for visualizing and quantifying the dynamics of tissue changes observed by MRI during thermal ablations. An approach was implemented for parallel two-dimensional optical flow calculations including the replacement of spurious velocities. Velocity magnitude results were found to be accurate in low-noise cases in tests using series of synthetic images. Optical flow results are presented from thermal ablation experiments utilizing a homogeneous polyacrylamide gel phantom and heterogeneous rabbit liver tissue in vivo, exhibiting heating and cooling with the accompanying quantitative characterization of the dilation and contraction of the thermally affected region. Results demonstrate that optical flow is capable of noninvasive real-time monitoring and control of interstitial laser therapy (ILT).