Distortion-invariant pattern recognition with Fourier-plane nonlinear filters

Optical signal detection in turbid water using multidimensional integral imaging with deep learning

Optical signal detection in turbid and occluded environments is a challenging task due to the light scattering and beam attenuation inside the medium. Three-dimensional (3D) integral imaging is an imaging approach which integrates two-dimensional images from multiple perspectives and has proved to be useful for challenging conditions such as occlusion and turbidity. In this manuscript, we present an approach for the detection of optical signals in turbid water and occluded environments using multidimensional integral imaging employing temporal encoding with deep learning. In our experiments, an optical signal is temporally encoded with gold code and transmitted through turbid water via a light-emitting diode (LED). A camera array captures videos of the optical signals from multiple perspectives and performs the 3D signal reconstruction of temporal signal. The convolutional neural network-based bidirectional Long Short-Term Network (CNN-BiLSTM) network is trained with clear water video sequences to perform classification on the binary transmitted signal. The testing data was collected in turbid water scenes with partial signal occlusion, and a sliding window with CNN-BiLSTM-based classification was performed on the reconstructed 3D video data to detect the encoded binary data sequence. The proposed approach is compared to previously presented correlation-based detection models. Furthermore, we compare 3D integral imaging to conventional two-dimensional (2D) imaging for signal detection using the proposed deep learning strategy. The experimental results using the proposed approach show that the multidimensional integral imaging-based methodology significantly outperforms the previously reported approaches and conventional 2D sensing-based methods. To the best of our knowledge, this is the first report on underwater signal detection using multidimensional integral imaging with deep neural networks.

Signal detection in turbid water using temporally encoded polarimetric integral imaging

To improve signal detection in a turbid medium, we propose temporally encoded single shot polarimetric integral imaging. An optical signal is temporally encoded using gold coded sequences and transmitted through a turbid medium. The encoded signals are captured as a sequence of elemental images by two orthogonal polarized image sensor arrays. Polarimetric and polarization difference imaging are used to suppress the partially polarized and unpolarized background noise such that only the polarized ballistic signal photons are captured at the sensor. Multidimensional integral imaging is used to obtain 4D reconstructed data, and multidimensional nonlinear correlation is performed on the reconstructed data to detect the optical signal. We compare the effectiveness of the proposed polarimetric underwater optical signal detection approach to conventional (non-polarimetric) integral imaging-based and 2D imaging-based signal detection systems. The underwater signal detection capabilities are measured through performance metrics such as receiver operating characteristic (ROC) curves, the area under the curve (AUC), and the number of detection errors. Furthermore, statistical measures, including the Kullback-Leibler divergence, signal-to-noise ratio (SNR), and peak-to-correlation energy (PCE), are also calculated to show the improved performance of the proposed system. Our experimental results show that the proposed polarimetric integral-imaging approach significantly outperforms the conventional imaging-based methods. To the best of our knowledge, this is the first report on temporally encoded single shot polarimetric integral imaging for signal detection in turbid water.

Human gesture recognition under degraded environments using 3D-integral imaging and deep learning

In this paper, we propose a spatio-temporal human gesture recognition algorithm under degraded conditions using three-dimensional integral imaging and deep learning. The proposed algorithm leverages the advantages of integral imaging with deep learning to provide an efficient human gesture recognition system under degraded environments such as occlusion and low illumination conditions. The 3D data captured using integral imaging serves as the input to a convolutional neural network (CNN). The spatial features extracted by the convolutional and pooling layers of the neural network are fed into a bi-directional long short-term memory (BiLSTM) network. The BiLSTM network is designed to capture the temporal variation in the input data. We have compared the proposed approach with conventional 2D imaging and with the previously reported approaches using spatio-temporal interest points with support vector machines (STIP-SVMs) and distortion invariant non-linear correlation-based filters. Our experimental results suggest that the proposed approach is promising, especially in degraded environments. Using the proposed approach, we find a substantial improvement over previously published methods and find 3D integral imaging to provide superior performance over the conventional 2D imaging system. To the best of our knowledge, this is the first report that examines deep learning algorithms based on 3D integral imaging for human activity recognition in degraded environments.

Spatial-temporal human gesture recognition under degraded conditions using three-dimensional integral imaging

We present spatial-temporal human gesture recognition in degraded conditions including low light levels and occlusions using passive sensing three-dimensional (3D) integral imaging (InIm) system and 3D correlation filters. The 4D (lateral, longitudinal, and temporal) reconstructed data is processed using a variety of algorithms including linear and non-linear distortion-invariant filters; and compared with previously reported space-time interest points (STIP) feature detector, 3D histogram of oriented gradients (3D HOG) feature descriptor, with a standard bag-of-features support vector machine (SVM) framework, etc. The gesture recognition results with different classification algorithms are compared using a variety of performance metrics such as receiver operating characteristic (ROC) curves, area under the curve (AUC), SNR, the probability of classification errors, and confusion matrix. Integral imaging video sequences of human gestures are captured under degraded conditions such as low light illumination and in the presence of partial occlusions. A four-dimensional (4D) reconstructed video sequence is computed that provides lateral and depth information of a scene over time i.e. (x, y, z, t). The total-variation denoising algorithm is applied to the signal to further reduce noise and preserve data in the video frames. We show that the 4D signal consists of decreased scene noise, partial occlusion removal, and improved SNR due to the computational InIm and/or denoising algorithms. Finally, gesture recognition is processed with classification algorithms, such as distortion-invariant correlation filters; and STIP, 3D HOG with SVM, which are applied to the reconstructed 4D gesture signal to classify the human gesture. Experiments are conducted using a synthetic aperture InIm system in ambient light. Our experiments indicate that the proposed approach is promising in detection of human gestures in degraded conditions such as low illumination conditions with partial occlusion. To the best of our knowledge, this is the first report on spatial-temporal human gesture recognition in degraded conditions using passive sensing 4D integral imaging with nonlinear correlation filters.

Coregistered photoacoustic and ultrasound imaging and classification of ovarian cancer: ex vivo and in vivo studies

Most ovarian cancers are diagnosed at advanced stages due to the lack of efficacious screening techniques. Photoacoustic tomography (PAT) has a potential to image tumor angiogenesis and detect early neovascular changes of the ovary. We have developed a coregistered PAT and ultrasound (US) prototype system for real-time assessment of ovarian masses. Features extracted from PAT and US angular beams, envelopes, and images were input to a logistic classifier and a support vector machine (SVM) classifier to diagnose ovaries as benign or malignant. A total of 25 excised ovaries of 15 patients were studied and the logistic and SVM classifiers achieved sensitivities of 70.4 and 87.7%, and specificities of 95.6 and 97.9%, respectively. Furthermore, the ovaries of two patients were noninvasively imaged using the PAT/US system before surgical excision. By using five significant features and the logistic classifier, 12 out of 14 images (86% sensitivity) from a malignant ovarian mass and all 17 images (100% specificity) from a benign mass were accurately classified; the SVM correctly classified 10 out of 14 malignant images (71% sensitivity) and all 17 benign images (100% specificity). These initial results demonstrate the clinical potential of the PAT/US technique for ovarian cancer diagnosis.

Utilizing spatial and spectral features of photoacoustic imaging for ovarian cancer detection and diagnosis

A composite set of ovarian tissue features extracted from photoacoustic spectral data, beam envelope, and co-registered ultrasound and photoacoustic images are used to characterize malignant and normal ovaries using logistic and support vector machine (SVM) classifiers. Normalized power spectra were calculated from the Fourier transform of the photoacoustic beamformed data, from which the spectral slopes and 0-MHz intercepts were extracted. Five features were extracted from the beam envelope and another 10 features were extracted from the photoacoustic images. These 17 features were ranked by their p-values from t -tests on which a filter type of feature selection method was used to determine the optimal feature number for final classification. A total of 169 samples from 19 ex vivo ovaries were randomly distributed into training and testing groups. Both classifiers achieved a minimum value of the mean misclassification error when the seven features with lowest p-values were selected. Using these seven features, the logistic and SVM classifiers obtained sensitivities of 96.39 ± 3.35% and 97.82 ± 2.26%, and specificities of 98.92 ± 1.39% and 100%, respectively, for the training group. For the testing group, logistic and SVM classifiers achieved sensitivities of 92.71 ± 3.55% and 92.64 ± 3.27%, and specificities of 87.52 ± 8.78% and 98.49 ± 2.05%, respectively.

Review of holographic-based three-dimensional object recognition techniques [invited]

With the advancement of computing and optical technologies, it is now possible to capture digital holograms of real-life object scenes. Theoretically, through the analysis of a hologram, the three-dimensional (3D) objects coded on the hologram can be identified. However, being different from an optical image, a hologram is composed of complicated fringes that cannot be analyzed easily with traditional computer vision methods. Over the years, numerous important research investigations have been attempted to provide viable solutions to this problem. The aim of this work is three-fold. First, we provide a quick walkthrough on the overall development of holographic-based 3D object recognition (H3DOR) in the past five decades, from film-based approaches to digital-based innovation. Second, we describe in more detail a number of selected H3DOR methods that are introduced at different timelines, starting from the late sixties and then from the seventies, where viable digital holographic-based 3D recognition methods began to emerge. Finally, we present our work on digital holographic, pose-invariant 3D object recognition that is based on a recently introduced virtual diffraction plane framework. As our method has not been reported elsewhere, we have included some experimental results to demonstrate the feasibility of the approach.

Recognition algorithm for assisting ovarian cancer diagnosis from coregistered ultrasound and photoacoustic images: ex vivo study

Unique features and the underlining hypotheses of how these features may relate to the tumor physiology in coregistered ultrasound and photoacoustic images of ex vivo ovarian tissue are introduced. The images were first compressed with wavelet transform. The mean Radon transform of photoacoustic images was then computed and fitted with a Gaussian function to find the centroid of a suspicious area for shift-invariant recognition process. Twenty-four features were extracted from a training set by several methods, including Fourier transform, image statistics, and different composite filters. The features were chosen from more than 400 training images obtained from 33 ex vivo ovaries of 24 patients, and used to train three classifiers, including generalized linear model, neural network, and support vector machine (SVM). The SVM achieved the best training performance and was able to exclusively separate cancerous from non-cancerous cases with 100% sensitivity and specificity. At the end, the classifiers were used to test 95 new images obtained from 37 ovaries of 20 additional patients. The SVM classifier achieved 76.92% sensitivity and 95.12% specificity. Furthermore, if we assume that recognizing one image as a cancer is sufficient to consider an ovary as malignant, the SVM classifier achieves 100% sensitivity and 87.88% specificity.

Activity inference for Ambient Intelligence through handling artifacts in a healthcare environment

Human activity inference is not a simple process due to distinct ways of performing it. Our proposal presents the SCAN framework for activity inference. SCAN is divided into three modules: (1) artifact recognition, (2) activity inference, and (3) activity representation, integrating three important elements of Ambient Intelligence (AmI) (artifact-behavior modeling, event interpretation and context extraction). The framework extends the roaming beat (RB) concept by obtaining the representation using three kinds of technologies for activity inference. The RB is based on both analysis and recognition from artifact behavior for activity inference. A practical case is shown in a nursing home where a system affording 91.35% effectiveness was implemented in situ. Three examples are shown using RB representation for activity representation. Framework description, RB description and CALog system overcome distinct problems such as the feasibility to implement AmI systems, and to show the feasibility for accomplishing the challenges related to activity recognition based on artifact recognition. We discuss how the use of RBs might positively impact the problems faced by designers and developers for recovering information in an easier manner and thus they can develop tools focused on the user.

Detecting three-dimensional location and shape of noisy distorted three-dimensional objects with ladar trained optimum nonlinear filters

We propose a filtering technique that uses laser radar (ladar) data to detect a target's three-dimensional (3D) coordinates and shape within an input scene. A two-dimensional ladar range image is converted into 3D space, and then the 3D optimum nonlinear filtering technique is used to detect the 3D coordinates of targets (including the target's distance from the sensor). The 3D optimum nonlinear filter is designed to detect distorted targets (i.e., out-of-plane and in-plane rotations and scale changes) and to be noise robust. The nonlinear filter is derived to minimize the mean of the output energy in response to the input scene in the presence of disjoint background noise and additive noise and to maintain a fixed output peak for the members of the true-class target training set. The system is tested with real ladar imagery in the presence of background clutter. The background clutter used in the system evaluation includes false objects that are similar to the true targets. The correlation output of ladar images shows a dominant peak at the target's 3D coordinates.

Comparison of passive ranging integral imaging and active imaging digital holography for three-dimensional object recognition

We present an overview of three-dimensional (3D) object recognition techniques that use active sensing by interferometric imaging (digital holography) and passive sensing by integral imaging. We describe how each technique can be used to retrieve the depth information of a 3D scene and how this information can then be used for 3D object recognition. We explore various algorithms for 3D recognition such as nonlinear correlation and target distortion tolerance. We also provide a comparison of the advantages and disadvantages of the two techniques.

Pose estimation from a two-dimensional view by use of composite correlation filters and neural networks

We present a technique to estimate the pose of a three-dimensional object from a two-dimensional view. We first compute the correlation between the unknown image and several synthetic-discriminant-function filters constructed with known views of the object. We consider both linear and nonlinear correlations. The filters are constructed in such a way that the obtained correlation values depend on the pose parameters. We show that this dependence is not perfectly linear, in particular for nonlinear correlation. Therefore we use a two-layer neural network to retrieve the pose parameters from the correlation values. We demonstrate the technique by simultaneously estimating the in-plane and out-of-plane orientations of an airplane within an 8-deg portion. We show that a nonlinear correlation is necessary to identify the object and also to estimate its pose. On the other hand, linear correlation is more accurate and more robust. A combination of linear and nonlinear correlations gives the best results.

Pattern recognition with a digital holographic microscope working in partially coherent illumination

We describe the implementation of the automatic spatial-frequency-selection filter for recognition of patterns obtained with a digital holographic microscope working with a partially coherent source. The microscope provides the complex-optical-amplitude field that allows a refocusing plane-by-plane of the sample under investigation by numerical computation of the optical propagation. By inserting a correlation filter in the propagation equation, the correlation between the filter and the propagated optical field is obtained. In this way, the pattern is located in the direction of the optical axis. Owing to the very weak noise level generated by the partially coherent source, the correlation process is shift invariant. Therefore the samples can be located in the three dimensions. To have a robust recognition process, a generalized version of the automatic spatial-frequency-selection filters has been implemented. The method is experimentally demonstrated in a two-class problem for the recognition of protein crystals.

Optimum nonlinear composite filter for distortion-tolerant pattern recognition

We describe a nonlinear distortion-tolerant filter for pattern recognition that is optimum in terms of tolerance to input noise and discrimination capability. This filter was derived by minimization of the output energy that is due to the overlapping additive noise and the input scene, and the output of the filter meets the design constraints obtained from the training data set. The performance of this filter was tested with an input scene containing one of the training data sets, a nontraining true target, and a false object in the presence of overlapping additive noise and nonoverlapping background noise. We carried out Monte Carlo runs to measure the statistical performance of the filter and obtained receiver operating characteristics curves to show the detection capabilities of the filter.

Fractional profilometry correlator for three-dimensional object recognition

A novel method for three-dimensional shape recognition is proposed. It combines the Fourier-transform-profilometry technique with a two-dimensional fractional correlation algorithm. A grating is projected onto the object surface, resulting in a distorted grating pattern that carries information about the height and the shape of the object. Three-dimensional objects are recognized by a fractional correlator by use of the transformed complex amplitude. An optoelectronic hybrid implementation is also suggested.

Neural network for three-dimensional object recognition based on digital holography

We present a two-layer neural network for processing of three-dimensional (3D) images that are obtained by digital holography. The network is trained with a real 3D object to compute the weights of the layers. Experiments are presented to illustrate the system performance. The system is designed to detect a 3D object in the presence of various distortions. As an example, experiments are presented to illustrate how the system is able to recognize a 3D object with 360 degrees out-of-plane rotation.

Noise robustness of nonlinear filters for image recognition

We analyze the performance of the Fourier plane nonlinear filters in terms of signal-to-noise ratio (SNR). We obtain a range of nonlinearities for which SNR is robust to the variations in input-noise bandwidth. This is shown both by analytical estimates of the SNR for nonlinear filters and by experimental simulations. Specifically, we analyze the SNR when Fourier plane nonlinearity is applied to the input signal. Using the Karhunen-Loève series expansion of the noise process, we obtain precise analytic expressions of the SNR for Fourier plane nonlinear filters in the presence of various types of additive-noise processes. We find a range of nonlinearities that need to be applied that keep the output SNR of the filter stable relative to changes in the noise bandwidth.

Distortion-tolerant three-dimensional object recognition with digital holography

We present a technique to implement three-dimensional (3-D) object recognition based on phase-shift digital holography. We use a nonlinear composite correlation filter to achieve distortion tolerance. We take advantage of the properties of holograms to make the composite filter by using one single hologram. Experiments are presented to illustrate the recognition of a 3-D object in the presence of out-of-plane rotation and longitudinal shift along the z axis.

Nonlinear joint fractional transform correlator

An important tool in optical pattern recognition, the joint fractional transform correlator (JFTC), was introduced recently. We analyze the peak properties of fractional correlation (FC) by symbolic derivation and computer simulation. We show that the FC has a maximum correlation peak when the second fractional Fourier transform is reduced to the conventional Fourier transform. We introduce nonlinear operations in a joint fractional transform power spectrum and propose a differential JFTC and a binary differential JFTC. Numerical simulations show that such nonlinear JFTCs exhibit remarkable improvement in correlation peak intensity, discrimination capability, and signal-to-noise ratio. An optoelectronic setup that can implement such nonlinear JFTCs is also proposed.

Multifractional correlation

Multifractional correlation is proposed that is based on a new generalized fractional Fourier transform to which we refer as a multifractional Fourier transform. The multifractional correlation yields remarkable improvements in the correlation output peak intensity, peak sharpness, and light efficiency compared with convention correlation, which uses matched and phase-only filters, and still maintains better target discrimination capability and a reasonable robustness to noise. An optoelectronic hybrid system that can implement the multifractional correlation is also suggested.

Linear and nonlinear operation of a time-to-space processor

The operational characteristics of a time-to-space processor based on three-wave mixing for instantaneous imaging of ultrafast waveforms are investigated. We assess the effects of various system parameters on the processor's important attributes: time window of operation and signal conversion efficiency. Both linear and nonlinear operation regimes are considered, with use of a Gaussian pulse profile and a Gaussian spatial mode model. This model enables us to define a resolution measure for the processor, which is found to be an important characteristic. When the processor is operated in the linear interaction regime, we find that the conversion efficiency of a temporal signal to a spatial image is inversely proportional to the resolution measure. In the nonlinear interaction regime, nonuniform signal conversion due to fundamental wave depletion gives rise to a phenomenon that can be used to enhanced the imaging operation. We experimentally verify this nonlinear operation.

Positioning of Noncooperative Objects by use of Joint Transform Correlation Combined with Fringe Projection

Automated assembly and quality control require reliable systems for the detection of the position and the orientation of complicated objects. Correlation methods are well suited, but they are affected by structured backgrounds, varying illumination conditions, and textured or dirty object surfaces. Using fringe projection to exploit the three-dimensional topography of objects, we improve the performance of a nonlinear joint transform correlator. Positioning of noncooperative objects with subpixel accuracy is demonstrated. Additionally, the tilt angle of an arbitrarily shaped object is measured by projection of object-adapted fringes that produce a homogeneous fringe pattern in the image plane. An accuracy of better than 1 degrees is achieved.

Synthetic discriminant function filter employing nonlinear space-domain preprocessing on bandpass-filtered images

Previously [Appl. Opt. 36, p. 9212 (1997)] we examined the performance of the linear and nonlinear preprocessed difference-of-Gaussians filter, and it was shown that this operation results in greater tolerance to in-class variations while maintaining excellent discrimination ability. The introduction of nonlinearity was shown to provide greater robustness to the filter's response to noise and background clutter in the input scene. We incorporate this new operation into the synthesis of a modified synthetic discriminant function filter. The filter is shown to produce sharp peaks, excellent discrimination without the need to include out-of-class objects, and good invariance to out-of-plane rotation over a distortion range of up to 90 degrees . Additionally, the introduction of nonlinearity is shown to provide greater robustness of the filter response to background clutter in the input scene.

Application of nonlinearity to wavelet-transformed images to improve correlation filter performance

A useful filter for pattern recognition must strike a compromise between the conflicting requirements of in-class distortion tolerance and out-of-class discrimination. Such a filter will be bandpass in nature, the high-frequency response being attenuated to provide less sensitivity to in-class variations, while the low frequencies must be removed, since they compromise the discrimination ability of the filter. A convenient bandpass is the difference of Gaussian (DOG) function, which provides a close approximation to the Laplacian of Gaussian. We describe the effect of a preprocessing operation applied to a DOG filtered image. This operation is shown to result in greater tolerance to in-class variation while maintaining an excellent discrimination ability. Additionally, the introduction of nonlinearity is shown to provide greater robustness in the filter response to noise and background clutter in the input scene.