Kalman filter-based algorithms for estimating depth from image sequences

A multi-Kalman filter-based approach for decoding arm kinematics from EMG recordings

BACKGROUND

Remarkable work has been recently introduced to enhance the usage of Electromyography (EMG) signals in operating prosthetic arms. Despite the rapid advancements in this field, providing a reliable, naturalistic myoelectric prosthesis remains a significant challenge. Other challenges include the limited number of allowed movements, lack of simultaneous, continuous control and the high computational power that could be needed for accurate decoding. In this study, we propose an EMG-based multi-Kalman filter approach to decode arm kinematics; specifically, the elbow angle (θ), wrist joint horizontal (X) and vertical (Y) positions in a continuous and simultaneous manner.

RESULTS

Ten subjects were examined from which we recorded arm kinematics and EMG signals of the biceps, triceps, lateral and anterior deltoid muscles corresponding to a randomized set of movements. The performance of the proposed decoder is assessed using the correlation coefficient (CC) and the normalized root-mean-square error (NRMSE) computed between the actual and the decoded kinematic. Results demonstrate that when training and testing the decoder using same-subject data, an average CC of 0.68 ± 0.1, 0.67 ± 0.12 and 0.64 ± 0.11, and average NRMSE of 0.21 ± 0.06, 0.18 ± 0.03 and 0.24 ± 0.07 were achieved for θ, X, and Y, respectively. When training the decoder using the data of one subject and decoding the data of other subjects, an average CC of 0.61 ± 0.19, 0.61 ± 0.16 and 0.48 ± 0.17, and an average NRMSE of 0.23 ± 0.07, 0.2 ± 0.05 and 0.38 ± 0.15 were achieved for θ, X, and Y, respectively.

CONCLUSIONS

These results suggest the efficacy of the proposed approach and indicates the possibility of obtaining a subject-independent decoder.

Technical Consideration towards Robust 3D Reconstruction with Multi-View Active Stereo Sensors

It is possible to construct cost-efficient three-dimensional (3D) or four-dimensional (4D) scanning systems using multiple affordable off-the-shelf RGB-D sensors to produce high-quality reconstructions of 3D objects. However, the quality of these systems' reconstructions is sensitive to a number of factors in reconstruction pipelines, such as multi-view calibration, depth estimation, 3D reconstruction, and color mapping accuracy, because the successive pipelines to reconstruct 3D meshes from multiple active stereo sensors are strongly correlated with each other. This paper categorizes the pipelines into sub-procedures and analyze various factors that can significantly affect reconstruction quality. Thus, this paper provides analytical and practical guidelines for high-quality 3D reconstructions with off-the-shelf sensors. For each sub-procedure, this paper shows comparisons and evaluations of several methods using data captured by 18 RGB-D sensors and provide analyses and discussions towards robust 3D reconstruction. Through various experiments, it has been demonstrated that significantly more accurate 3D scans can be obtained with the considerations along the pipelines. We believe our analyses, benchmarks, and guidelines will help anyone build their own studio and their further research for 3D reconstruction.

The Alpha-Beta Family of Filters to Solve the Threshold Problem: A Comparison

Typically, devices work to improve life quality, measure parameters, and make decisions. They also signalize statuses, and take actions accordingly. When working, they measure different values. These are to be compared against thresholds. Some time ago, vision systems came into play. They use camera(s) to deliver(s) images to a processor module. The received images are processed to perform detections (typically, they focus to detect objects, pedestrians, mopeds, cyclists, etc.). Images are analyzed and thresholds are used to compare the computed values. The important thing is that images are affected by noise. Therefore, the vision system performance can be affected by weather in some applications (for example, in automotive). An interesting case in this domain is when the measured/computed values show small variations near the threshold (not exceeding) but very close to it. The system is not able to signalize/declare a state in this case. It is also important to mention that changing the threshold does not guarantee solving the problem in any future case, since this may happen again. This paper proposes the Alpha-Beta family of filters as a solution to this problem. The members can track a signal based on measured values. This reveals errors when the tracked-signal’s first derivative changes sign. These errors are used in this paper to bypass the threshold problem. Since these errors appear in both situations (when the first derivative decreases from positive to negative and increases from negative to positive), the proposed method works when the observed data are in the vicinity of the threshold but above it.

Integrating Sensor Models in Deep Learning Boosts Performance: Application to Monocular Depth Estimation in Warehouse Automation

Deep learning is the mainstream paradigm in computer vision and machine learning, but performance is usually not as good as expected when used for applications in robot vision. The problem is that robot sensing is inherently active, and often, relevant data is scarce for many application domains. This calls for novel deep learning approaches that can offer a good performance at a lower data consumption cost. We address here monocular depth estimation in warehouse automation with new methods and three different deep architectures. Our results suggest that the incorporation of sensor models and prior knowledge relative to robotic active vision, can consistently improve the results and learning performance from fewer than usual training samples, as compared to standard data-driven deep learning.

DepthLearn: Learning to Correct the Refraction on Point Clouds Derived from Aerial Imagery for Accurate Dense Shallow Water Bathymetry Based on SVMs-Fusion with LiDAR Point Clouds

The determination of accurate bathymetric information is a key element for near offshore activities; hydrological studies, such as coastal engineering applications, sedimentary processes, hydrographic surveying, archaeological mapping and biological research. Through structure from motion (SfM) and multi-view-stereo (MVS) techniques, aerial imagery can provide a low-cost alternative compared to bathymetric LiDAR (Light Detection and Ranging) surveys, as it offers additional important visual information and higher spatial resolution. Nevertheless, water refraction poses significant challenges on depth determination. Till now, this problem has been addressed through customized image-based refraction correction algorithms or by modifying the collinearity equation. In this article, in order to overcome the water refraction errors in a massive and accurate way, we employ machine learning tools, which are able to learn the systematic underestimation of the estimated depths. In particular, an SVR (support vector regression) model was developed, based on known depth observations from bathymetric LiDAR surveys, which is able to accurately recover bathymetry from point clouds derived from SfM-MVS procedures. Experimental results and validation were based on datasets derived from different test-sites, and demonstrated the high potential of our approach. Moreover, we exploited the fusion of LiDAR and image-based point clouds towards addressing challenges of both modalities in problematic areas.

The opening of phenome-assisted selection era in the early seedling stage

Faster and more efficient breeding cycle is not an option to deal with unpredictable and fast global climate changes. Phenomics for collecting huge number of individuals in accurate manner could be an answer to solve this problem. We collected image data to measure plant height and manual data for shoot length to be compared. QTLs clustered of plant height and shoot length were detected in 2-week old seedlings, which was consistent with many other reports using various genetic resources in matured stage. Further, these traits are highly correlated with yield by pleiotropism or tight linkage of those traits. It implies the “phenome-assisted selection” can be applied for yield trait in rice in the very early stage to shorten the breeding cycle significantly in fast but low-cost manner.

Grasping and Tracking Using Constant Curvature Dynamic Contours

In this paper we present our constant curvature dynamic contours (snakes) and three applications of these: visual servoing and grasping, occluding contour depth extraction, and localization of miniature mobile robots. For the first application, a novel deformable contour model is implemented for the automatic determination of plausible grasp axes of unknown objects using an eye-in-hand robotic system. The system finds potential grasp point pairs, ranks them based upon measurements taken from the contour, and executes a vision-guided grasp using the highest ranked grasp point pair to determine the gripper alignment. Our method is based upon statistical active deformable models. We have developed a new snake model that is applicable to real-time vision problems. The grasping method is experimentally verified using both simple and complex unknown grasping targets. These experiments demonstrate the effectiveness of using the proposed snakes to grasp previously unknown objects in minimally structured environments. We also present a novel method for active monocular depth recovery (second application of our snakes). It combines new, highly stable active deformable models with a structured camera motion along the optical axis to produce depth estimates for all snake control points. The method has a simple formulation and is suitable for real-time, vision-based robotic applications. Experiments with a variety of objects and depths demonstrate the practicality of the method. Finally, we present a novel method for localizing miniature mobile robots (Scouts) using dynamic contours. The miniature robot is tracked as it moves and jumps in the environment. The proposed dynamic contours are very effective in tracking the fast accelerations and decelerations of this small robot. We show initial experimental results emphasizing the task of monitoring a Scout's jumps.

Mobile Robot Localization from Large-Scale Appearance Mosaics

A new practical, high-performance mobile robot localization technique is described that is motivated by the fact that many man-made environments contain substantially flat, visually textured surfaces of persistent appearance. While the tracking of image regions is much studied in computer vision, appearance is still a largely unexploited localization resource in commercially relevant applications. We show how prior appearance models can be used to enable highly repeatable mobile robot guidance that, unlike commercial alternatives, is both infrastructure-free and free-ranging. Very large-scale mosaics are constructed and used to localize a mobile robot operating in the modeled environment. Straightforward techniques from vision-based localization and mosaicking are used to produce a field-relevant AGV guidance system based only on vision and odometry. The feasibility, design, implementation, and precommercial field qualification of such a guidance system are described.

Matching methods evaluation framework for stereoscopic breast x-ray images

Three-dimensional (3-D) imaging has been intensively studied in the past few decades. Depth information is an important added value of 3-D systems over two-dimensional systems. Special focuses were devoted to the development of stereo matching methods for the generation of disparity maps (i.e., depth information within a 3-D scene). Dedicated frameworks were designed to evaluate and rank the performance of different stereo matching methods but never considering x-ray medical images. Yet, 3-D x-ray acquisition systems and 3-D medical displays have already been introduced into the diagnostic market. To access the depth information within x-ray stereoscopic images, computing accurate disparity maps is essential. We aimed at developing a framework dedicated to x-ray stereoscopic breast images used to evaluate and rank several stereo matching methods. A multiresolution pyramid optimization approach was integrated to the framework to increase the accuracy and the efficiency of the stereo matching techniques. Finally, a metric was designed to score the results of the stereo matching compared with the ground truth. Eight methods were evaluated and four of them [locally scaled sum of absolute differences (LSAD), zero mean sum of absolute differences, zero mean sum of squared differences, and locally scaled mean sum of squared differences] appeared to perform equally good with an average error score of 0.04 (0 is the perfect matching). LSAD was selected for generating the disparity maps.

On Stereo Confidence Measures for Global Methods: Evaluation, New Model and Integration into Occupancy Grids

Stereo confidence measures are important functions for global reconstruction methods and some applications of stereo. In this article we evaluate and compare several models of confidence which are defined at the whole disparity range. We propose a new stereo confidence measure to which we call the Histogram Sensor Model (HSM), and show how it is one of the best performing functions overall. We also introduce, for parametric models, a systematic method for estimating their parameters which is shown to lead to better performance when compared to parameters as computed in previous literature. All models were evaluated when applied to two different cost functions at different window sizes and model parameters. Contrary to previous stereo confidence measure benchmark literature, we evaluate the models with criteria important not only to winner-take-all stereo, but also to global applications. To this end, we evaluate the models on a real-world application using a recent formulation of 3D reconstruction through occupancy grids which integrates stereo confidence at all disparities. We obtain and discuss our results on both indoors' and outdoors' publicly available datasets.

Autonomous space target recognition and tracking approach using star sensors based on a Kalman filter

When imaged by detectors, space targets (including satellites and debris) and background stars have similar point-spread functions, and both objects appear to change as detectors track targets. Therefore, traditional tracking methods cannot separate targets from stars and cannot directly recognize targets in 2D images. Consequently, we propose an autonomous space target recognition and tracking approach using a star sensor technique and a Kalman filter (KF). A two-step method for subpixel-scale detection of star objects (including stars and targets) is developed, and the combination of the star sensor technique and a KF is used to track targets. The experimental results show that the proposed method is adequate for autonomously recognizing and tracking space targets.

Modeling of nonlinear biological phenomena modeled by S-systems

A central challenge in computational modeling of biological systems is the determination of the model parameters. In such cases, estimating these variables or parameters from other easily obtained measurements can be extremely useful. For example, time-series dynamic genomic data can be used to develop models representing dynamic genetic regulatory networks, which can be used to design intervention strategies to cure major diseases and to better understand the behavior of biological systems. Unfortunately, biological measurements are usually highly infected by errors that hide the important characteristics in the data. Therefore, these noisy measurements need to be filtered to enhance their usefulness in practice. This paper addresses the problem of state and parameter estimation of biological phenomena modeled by S-systems using Bayesian approaches, where the nonlinear observed system is assumed to progress according to a probabilistic state space model. The performances of various conventional and state-of-the-art state estimation techniques are compared. These techniques include the extended Kalman filter (EKF), unscented Kalman filter (UKF), particle filter (PF), and the developed variational Bayesian filter (VBF). Specifically, two comparative studies are performed. In the first comparative study, the state variables (the enzyme CadA, the model cadBA, the cadaverine Cadav and the lysine Lys for a model of the Cad System in Escherichia coli (CSEC)) are estimated from noisy measurements of these variables, and the various estimation techniques are compared by computing the estimation root mean square error (RMSE) with respect to the noise-free data. In the second comparative study, the state variables as well as the model parameters are simultaneously estimated. In this case, in addition to comparing the performances of the various state estimation techniques, the effect of the number of estimated model parameters on the accuracy and convergence of these techniques is also assessed. The results of both comparative studies show that the UKF provides a higher accuracy than the EKF due to the limited ability of EKF to accurately estimate the mean and covariance matrix of the estimated states through lineralization of the nonlinear process model. The results also show that the VBF provides a relative improvement over PF. This is because, unlike the PF which depends on the choice of sampling distribution used to estimate the posterior distribution, the VBF yields an optimum choice of the sampling distribution, which also utilizes the observed data. The results of the second comparative study show that, for all techniques, estimating more model parameters affects the estimation accuracy as well as the convergence of the estimated states and parameters. The VBF, however, still provides advantages over other methods with respect to estimation accuracy as well convergence.

Parametric Bayesian filters for nonlinear stochastic dynamical systems: a survey

Nonlinear stochastic dynamical systems are commonly used to model physical processes. For linear and Gaussian systems, the Kalman filter is optimal in minimum mean squared error sense. However, for nonlinear or non-Gaussian systems, the estimation of states or parameters is a challenging problem. Furthermore, it is often required to process data online. Therefore, apart from being accurate, the feasible estimation algorithm also needs to be fast. In this paper, we review Bayesian filters that possess the aforementioned properties. Each filter is presented in an easy way to implement algorithmic form. We focus on parametric methods, among which we distinguish three types of filters: filters based on analytical approximations (extended Kalman filter, iterated extended Kalman filter), filters based on statistical approximations (unscented Kalman filter, central difference filter, Gauss-Hermite filter), and filters based on the Gaussian sum approximation (Gaussian sum filter). We discuss each of these filters, and compare them with illustrative examples.

A robust approach for a filter-based monocular simultaneous localization and mapping (SLAM) system

Simultaneous localization and mapping (SLAM) is an important problem to solve in robotics theory in order to build truly autonomous mobile robots. This work presents a novel method for implementing a SLAM system based on a single camera sensor. The SLAM with a single camera, or monocular SLAM, is probably one of the most complex SLAM variants. In this case, a single camera, which is freely moving through its environment, represents the sole sensor input to the system. The sensors have a large impact on the algorithm used for SLAM. Cameras are used more frequently, because they provide a lot of information and are well adapted for embedded systems: they are light, cheap and power-saving. Nevertheless, and unlike range sensors, which provide range and angular information, a camera is a projective sensor providing only angular measurements of image features. Therefore, depth information (range) cannot be obtained in a single step. In this case, special techniques for feature system-initialization are needed in order to enable the use of angular sensors (as cameras) in SLAM systems. The main contribution of this work is to present a novel and robust scheme for incorporating and measuring visual features in filtering-based monocular SLAM systems. The proposed method is based in a two-step technique, which is intended to exploit all the information available in angular measurements. Unlike previous schemes, the values of parameters used by the initialization technique are derived directly from the sensor characteristics, thus simplifying the tuning of the system. The experimental results show that the proposed method surpasses the performance of previous schemes.

A VISION BASED SYSTEM FOR OBJECT DETECTION IN UNDERWATER IMAGES

In this paper, a vision-based system for underwater object detection is presented. The system is able to detect automatically a pipeline placed on the sea bottom, and some objects, e.g. trestles and anodes, placed in its neighborhoods. A color compensation procedure has been introduced in order to reduce problems connected with the light attenuation in the water. Artificial neural networks are then applied in order to classify in real-time the pixels of the input image into different classes, corresponding e.g. to different objects present in the observed scene. Geometric reasoning is applied to reduce the detection of false objects and to improve the accuracy of true detected objects. The results on real underwater images representing a pipeline structure in different scenarios are shown. The presence of seaweed and sand, different illumination conditions and water depth, different pipeline diameter and small variations of the camera tilt angle are considered to evaluate the algorithm performances.

Stages as models of scene geometry

Reconstruction of 3D scene geometry is an important element for scene understanding, autonomous vehicle and robot navigation, image retrieval, and 3D television. We propose accounting for the inherent structure of the visual world when trying to solve the scene reconstruction problem. Consequently, we identify geometric scene categorization as the first step toward robust and efficient depth estimation from single images. We introduce 15 typical 3D scene geometries called stages, each with a unique depth profile, which roughly correspond to a large majority of broadcast video frames. Stage information serves as a first approximation of global depth, narrowing down the search space in depth estimation and object localization. We propose different sets of low-level features for depth estimation, and perform stage classification on two diverse data sets of television broadcasts. Classification results demonstrate that stages can often be efficiently learned from low-dimensional image representations.

Feature Depth Observation for Image-based Visual Servoing: Theory and Experiments

In the classical image-based visual servoing framework, error signals are directly computed from image feature parameters, allowing, in principle, control schemes to be obtained that need neither a complete three-dimensional (3D) model of the scene nor a perfect camera calibration. However, when the computation of control signals involves the interaction matrix, the current value of some 3D parameters is requiredfor each considered feature, and typically a rough approximation of this value is used. With reference to the case of a point feature, for which the relevant 3D parameter is the depth Z, we propose a visual servoing approach where Z is observed and made available for servoing. This is achieved by interpreting depth as an unmeasurable state with known dynamics, and by building a non-linear observer that asymptotically recovers the actual value of Z for the selected feature. A byproduct of our analysis is the rigorous characterization of camera motions that actually allow such observation. Moreover, in the case of a partially uncalibrated camera, it is possible to exploit complementary camera motions in order to preliminarily estimate the focal length without knowing Z. Simulations and experimental results are presented for a mobile robot with an on-board camera in order to illustrate the benefits of integrating the depth observation within classical visual servoing schemes.

Alignment of continuous video onto 3D point clouds

We propose a general framework for aligning continuous (oblique) video onto 3D sensor data. We align a point cloud computed from the video onto the point cloud directly obtained from a 3D sensor. This is in contrast to existing techniques where the 2D images are aligned to a 3D model derived from the 3D sensor data. Using point clouds enables the alignment for scenes full of objects that are difficult to model; for example, trees. To compute 3D point clouds from video, motion stereo is used along with a state-of-the-art algorithm for camera pose estimation. Our experiments with real data demonstrate the advantages of the proposed registration algorithm for texturing models in large-scale semiurban environments. The capability to align video before a 3D model is built from the 3D sensor data offers new practical opportunities for 3D modeling. We introduce a novel modeling-through-registration approach that fuses 3D information from both the 3D sensor and the video. Initial experiments with real data illustrate the potential of the proposed approach.

Statistical processing of large image sequences

The dynamic estimation of large-scale stochastic image sequences, as frequently encountered in remote sensing, is important in a variety of scientific applications. However, the size of such images makes conventional dynamic estimation methods, for example, the Kalman and related filters, impractical. In this paper, we present an approach that emulates the Kalman filter, but with considerably reduced computational and storage requirements. Our approach is illustrated in the context of a 512 x 512 image sequence of ocean surface temperature. The static estimation step, the primary contribution here, uses a mixture of stationary models to accurately mimic the effect of a nonstationary prior, simplifying both computational complexity and modeling. Our approach provides an efficient, stable, positive-definite model which is consistent with the given correlation structure. Thus, the methods of this paper may find application in modeling and single-frame estimation.

Conditional filters for image sequence-based tracking--application to point tracking

In this paper, a new conditional formulation of classical filtering methods is proposed. This formulation is dedicated to image sequence-based tracking. These conditional filters allow solving systems whose measurements and state equation are estimated from the image data. In particular, the model that is considered for point tracking combines a state equation relying on the optical flow constraint and measurements provided by a matching technique. Based on this, two point trackers are derived. The first one is a linear tracker well suited to image sequences exhibiting global-dominant motion. This filter is determined through the use of a new estimator, called the conditional linear minimum variance estimator. The second one is a nonlinear tracker, implemented from a conditional particle filter. It allows tracking of points whose motion may be only locally described. These conditional trackers significantly improve results in some general situations. In particular, they allow for dealing with noisy sequences, abrupt changes of trajectories, occlusions, and cluttered background.

Sampling the disparity space image

A central issue in stereo algorithm design is the choice of matching cost. Many algorithms simply use squared or absolute intensity differences based on integer disparity steps. In this paper, we address potential problems with such approaches. We begin with a careful analysis of the properties of the continuous disparity space image (DSI) and propose several new matching cost variants based on symmetrically matching interpolated image signals. Using stereo images with ground truth, we empirically evaluate the performance of the different cost variants and show that proper sampling can yield improved matching performance.

Probabilistic motion estimation based on temporal coherence

We develop a theory for the temporal integration of visual motion motivated by psychophysical experiments. The theory proposes that input data are temporally grouped and used to predict and estimate the motion flows in the image sequence. This temporal grouping can be considered a generalization of the data association techniques that engineers use to study motion sequences. Our temporal grouping theory is expressed in terms of the Bayesian generalization of standard Kalman filtering. To implement the theory, we derive a parallel network that shares some properties of cortical networks. Computer simulations of this network demonstrate that our theory qualitatively accounts for psychophysical experiments on motion occlusion and motion outliers. In deriving our theory, we assumed spatial factorizability of the probability distributions and made the approximation of updating the marginal distributions of velocity at each point. This allowed us to perform local computations and simplified our implementation. We argue that these approximations are suitable for the stimuli we are considering (for which spatial coherence effects are negligible).

An optimal estimation approach to visual perception and learning

How does the visual system learn an internal model of the external environment? How is this internal model used during visual perception? How are occlusions and background clutter so effortlessly discounted for when recognizing a familiar object? How is a particular object of interest attended to and recognized in the presence of other objects in the field of view? In this paper, we attempt to address these questions from the perspective of Bayesian optimal estimation theory. Using the concept of generative models and the statistical theory of Kalman filtering, we show how static and dynamic events occurring in the visual environment may be learned and recognized given only the input images. We also describe an extension of the Kalman filter model that can handle multiple objects in the field of view. The resulting robust Kalman filter model demonstrates how certain forms of attention can be viewed as an emergent property of the interaction between top-down expectations and bottom-up signals. Experimental results are provided to help demonstrate the ability of such a model to perform robust segmentation and recognition of objects and image sequences in the presence of occlusions and clutter.

A parallel noise-robust algorithm to recover depth information from radial flow fields

A parallel algorithm operating on the units ('neurons') of an artificial retina is proposed to recover depth information in a visual scene from radial flow fields induced by ego motion along a given axis. The system consists of up to 600 radii with fewer than 65 radially arranged neurons on each radius. Neurons are connected only to their nearest neighbors, and they are excited as soon as a sufficiently strong gray-level change occurs. The time difference of two subsequently activated neurons is then used by the last-excited neuron to compute the depth information. All algorithmic calculations remain strictly local, and information is exchanged only between adjacent active neurons (except for the final read-out). This, in principle, permits parallel implementation. Furthermore, it is demonstrated that the calculation of the object coordinates requires only a single multiplication with a constant, which is dependent on only the retinal position of the active neuron. The initial restriction to local operations makes the algorithm very noise sensitive. In order to solve this problem, a predication mechanism is introduced. After an object coordinate has been determined, the active neuron computes the time when the next neuronal excitation should take place. This estimated time is transferred to the respective next neuron, which will wait for this excitation only within a certain time window. If the excitation fails to arrive within this window, the previously computed object coordinate is regarded as noisy and discarded. We will show that this predictive mechanism relies also on only a (second) single multiplication with another neuron-dependent constant. Thus, computational complexity remains low, and noisy depth coordinates are efficiently eliminated. Thus, the algorithm is very fast and operates in real time on 128 x 128 images even in a serial implementation on a relatively slow computer. The algorithm is tested on scenes of growing complexity, and a detailed error analysis is provided showing that the depth error remains very low in most cases. A comparison to standard flow-field analysis shows that our algorithm outperforms the older method by far. The analysis of the algorithm also shows that it is generally applicable despite its restrictions, because it is fast and accurate enough such that a complete depth percept can be composed from radial flow field segments. Finally, we suggest how to generalize the algorithm, waiving the restriction of radial flow.

Dynamic model of visual recognition predicts neural response properties in the visual cortex

The responses of visual cortical neurons during fixation tasks can be significantly modulated by stimuli from beyond the classical receptive field. Modulatory effects in neural responses have also been recently reported in a task where a monkey freely views a natural scene. In this article, we describe a hierarchical network model of visual recognition that explains these experimental observations by using a form of the extended Kalman filter as given by the minimum description length (MDL) principle. The model dynamically combines input-driven bottom-up signals with expectation-driven top-down signals to predict current recognition state. Synaptic weights in the model are adapted in a Hebbian manner according to a learning rule also derived from the MDL principle. The resulting prediction-learning scheme can be viewed as implementing a form of expectation-maximization (EM) algorithm. The architecture of the model posits an active computational role of the reciprocal connections between adjoining visual cortical areas in determining neural response properties. In particular, the model demonstrates the possible role of feedback from higher cortical areas in mediating neurophysiological effects due to stimuli from beyond the classical receptive field. Simulations of the model are provided that help explain the experimental observations regarding neural responses in both free viewing and fixation conditions.

Analog VLSI implementation for stereo correspondence between 2-D images

Many robotics and navigation systems utilizing stereopsis to determine depth have rigid size and power constraints and require direct physical implementation of the stereo algorithm. The main challenges lie in managing the communication between image sensor and image processor arrays, and in parallelizing the computation to determine stereo correspondence between image pixels in real-time. This paper describes the first comprehensive system level demonstration of a dedicated low-power analog VLSI (very large scale integration) architecture for stereo correspondence suitable for real-time implementation. The inputs to the implemented chip are the ordered pixels from a stereo image pair, and the output is a two-dimensional disparity map. The approach combines biologically inspired silicon modeling with the necessary interfacing options for a complete practical solution that can be built with currently available technology in a compact package. Furthermore, the strategy employed considers multiple factors that may degrade performance, including the spatial correlations in images and the inherent accuracy limitations of analog hardware, and augments the design with countermeasures.

Temporal coherence theory for the detection and measurement of visual motion

A recent challenge to the completeness of some influential models of local-motion detection has come from experiments in which subjects had to detect a single dot moving along a trajectory amidst noise dots undergoing Brownian motion. We propose and test a new theory of the detection and measurement of visual motion, which can account for these signal-in-Brownian-noise experiments. The theory postulates that the signals from local-motion detectors are made coherent in space and time by a special purpose network, and that this coherence boosts signals of features moving along non-random trajectories over time. Two experiments were performed to estimate parameters and test the theory. These experiments showed that detection is impaired with increasing eccentricity, an effect that varies inversely with step size. They also showed that detection improves over durations extending to at least 600 msec. An implementation of the theory accounts for these psychophysical detection measurements.