Local reliability weighting explains identification of partially masked objects in natural images

Optimal Visual Search with Highly Heuristic Decision Rules

Visual search is a fundamental natural task for humans and other animals. We investigated the decision processes humans use when searching briefly presented displays having well-separated potential target-object locations. Performance was compared with the Bayesian-optimal decision process under the assumption that the information from the different potential target locations is statistically independent. Surprisingly, humans performed slightly better than optimal, despite humans' substantial loss of sensitivity in the fovea ("foveal neglect"), and the implausibility of the human brain replicating the optimal computations. We show that three factors can quantitatively explain these seemingly paradoxical results. Most importantly, simple and fixed heuristic decision rules reach near optimal search performance. Secondly, foveal neglect primarily affects only the central potential target location. Finally, spatially correlated neural noise causes search performance to exceed that predicted for independent noise. These findings have far-reaching implications for understanding visual search tasks and other identification tasks in humans and other animals.

Behavioral Motor Performance

The human sensorimotor control system has exceptional abilities to perform skillful actions. We easily switch between strenuous tasks that involve brute force, such as lifting a heavy sewing machine, and delicate movements such as threading a needle in the same machine. Using a structure with different control architectures, the motor system is capable of updating its ability to perform through our daily interaction with the fluctuating environment. However, there are issues that make this a difficult computational problem for the brain to solve. The brain needs to control a nonlinear, nonstationary neuromuscular system, with redundant and occasionally undesired degrees of freedom, in an uncertain environment using a body in which information transmission is subject to delays and noise. To gain insight into the mechanisms of motor control, here we survey movement laws and invariances that shape our everyday motion. We then examine the major solutions to each of these problems in the three parts of the sensorimotor control system, sensing, planning, and acting. We focus on how the sensory system, the control architectures, and the structure and operation of the muscles serve as complementary mechanisms to overcome deviations and disturbances to motor behavior and give rise to skillful motor performance. We conclude with possible future research directions based on suggested links between the operation of the sensorimotor system across the movement stages. © 2024 American Physiological Society. Compr Physiol 14:5179-5224, 2024.

Phase-dependent asymmetry of pattern masking in natural images explained by intrinsic position uncertainty

A number of recent studies have been directed at measuring and modeling detection of targets at specific locations in natural backgrounds, a key subtask of visual search in natural environments. A useful approach is to bin natural background patches into joint histograms with bins along specific background dimensions. By measuring psychometric functions in a sparse subset of these bins, it is possible to estimate how the included dimensions jointly affect detectability over the whole space of natural backgrounds. In previous studies, we found that threshold is proportional to the product of the background luminance, contrast, and similarity; a result predicted by a simple template-matching observer with divisive normalization along each of the dimensions. The measure of similarity was the cosine similarity of the amplitude spectra of the target and background (SA)-a phase-invariant measure. Here, we investigated the effect of the cosine similarity of the target and background images (SI|A)-a phase-dependent measure. We found that threshold decreases monotonically with SI|A in agreement with a recent study (Rideaux et al., 2022). In contrast, the template-matching observer predicts threshold to be a U-shaped function of SI|A reaching a minimum when the target and background are orthogonal (SI|A = 0). Surprisingly, when the template-matching observer includes a small amount of intrinsic position uncertainty (measured in a separate experiment) the pattern of thresholds is explained.

Scene saliencies in egocentric vision and their creation by parents and infants

Across the lifespan, humans are biased to look first at what is easy to see, with a handful of well-documented visual saliences shaping our attention (e.g., Itti & Koch, 2001). These attentional biases may emerge from the contexts in which moment-tomoment attention occurs, where perceivers and their social partners actively shape bottom-up saliences, moving their bodies and objects to make targets of interest more salient. The goal of the present study was to determine the bottom-up saliences present in infant egocentric images and to provide evidence on the role that infants and their mature social partners play in highlighting targets of interest via these saliences. We examined 968 unique scenes in which an object had purposefully been placed in the infant's egocentric view, drawn from videos created by one-year-old infants wearing a head camera during toy-play with a parent. To understand which saliences mattered in these scenes, we conducted a visual search task, asking participants (n = 156) to find objects in the egocentric images. To connect this to the behaviors of perceivers, we then characterized the saliences of objects placed by infants or parents compared to objects that were otherwise present in the scenes. Our results show that body-centric properties, such as increases in the centering and visual size of the object, as well as decreases in the number of competing objects immediately surrounding it, both predicted faster search time and distinguished placed and unplaced objects. The present results suggest that the bottom-up saliences that can be readily controlled by perceivers and their social partners may most strongly impact our attention. This finding has implications for the functional role of saliences in human vision, their origin, the social structure of perceptual environments, and how the relation between bottom-up and top-down control of attention in these environments may support infant learning.

Equivalent noise characterization of human lightness constancy

A goal of visual perception is to provide stable representations of task-relevant scene properties (e.g. object reflectance) despite variation in task-irrelevant scene properties (e.g. illumination and reflectance of other nearby objects). To study such stability in the context of the perceptual representation of lightness, we introduce a threshold-based psychophysical paradigm. We measure how thresholds for discriminating the achromatic reflectance of a target object (task-relevant property) in rendered naturalistic scenes are impacted by variation in the reflectance functions of background objects (task-irrelevant property), using a two-alternative forced-choice paradigm in which the reflectance of the background objects is randomized across the two intervals of each trial. We control the amount of background reflectance variation by manipulating a statistical model of naturally occurring surface reflectances. For low background object reflectance variation, discrimination thresholds were nearly constant, indicating that observers' internal noise determines threshold in this regime. As background object reflectance variation increases, its effects start to dominate performance. A model based on signal detection theory allows us to express the effects of task-irrelevant variation in terms of the equivalent noise, that is relative to the intrinsic precision of the task-relevant perceptual representation. The results indicate that although naturally occurring background object reflectance variation does intrude on the perceptual representation of target object lightness, the effect is modest - within a factor of two of the equivalent noise level set by internal noise.

Detection of targets in filtered noise: whitening in space and spatial frequency

Most studies of detection in complex backgrounds have measured and modeled human performance for statistically uniform (stationary) backgrounds. However, natural and medical images have statistical properties that vary over space. We measured detection of various target shapes presented in Gaussian 1/f noise backgrounds that were statistically uniform over space, and in ones that modulated in contrast over space. We find that the pattern of human thresholds is not consistent with the ideal observer but is consistent with a suboptimal observer that performs partial whitening in spatial frequency and whitening (reliability-weighting) in space, and has a small level of intrinsic position uncertainty.

Spatial structure, phase, and the contrast of natural images

The sensitivity of the human visual system is thought to be shaped by environmental statistics. A major endeavor in vision science, therefore, is to uncover the image statistics that predict perceptual and cognitive function. When searching for targets in natural images, for example, it has recently been proposed that target detection is inversely related to the spatial similarity of the target to its local background. We tested this hypothesis by measuring observers' sensitivity to targets that were blended with natural image backgrounds. Targets were designed to have a spatial structure that was either similar or dissimilar to the background. Contrary to masking from similarity, we found that observers were most sensitive to targets that were most similar to their backgrounds. We hypothesized that a coincidence of phase alignment between target and background results in a local contrast signal that facilitates detection when target-background similarity is high. We confirmed this prediction in a second experiment. Indeed, we show that, by solely manipulating the phase of a target relative to its background, the target can be rendered easily visible or undetectable. Our study thus reveals that, in addition to its structural similarity, the phase of the target relative to the background must be considered when predicting detection sensitivity in natural images.

Detection of occluding targets in natural backgrounds

Detection of target objects in the surrounding environment is a common visual task. There is a vast psychophysical and modeling literature concerning the detection of targets in artificial and natural backgrounds. Most studies involve detection of additive targets or of some form of image distortion. Although much has been learned from these studies, the targets that most often occur under natural conditions are neither additive nor distorting; rather, they are opaque targets that occlude the backgrounds behind them. Here, we describe our efforts to measure and model detection of occluding targets in natural backgrounds. To systematically vary the properties of the backgrounds, we used the constrained sampling approach of Sebastian, Abrams, and Geisler (2017). Specifically, millions of calibrated gray-scale natural-image patches were sorted into a 3D histogram along the dimensions of luminance, contrast, and phase-invariant similarity to the target. Eccentricity psychometric functions (accuracy as a function of retinal eccentricity) were measured for four different occluding targets and 15 different combinations of background luminance, contrast, and similarity, with a different randomly sampled background on each trial. The complex pattern of results was consistent across the three subjects, and was largely explained by a principled model observer (with only a single efficiency parameter) that combines three image cues (pattern, silhouette, and edge) and four well-known properties of the human visual system (optical blur, blurring and downsampling by the ganglion cells, divisive normalization, intrinsic position uncertainty). The model also explains the thresholds for additive foveal targets in natural backgrounds reported in Sebastian et al. (2017).