Perceptual spaces: mathematical structures to neural mechanisms

Towards a structural turn in consciousness science

Recent activities in virtually all fields engaged in consciousness studies indicate early signs of a structural turn, where verbal descriptions or simple formalisations of conscious experiences are replaced by structural tools, most notably mathematical spaces. My goal here is to offer three comments that, in my opinion, are essential to avoid misunderstandings in these developments early on. These comments concern metaphysical premises of structural approaches, the viability of structure-preserving mappings, and the question of what a structure of conscious experience is in the first place. I will also explain what, in my opinion, are the great promises of structural methodologies and how they might impact consciousness science at large.

The Geometry of Low- and High-Level Perceptual Spaces

Low-level features are typically continuous (e.g., the gamut between two colors), but semantic information is often categorical (there is no corresponding gradient between dog and turtle) and hierarchical (animals live in land, water, or air). To determine the impact of these differences on cognitive representations, we characterized the geometry of perceptual spaces of five domains: a domain dominated by semantic information (animal names presented as words), a domain dominated by low-level features (colored textures), and three intermediate domains (animal images, lightly texturized animal images that were easy to recognize, and heavily texturized animal images that were difficult to recognize). Each domain had 37 stimuli derived from the same animal names. From 13 participants (9F), we gathered similarity judgments in each domain via an efficient psychophysical ranking paradigm. We then built geometric models of each domain for each participant, in which distances between stimuli accounted for participants' similarity judgments and intrinsic uncertainty. Remarkably, the five domains had similar global properties: each required 5-7 dimensions, and a modest amount of spherical curvature provided the best fit. However, the arrangement of the stimuli within these embeddings depended on the level of semantic information: dendrograms derived from semantic domains (word, image, and lightly texturized images) were more "tree-like" than those from feature-dominated domains (heavily texturized images and textures). Thus, the perceptual spaces of domains along this feature-dominated to semantic-dominated gradient shift to a tree-like organization when semantic information dominates, while retaining a similar global geometry.

Ordinal Characterization of Similarity Judgments

Characterizing judgments of similarity within a perceptual or semantic domain, and making inferences about the underlying structure of this domain from these judgments, has an increasingly important role in cognitive and systems neuroscience. We present a new framework for this purpose that makes very limited assumptions about how perceptual distances are converted into similarity judgments. The approach starts from a dataset of empirical judgments of relative similarities: the fraction of times that a subject chooses one of two comparison stimuli to be more similar to a reference stimulus. These empirical judgments provide Bayesian estimates of underling choice probabilities. From these estimates, we derive three indices that characterize the set of judgments, measuring consistency with a symmetric dis-similarity, consistency with an ultrametric space, and consistency with an additive tree. We illustrate this approach with example psychophysical datasets of dis-similarity judgments in several visual domains and provide code that implements the analyses.

A Psychophysics Paradigm for the Collection and Analysis of Similarity Judgments

Similarity judgments are commonly used to study mental representations and their neural correlates. This approach has been used to characterize perceptual spaces in many domains: colors, objects, images, words, and sounds. Ideally, one might want to compare estimates of perceived similarity between all pairs of stimuli, but this is often impractical. For example, if one asks a subject to compare the similarity of two items with the similarity of two other items, the number of comparisons grows with the fourth power of the stimulus set size. An alternative strategy is to ask a subject to rate similarities of isolated pairs, e.g., on a Likert scale. This is much more efficient (the number of ratings grows quadratically with set size rather than quartically), but these ratings tend to be unstable and have limited resolution, and the approach also assumes that there are no context effects. Here, a novel ranking paradigm for efficient collection of similarity judgments is presented, along with an analysis pipeline (software provided) that tests whether Euclidean distance models account for the data. Typical trials consist of eight stimuli around a central reference stimulus: the subject ranks stimuli in order of their similarity to the reference. By judicious selection of combinations of stimuli used in each trial, the approach has internal controls for consistency and context effects. The approach was validated for stimuli drawn from Euclidean spaces of up to five dimensions. The approach is illustrated with an experiment measuring similarities among 37 words. Each trial yields the results of 28 pairwise comparisons of the form, "Was A more similar to the reference than B was to the reference?" While directly comparing all pairs of pairs of stimuli would have required 221445 trials, this design enables reconstruction of the perceptual space from 5994 such comparisons obtained from 222 trials.

A physicochemical model of odor sampling

We present a general physicochemical sampling model for olfaction, based on established pharmacological laws, in which arbitrary combinations of odorant ligands and receptors can be generated and their individual and collective effects on odor representations and olfactory performance measured. Individual odor ligands exhibit receptor-specific affinities and efficacies; that is, they may bind strongly or weakly to a given receptor, and can act as strong agonists, weak agonists, partial agonists, or antagonists. Ligands interacting with common receptors compete with one another for dwell time; these competitive interactions appropriately simulate the degeneracy that fundamentally defines the capacities and limitations of odorant sampling. The outcome of these competing ligand-receptor interactions yields a pattern of receptor activation levels, thereafter mapped to glomerular presynaptic activation levels based on the convergence of sensory neuron axons. The metric of greatest interest is the mean discrimination sensitivity, a measure of how effectively the olfactory system at this level is able to recognize a small change in the physicochemical quality of a stimulus. This model presents several significant outcomes, both expected and surprising. First, adding additional receptors reliably improves the system’s discrimination sensitivity. Second, in contrast, adding additional ligands to an odorscene initially can improve discrimination sensitivity, but eventually will reduce it as the number of ligands increases. Third, the presence of antagonistic ligand-receptor interactions produced clear benefits for sensory system performance, generating higher absolute discrimination sensitivities and increasing the numbers of competing ligands that could be present before discrimination sensitivity began to be impaired. Finally, the model correctly reflects and explains the modest reduction in odor discrimination sensitivity exhibited by transgenic mice in which the specificity of glomerular targeting by primary olfactory neurons is partially disrupted.

We understand most sensory systems by comparing the responses of the system against objective external physical measurements. For example, we know that our ability to distinguish small changes in color is greater for some colors than for others, and that we can distinguish sounds more acutely when they are within the range of pitches used for speech. Similar principles presumably apply to the sense of smell, but odorous chemicals are harder to physically quantify than light or sound because they cannot be organized in terms of a straightforward physical variable like wavelength or frequency. That said, the physical properties of interactions between chemicals and cellular receptors (such as those in the olfactory system) are well understood. What we lack is a systematic framework within which these pharmacological principles can be organized to study odor sampling in the way that we have long studied visual and auditory sampling. We here propose and describe such a framework for odor sampling, and show that it successfully replicates some established but unexplained experimental results.

Geometrical structure of perceptual color space: Mental representations and adaptation invariance

Similarity between percepts and concepts is used to accomplish many everyday tasks, e.g., object identification; so this similarity is widely used to construct geometrical spaces that represent stimulus qualities, but the intrinsic validity of the geometry, i.e., whether similarity operations support a particular geometry, is almost never tested critically. We introduce an experimental approach for equating relative similarities by setting perceived midpoints between pairs of stimuli. Midpoint settings are used with Varignon's Theorem to test the intrinsic geometry of a representation space, and its mapping to a physical space of stimuli. For perceptual color space, we demonstrate that geometrical structure depends on the mental representation used in judging similarity: An affine geometry was valid when observers used an opponent-color mental representation. Similarities based on a conceptual space of complementary colors thus power a geometric coordinate system. An affine geometry implies that similarity can be judged within straight lines and across parallel lines, and its neural coding could involve ratios of responses. We show that this perceptual space is invariant to changes in illumination color, providing a formal justification to generalize color constancy results measured for color categories, to all of color space. The midpoint measurements deviate significantly from midpoints in the extensively used "uniform" color spaces CIELAB and CIELUV, showing that these spaces do not provide adequate metric representation of perceived colors. Our paradigm can thus test for intrinsic geometrical assumptions underlying the representation space for many perceptual modalities, and for the extrinsic perceptual geometry of the space of physical stimuli.

A Spike Time-Dependent Online Learning Algorithm Derived From Biological Olfaction

We have developed a spiking neural network (SNN) algorithm for signal restoration and identification based on principles extracted from the mammalian olfactory system and broadly applicable to input from arbitrary sensor arrays. For interpretability and development purposes, we here examine the properties of its initial feedforward projection. Like the full algorithm, this feedforward component is fully spike timing-based, and utilizes online learning based on local synaptic rules such as spike timing-dependent plasticity (STDP). Using an intermediate metric to assess the properties of this initial projection, the feedforward network exhibits high classification performance after few-shot learning without catastrophic forgetting, and includes a none of the above outcome to reflect classifier confidence. We demonstrate online learning performance using a publicly available machine olfaction dataset with challenges including relatively small training sets, variable stimulus concentrations, and 3 years of sensor drift.

The multidimensional brain

Brain activity takes place in three spatial-plus time dimensions. This rather obvious claim has been recently questioned by papers that, taking into account the big data outburst and novel available computational tools, are starting to unveil a more intricate state of affairs. Indeed, various brain activities and their correlated mental functions can be assessed in terms of trajectories embedded in phase spaces of dimensions higher than the canonical ones. In this review, I show how further dimensions may not just represent a convenient methodological tool that allows a better mathematical treatment of otherwise elusive cortical activities, but may also reflect genuine functional or anatomical relationships among real nervous functions. I then describe how to extract hidden multidimensional information from real or artificial neurodata series, and make clear how our mind dilutes, rather than concentrates as currently believed, inputs coming from the environment. Finally, I argue that the principle "the higher the dimension, the greater the information" may explain the occurrence of mental activities and elucidate the mechanisms of human diseases associated with dimensionality reduction.

Patterns of recognition of Arabic consonants by non-native children with cochlear implants and normal hearing

OBJECTIVE

This study examined the patterns of recognition of Arabic consonants, via information transmission analysis for phonological features, in a group of Malay children with normal hearing (NH) and cochlear implants (CI).

METHOD

A total of 336 and 616 acoustic tokens were collected from six CI and 11 NH Malay children, respectively. The groups were matched for hearing age and duration of exposure to Arabic sounds. All the 28 Arabic consonants in the form of consonant-vowel /a/ were presented randomly twice via a loudspeaker at approximately 65 dB SPL. The participants were asked to repeat verbally the stimulus heard in each presentation.

RESULTS

Within the native Malay perceptual space, the two groups responded differently to the Arabic consonants. The dispersed uncategorized assimilation in the CI group was distinct in the confusion matrix (CM), as compared to the NH children. Consonants /ħ/, /tˁ/, /sˁ/ and /ʁ/ were difficult for the CI children, while the most accurate item was /k/ (84%). The CI group transmitted significantly reduced information, especially for place feature transmission, then the NH group (p < 0.001). Significant interactions between place-hearing status and manner-hearing status were also obtained, suggesting there were information transmission differences in the pattern of consonants recognition between the study groups.

CONCLUSION

CI and NH Malay children may be using different acoustic cues to recognize Arabic sounds, which contribute to the different assimilation categories' patterns within the Malay perceptual space.

Cascaded Amplitude Modulations in Sound Texture Perception

Sound textures, such as crackling fire or chirping crickets, represent a broad class of sounds defined by their homogeneous temporal structure. It has been suggested that the perception of texture is mediated by time-averaged summary statistics measured from early auditory representations. In this study, we investigated the perception of sound textures that contain rhythmic structure, specifically second-order amplitude modulations that arise from the interaction of different modulation rates, previously described as “beating” in the envelope-frequency domain. We developed an auditory texture model that utilizes a cascade of modulation filterbanks that capture the structure of simple rhythmic patterns. The model was examined in a series of psychophysical listening experiments using synthetic sound textures—stimuli generated using time-averaged statistics measured from real-world textures. In a texture identification task, our results indicated that second-order amplitude modulation sensitivity enhanced recognition. Next, we examined the contribution of the second-order modulation analysis in a preference task, where the proposed auditory texture model was preferred over a range of model deviants that lacked second-order modulation rate sensitivity. Lastly, the discriminability of textures that included second-order amplitude modulations appeared to be perceived using a time-averaging process. Overall, our results demonstrate that the inclusion of second-order modulation analysis generates improvements in the perceived quality of synthetic textures compared to the first-order modulation analysis considered in previous approaches.

Two representations of a high-dimensional perceptual space

A perceptual space is a mental workspace of points in a sensory domain that supports similarity and difference judgments and enables further processing such as classification and naming. Perceptual spaces are present across sensory modalities; examples include colors, faces, auditory textures, and odors. Color is perhaps the best-studied perceptual space, but it is atypical in two respects. First, the dimensions of color space are directly linked to the three cone absorption spectra, but the dimensions of generic perceptual spaces are not as readily traceable to single-neuron properties. Second, generic perceptual spaces have more than three dimensions. This is important because representing each distinguishable point in a high-dimensional space by a separate neuron or population is unwieldy; combinatorial strategies may be needed to overcome this hurdle. To study the representation of a complex perceptual space, we focused on a well-characterized 10-dimensional domain of visual textures. Within this domain, we determine perceptual distances in a threshold task (segmentation) and a suprathreshold task (border salience comparison). In N=4 human observers, we find both quantitative and qualitative differences between these sets of measurements. Quantitatively, observers' segmentation thresholds were inconsistent with their uncertainty determined from border salience comparisons. Qualitatively, segmentation thresholds suggested that distances are determined by a coordinate representation with Euclidean geometry. Border salience comparisons, in contrast, indicated a global curvature of the space, and that distances are determined by activity patterns across broadly tuned elements. Thus, our results indicate two representations of this perceptual space, and suggest that they use differing combinatorial strategies.

SIGNIFICANCE STATEMENT

To move from sensory signals to decisions and actions, the brain carries out a sequence of transformations. An important stage in this process is the construction of a "perceptual space" - an internal workspace of sensory information that captures similarities and differences, and enables further processing, such as classification and naming. Perceptual spaces for color, faces, visual and haptic textures and shapes, sounds, and odors (among others) are known to exist. How such spaces are represented is at present unknown. Here, using visual textures as a model, we investigate this. Psychophysical measurements suggest roles for two combinatorial strategies: one based on projections onto coordinate-like axes, and one based on patterns of activity across broadly tuned elements scattered throughout the space.

The functional role of all postsynaptic potentials examined from a first-person frame of reference

When assigning a central role to the neuronal firing, a large number of incoming postsynaptic potentials not utilized during both supra- and subthreshold neuronal activations are not given any functional significance. Local synaptic potentials at the apical dendrites get attenuated as they arrive at the soma to nearly a twentieth of what a synapse proximal to the soma produces. Conservation of these functions necessitates searching for their functional roles. Potentials induced at the postsynapses of neurons of all the neuronal orders activated by sensory inputs carry small bits of sensory information. The activation of these postsynapses by any means other than the activation from their corresponding presynaptic terminals, that also contribute to oscillating potentials, induce the semblance of the arrival of activity from their presynaptic terminals. This is a candidate mechanism for inducing the first-person internal sensory elements of various higher brain functions as a systems property. They also contribute to the firing of subthreshold-activated neurons, including motor neurons. Operational mechanism of inter-postsynaptic functional LINKs can provide necessary structural requirements for these functions. The functional independence of the distal dendritic compartment and recent evidence for in vivo dendritic spikes indicate their independent role in the formation of internal sensory elements. In these contexts, a neuronal soma is flanked by a large number of quasi-functional internal sensory processing units operated using very little energy, even when a neuron is not firing. A large number of possible combinations of internal sensory units explains the corresponding number of specific memory retrievals by the system in response to various cue stimuli.

A framework for the first-person internal sensation of visual perception in mammals and a comparable circuitry for olfactory perception in Drosophila

Perception is a first-person internal sensation induced within the nervous system at the time of arrival of sensory stimuli from objects in the environment. Lack of access to the first-person properties has limited viewing perception as an emergent property and it is currently being studied using third-person observed findings from various levels. One feasible approach to understand its mechanism is to build a hypothesis for the specific conditions and required circuit features of the nodal points where the mechanistic operation of perception take place for one type of sensation in one species and to verify it for the presence of comparable circuit properties for perceiving a different sensation in a different species. The present work explains visual perception in mammalian nervous system from a first-person frame of reference and provides explanations for the homogeneity of perception of visual stimuli above flicker fusion frequency, the perception of objects at locations different from their actual position, the smooth pursuit and saccadic eye movements, the perception of object borders, and perception of pressure phosphenes. Using results from temporal resolution studies and the known details of visual cortical circuitry, explanations are provided for (a) the perception of rapidly changing visual stimuli, (b) how the perception of objects occurs in the correct orientation even though, according to the third-person view, activity from the visual stimulus reaches the cortices in an inverted manner and (c) the functional significance of well-conserved columnar organization of the visual cortex. A comparable circuitry detected in a different nervous system in a remote species-the olfactory circuitry of the fruit fly Drosophila melanogaster-provides an opportunity to explore circuit functions using genetic manipulations, which, along with high-resolution microscopic techniques and lipid membrane interaction studies, will be able to verify the structure-function details of the presented mechanism of perception.

A perceptual space of local image statistics

Local image statistics are important for visual analysis of textures, surfaces, and form. There are many kinds of local statistics, including those that capture luminance distributions, spatial contrast, oriented segments, and corners. While sensitivity to each of these kinds of statistics have been well-studied, much less is known about visual processing when multiple kinds of statistics are relevant, in large part because the dimensionality of the problem is high and different kinds of statistics interact. To approach this problem, we focused on binary images on a square lattice - a reduced set of stimuli which nevertheless taps many kinds of local statistics. In this 10-parameter space, we determined psychophysical thresholds to each kind of statistic (16 observers) and all of their pairwise combinations (4 observers). Sensitivities and isodiscrimination contours were consistent across observers. Isodiscrimination contours were elliptical, implying a quadratic interaction rule, which in turn determined ellipsoidal isodiscrimination surfaces in the full 10-dimensional space, and made predictions for sensitivities to complex combinations of statistics. These predictions, including the prediction of a combination of statistics that was metameric to random, were verified experimentally. Finally, check size had only a mild effect on sensitivities over the range from 2.8 to 14min, but sensitivities to second- and higher-order statistics was substantially lower at 1.4min. In sum, local image statistics form a perceptual space that is highly stereotyped across observers, in which different kinds of statistics interact according to simple rules.