What is the link between language and spatial images? Behavioral and neural findings in blind and sighted individuals

The Topo-Speech sensory substitution system as a method of conveying spatial information to the blind and vision impaired

Humans, like most animals, integrate sensory input in the brain from different sensory modalities. Yet humans are distinct in their ability to grasp symbolic input, which is interpreted into a cognitive mental representation of the world. This representation merges with external sensory input, providing modality integration of a different sort. This study evaluates the Topo-Speech algorithm in the blind and visually impaired. The system provides spatial information about the external world by applying sensory substitution alongside symbolic representations in a manner that corresponds with the unique way our brains acquire and process information. This is done by conveying spatial information, customarily acquired through vision, through the auditory channel, in a combination of sensory (auditory) features and symbolic language (named/spoken) features. The Topo-Speech sweeps the visual scene or image and represents objects' identity by employing naming in a spoken word and simultaneously conveying the objects' location by mapping the x-axis of the visual scene or image to the time it is announced and the y-axis by mapping the location to the pitch of the voice. This proof of concept study primarily explores the practical applicability of this approach in 22 visually impaired and blind individuals. The findings showed that individuals from both populations could effectively interpret and use the algorithm after a single training session. The blind showed an accuracy of 74.45%, while the visually impaired had an average accuracy of 72.74%. These results are comparable to those of the sighted, as shown in previous research, with all participants above chance level. As such, we demonstrate practically how aspects of spatial information can be transmitted through non-visual channels. To complement the findings, we weigh in on debates concerning models of spatial knowledge (the persistent, cumulative, or convergent models) and the capacity for spatial representation in the blind. We suggest the present study's findings support the convergence model and the scenario that posits the blind are capable of some aspects of spatial representation as depicted by the algorithm comparable to those of the sighted. Finally, we present possible future developments, implementations, and use cases for the system as an aid for the blind and visually impaired.

Temporal visual representation elicits early auditory-like responses in hearing but not in deaf individuals

It is evident that the brain is capable of large-scale reorganization following sensory deprivation, but the extent of such reorganization is to date, not clear. The auditory modality is the most accurate to represent temporal information, and deafness is an ideal clinical condition to study the reorganization of temporal representation when the audio signal is not available. Here we show that hearing, but not deaf individuals, show a strong ERP response to visual stimuli in temporal areas during a time-bisection task. This ERP response appears 50-90 ms after the flash and recalls some aspects of the N1 ERP component usually elicited by auditory stimuli. The same ERP is not evident for a visual space-bisection task, suggesting that the early recruitment of temporal cortex is specific for building a highly resolved temporal representation within the visual modality. These findings provide evidence that the lack of auditory input can interfere with typical development of complex visual temporal representations.

Superior verbal but not nonverbal memory in congenital blindness

Previous studies suggest that people who are congenitally blind outperform sighted people on some memory tasks. Whether blindness-associated memory advantages are specific to verbal materials or are also observed with nonverbal sounds has not been determined. Congenitally blind individuals (n = 20) and age and education matched blindfolded sighted controls (n = 22) performed a series of auditory memory tasks. These included: verbal forward and backward letter spans, a complex letter span with intervening equations, as well as two matched recognition tasks: one with verbal stimuli (i.e., letters) and one with nonverbal complex meaningless sounds. Replicating previously observed findings, blind participants outperformed sighted people on forward and backward letter span tasks. Blind participants also recalled more letters on the complex letter span task despite the interference of intervening equations. Critically, the same blind participants showed larger advantages on the verbal as compared to the nonverbal recognition task. These results suggest that blindness selectively enhances memory for verbal material. Possible explanations for blindness-related verbal memory advantages include blindness-induced memory practice and 'visual' cortex recruitment for verbal processing.

The neural correlates for spatial language: Perspective-dependent and -independent relationships in American Sign Language and spoken English

In American Sign Language (ASL) spatial relationships are conveyed by the location of the hands in space, whereas English employs prepositional phrases. Using event-related fMRI, we examined comprehension of perspective-dependent (PD) (left, right) and perspective-independent (PI) (in, on) sentences in ASL and audiovisual English (sentence-picture matching task). In contrast to non-spatial control sentences, PD sentences engaged the superior parietal lobule (SPL) bilaterally for ASL and English, consistent with a previous study with written English. The ASL-English conjunction analysis revealed bilateral SPL activation for PD sentences, but left-lateralized activation for PI sentences. The direct contrast between PD and PI expressions revealed greater SPL activation for PD expressions only for ASL. Increased SPL activation for ASL PD expressions may reflect the mental transformation required to interpret locations in signing space from the signer's viewpoint. Overall, the results suggest both overlapping and distinct neural regions support spatial language comprehension in ASL and English.

Visual representations of time elicit early responses in human temporal cortex

Time perception is inherently part of human life. All human sensory modalities are always involved in the complex task of creating a temporal representation of the external world. However, when representing time, people primarily rely on auditory information. Since the auditory system prevails in many audio-visual temporal tasks, one may expect that the early recruitment of the auditory network is necessary for building a highly resolved and flexible temporal representation in the visual modality. To test this hypothesis, we asked 17 healthy participants to temporally bisect three consecutive flashes while we recorded EEG. We demonstrated that visual stimuli during temporal bisection elicit an early (50-90 ​ms) response of an extended area of the temporal cortex, likely including auditory cortex too. The same activation did not appear during an easier spatial bisection task. These findings suggest that the brain may use auditory representations to deal with complex temporal representation in the visual system.

Mental Rotation of Digitally-Rendered Haptic Objects by the Visually-Impaired

In the event of visual impairment or blindness, information from other intact senses can be used as substitutes to retrain (and in extremis replace) visual functions. Abilities including reading, mental representation of objects and spatial navigation can be performed using tactile information. Current technologies can convey a restricted library of stimuli, either because they depend on real objects or renderings with low resolution layouts. Digital haptic technologies can overcome such limitations. The applicability of this technology was previously demonstrated in sighted participants. Here, we reasoned that visually-impaired and blind participants can create mental representations of letters presented haptically in normal and mirror-reversed form without the use of any visual information, and mentally manipulate such representations. Visually-impaired and blind volunteers were blindfolded and trained on the haptic tablet with two letters (either L and P or F and G). During testing, they haptically explored on any trial one of the four letters presented at 0°, 90°, 180°, or 270° rotation from upright and indicated if the letter was either in a normal or mirror-reversed form. Rotation angle impacted performance; greater deviation from 0° resulted in greater impairment for trained and untrained normal letters, consistent with mental rotation of these haptically-rendered objects. Performance was also generally less accurate with mirror-reversed stimuli, which was not affected by rotation angle. Our findings demonstrate, for the first time, the suitability of a digital haptic technology in the blind and visually-impaired. Classic devices remain limited in their accessibility and in the flexibility of their applications. We show that mental representations can be generated and manipulated using digital haptic technology. This technology may thus offer an innovative solution to the mitigation of impairments in the visually-impaired, and to the training of skills dependent on mental representations and their spatial manipulation.

Blindness, Psychosis, and the Visual Construction of the World

The relationship between visual loss and psychosis is complex: congenital visual loss appears to be protective against the development of a psychotic disorder, particularly schizophrenia. In later life, however, visual deprivation or visual loss can give rise to hallucinosis, disorders of visual insight such as blindsight or Anton syndrome, or, in the context of neurodegenerative disorders, more complex psychotic presentations. We draw on a computational psychiatric approach to consider the foundational role of vision in the construction of representations of the world and the effects of visual loss at different developmental stages. Using a Bayesian prediction error minimization model, we describe how congenital visual loss may be protective against the development of the kind of computational deficits postulated to underlie schizophrenia, by increasing the precision (and consequent stability) of higher-level (including supramodal) priors, focusing on visual loss-induced changes in NMDA receptor structure and function as a possible mechanistic substrate. In simple terms, we argue that when people cannot see from birth, they rely more heavily on the context they extract from the other senses, and the resulting model of the world is more impervious to the false inferences, made in the face of inevitably noisy perceptual input, that characterize schizophrenia. We show how a Bayesian prediction error minimization framework can also explain the relationship between later visual loss and other psychotic symptoms, as well as the effects of visual deprivation and hallucinogenic drugs, and outline experimentally testable hypotheses generated by this approach.

What Do Spatial Distortions in Patients' Drawing After Right Brain Damage Teach Us About Space Representation in Art?

The right cerebral hemisphere plays a crucial rule in spatial cognition, spanning from perception of elementary features, such as location, color, line orientation or shape to representation of different spaces (3D space, allocentric, egocentric, face, personal, peri-personal, or imaginal). One important aspect of its contribution concerns the perception of space symmetry and the representation of objects and scenes, with reference to the midline or body axis. This representation results from a balance between spatial attention processes depending from the two hemispheres. Healthy participants tend to show a discrete deviation of the midline plane representation toward the left side, that is likely to result from the predominance of the activity of the right cerebral hemisphere, mainly oriented toward the contralateral side of space. The visuospatial abilities of the right hemisphere, especially for the representation of the midline plane are crucially engaged in painting and drawing processes in artists. Interestingly, the distortions created by painters of the Cubism period, characterized by an asymmetry of objects and body representations, a specific enlargement or reduction of parts of space, or even by complex distortions of 3D space are analogous to those classically reported in right-brain-damaged patients (unilateral spatial neglect, hyperschematia, constructional apraxia). Understanding the pathological mechanisms of these representation disorders provides meaningful information to apprehend visual artist creations and esthetic perception of space.

On my right or on your left? Spontaneous spatial perspective taking in blind people

Spatial perspective taking is a human ability that permits to assume another person's spatial viewpoint. Data show that spatial perspective taking might arise even spontaneously by the mere presence of another person in the environment. We investigated whether this phenomenon is observable also in blind people. Blind and blindfolded sighted participants explored a tridimensional tactile map and memorized the localization of different landmarks. Then, after the presentation of sounds coming from three landmarks-positioned on the right, on the left, and in front-participants had to indicate the reciprocal position of the two lateral landmarks. Results showed that when the sound coming from the frontal landmark suggested the presence of a speaking (voice) or moving person (footsteps), several blind and sighted people adopted this person's perspective. These findings suggest that auditory stimuli can trigger spontaneous spatial perspective taking in sighted as well as in blind people.

Mixed metaphors: Electrophysiological brain responses to (un)expected concrete and abstract prepositional phrases

Languages around the world use spatial terminology, like prepositions, to describe non-spatial, abstract concepts, including time (e.g., in the moment). The Metaphoric Mapping Theory explains this pattern by positing that a universal human cognitive process underlies it, whereby abstract concepts are conceptualized via the application of concrete, three-dimensional space onto abstract domains. The alternative view is that the use of spatial propositions in abstract phrases is idiomatic, and thus does not trigger metaphoric mapping. In the current study, event-related potentials (ERPs) were used to examine the time-course of neural processing of concrete and abstract phrases consisting of the prepositions in or on followed by congruent and incongruent nouns (e.g., in the bowl/plate and in the moment/mend). ERPs were recorded from the onset of reference nouns in 28 adult participants using a 128-channel electrode net. Results show that congruency has differential effects on neural measures, depending on whether the noun is concrete or abstract. Incongruent reference nouns in concrete phrases (e.g., on the bowl) elicited a significant central negativity (an N400 effect), while incongruent reference nouns in abstract phrases (e.g., on the moment) did not. These results suggest that spatially incongruent concrete nouns are semantically unexpected (N400 effect). A P600 effect, which might indicate rechecking, reanalysis and/or reconstruction, was predicted for incongruent abstract nouns, but was not observed, possibly due to the variability in abstract stimuli. Findings cast doubt on accounts claiming that abstract uses of prepositions are cognitively and metaphorically linked to their spatial sense during natural, on-line processing.

Einfühlung as the breath of art: six modes of embodiment

Robert Vischer's concept of Einfühlung, feeling-into, translated as empathy, serves as the departure point for a proposal about viewing art using the body for a non-imitative form of empathy termed a transomatization and for other embodied operations. A transomatization occurs when viewers reinterpret a component or process of their own bodies to serve as a non-imitative stand-in, or correlate, for something outside of the self, specifically, some quality of an art work or its production. This creates an overlap of the self and other that might be experienced subjectively as a feeling of projection, an operation characteristic of empathy. Transomatizations and other embodied experiences are grounded in empathic, intersubjective modes of engaging others that begin in early life. As applications of the proposed concepts, six different embodiments of the viewer's breathing are explored in regard to Friedrich E. Church's 1848 oil painting Morning, Looking East over the Hudson Valley from the Catskill Mountains. Support for elements of the proposed concepts and applications is drawn from research in the biological and social sciences and from first person, embodied accounts of viewing.

Blindness and social trust: The effect of early visual deprivation on judgments of trustworthiness

Investigating the impact of early visual deprivation on evaluations related to social trust has received little attention to date. This is despite consistent evidence suggesting that early onset blindness may interfere with the normal development of social skills. In this study, we investigated whether early blindness affects judgments of trustworthiness regarding the actions of an agent, with trustworthiness representing the fundamental dimension in the social evaluation. Specifically, we compared performance between a group of early blind individuals with that of sighted controls in their evaluation of trustworthiness of an agent after hearing a pair of two positive or two negative social behaviors (impression formation). Participants then repeated the same evaluation following the presentation of a third (consistent or inconsistent) behavior regarding the same agent (impression updating). Overall, blind individuals tended to give similar evaluations compared to their sighted counterparts. However, they also valued positive behaviors significantly more than sighted controls when forming their impression of an agent's trustworthiness. Moreover, when inconsistent information was provided, blind individuals were more prone to revise their initial evaluation compared to controls. These results suggest that early visual deprivation may have a dramatic effect on the evaluation of social factors such as trustworthiness.

Spatial imagery relies on a sensory independent, though sensory sensitive, functional organization within the parietal cortex: a fMRI study of angle discrimination in sighted and congenitally blind individuals

Although vision offers distinctive information to space representation, individuals who lack vision since birth often show perceptual and representational skills comparable to those found in sighted individuals. However, congenitally blind individuals may result in impaired spatial analysis, when engaging in 'visual' spatial features (e.g., perspective or angle representation) or complex spatial mental abilities. In the present study, we measured behavioral and brain responses using functional magnetic resonance imaging in sighted and congenitally blind individuals during spatial imagery based on a modified version of the mental clock task (e.g., angle discrimination) and a simple recognition control condition, as conveyed across distinct sensory modalities: visual (sighted individuals only), tactile and auditory. Blind individuals were significantly less accurate during the auditory task, but comparable-to-sighted during the tactile task. As expected, both groups showed common neural activations in intraparietal and superior parietal regions across visual and non-visual spatial perception and imagery conditions, indicating the more abstract, sensory independent functional organization of these cortical areas, a property that we named supramodality. At the same time, however, comparisons in brain responses and functional connectivity patterns across experimental conditions demonstrated also a functional lateralization, in a way that correlated with the distinct behavioral performance in blind and sighted individuals. Specifically, blind individuals relied more on right parietal regions, mainly in the tactile and less in the auditory spatial processing. In sighted, spatial representation across modalities relied more on left parietal regions. In conclusions, intraparietal and superior parietal regions subserve supramodal spatial representations in sighted and congenitally blind individuals. Differences in their recruitment across non-visual spatial processing in sighted and blind individuals may be related to distinctive behavioral performance and/or mental strategies adopted when they deal with the same spatial representation as conveyed through different sensory modalities.

Nonvisual spatial navigation fMRI lateralizes mesial temporal lobe epilepsy in a patient with congenital blindness

Nonvisual spatial navigation functional magnetic resonance imaging (fMRI) may help clinicians determine memory lateralization in blind individuals with refractory mesial temporal lobe epilepsy (MTLE). We report on an exceptional case of a congenitally blind woman with late-onset left MTLE undergoing presurgical memory fMRI. To activate mesial temporal structures despite the lack of visual memory, the patient was requested to recall familiar routes using nonvisual multisensory and verbal cues. Our findings demonstrate the diagnostic value of a nonvisual fMRI task to lateralize MTLE despite congenital blindness and may therefore contribute to the risk assessment for postsurgical amnesia in rare cases with refractory MTLE and accompanying congenital blindness.

Space-time interdependence: evidence against asymmetric mapping between time and space

Time and space are intimately related, but what is the real nature of this relationship? Is time mapped metaphorically onto space such that effects are always asymmetric (i.e., space affects time more than time affects space)? Or do the two domains share a common representational format and have the ability to influence each other in a flexible manner (i.e., time can sometimes affect space more than vice versa)? In three experiments, we examined whether spatial representations from haptic perception, a modality of relatively low spatial acuity, would lead the effect of time on space to be substantially stronger than the effect of space on time. Participants touched (but could not see) physical sticks while listening to an auditory note, and then reproduced either the length of the stick or the duration of the note. Judgements of length were affected by concurrent stimulus duration, but not vice versa. When participants were allowed to see as well as touch the sticks, however, the higher acuity of visuohaptic perception caused the effects to converge so length and duration influenced each other to a similar extent. These findings run counter to the spatial metaphor account of time, and rather support the spatial representation account in which time and space share a common representational format and the directionality of space-time interaction depends on the perceptual acuity of the modality used to perceive space.

Stepping into a map: initial heading direction influences spatial memory flexibility

Learning a novel environment involves integrating first-person perceptual and motoric experiences with developing knowledge about the overall structure of the surroundings. The present experiments provide insights into the parallel development of these egocentric and allocentric memories by intentionally conflicting body- and world-centered frames of reference during learning, and measuring outcomes via online and offline measures. Results of two experiments demonstrate faster learning and increased memory flexibility following route perspective reading (Experiment 1) and virtual navigation (Experiment 2) when participants begin exploring the environment on a northward (vs. any other direction) allocentric heading. We suggest that learning advantages due to aligning body-centered (left/right/forward/back) with world-centered (NSEW) reference frames are indicative of three features of spatial memory development and representation. First, memories for egocentric and allocentric information develop in parallel during novel environment learning. Second, cognitive maps have a preferred orientation relative to world-centered coordinates. Finally, this preferred orientation corresponds to traditional orientation of physical maps (i.e., north is upward), suggesting strong associations between daily perceptual and motor experiences and the manner in which we preferentially represent spatial knowledge.

Hyperbole, abstract motion and spatial knowledge: sequential versus simultaneous scanning

Hyperbole is an interesting trope in the perspective of Space Grammar, since it is related to the displacing of a limit (Lausberg in Elemente der literarischen Rhetorik. M.H. Verlag, Munchen 1967; see the Ancient Greek meaning 'to throw over' > 'exaggerate'). Hyperbole semantic mechanisms are related to virtual scanning (Holmqvist and Płuciennik in Imagery in language. Peter Lang, Frankfurt am Main, pp 777-785, 2004). Basic concepts of SIZE and QUANTITY, related image-schemas (IS) and conceptual metaphors (UP IS MORE; IMPORTANT IS BIG: Lakoff 1987, Johnson 1987) are implied in hyperbole processing. The virtual scanning is the simulation of a perceptual domain (here, the vertically oriented space). The virtual limit is defined by expected values on the relevant scale. Since hyperbole is a form of intensification, its linguistic interest lies in cases involving the extremes of a scale, for which a limit can be determined (Schemann 1994). In this experimental study, we analyze the concept of 'limit' in terms of 'abstract motion' and 'oriented space' domains (Langacker 1990) with respect to hyperboles expressed by Italian Verbs of movement. The IS considered are PATH and SOURCE-PATH-GOAL. The latter corresponds to a virtual scale whose limit is arrived at, or overcome, in hyperboles.

The premotor theory of attention as an account for the Simon effect

The Simon effect refers to the phenomenon that responses are faster when the irrelevant location of a stimulus corresponds with the response location than when these locations do not correspond. In the current paper we examined the viability of an updated version of the premotor theory of attention (PMTA) as an account for the Simon effect. Two predictions were evaluated. First, in the case of focused attention at the relevant target position a strong reduction of the Simon effect should be observed as the Simon effect according to PMTA crucially depends on attentional orienting. Secondly, if attention is directed towards a location then this orienting by itself should already be sufficient for producing a Simon effect, as stimulus presence is not required. Our data confirmed these predictions thereby supporting the relevance of the PMTA for the Simon effect.

Spatial language processing in the blind: evidence for a supramodal representation and cortical reorganization

Neuropsychological and imaging studies have shown that the left supramarginal gyrus (SMG) is specifically involved in processing spatial terms (e.g. above, left of), which locate places and objects in the world. The current fMRI study focused on the nature and specificity of representing spatial language in the left SMG by combining behavioral and neuronal activation data in blind and sighted individuals. Data from the blind provide an elegant way to test the supramodal representation hypothesis, i.e. abstract codes representing spatial relations yielding no activation differences between blind and sighted. Indeed, the left SMG was activated during spatial language processing in both blind and sighted individuals implying a supramodal representation of spatial and other dimensional relations which does not require visual experience to develop. However, in the absence of vision functional reorganization of the visual cortex is known to take place. An important consideration with respect to our finding is the amount of functional reorganization during language processing in our blind participants. Therefore, the participants also performed a verb generation task. We observed that only in the blind occipital areas were activated during covert language generation. Additionally, in the first task there was functional reorganization observed for processing language with a high linguistic load. As the visual cortex was not specifically active for spatial contents in the first task, and no reorganization was observed in the SMG, the latter finding further supports the notion that the left SMG is the main node for a supramodal representation of verbal spatial relations.

Prior visual experience, and perception and memory of shape in people with total blindness

The aim of this study was to explore the role of prior visual experience for tactile differentiation of object shapes. The study investigated whether people who lost their vision later in life were able to identify and recognize object shapes more accurately and faster than those who were blind from their birth. Four experiments were conducted. The first two were concerned with tactile shape differentiation, the second two with shape recognition. The hypotheses were only partially confirmed. The ‘late’ blind participants distinguished shapes more accurately than the congenitally blind (particularly in ‘simple’ perception tasks). This finding may suggest that people who have prior visual experience use an allocentric strategy when visualizing object shapes in their imagery. The ‘late’ blind participants performed the tasks more slowly than those who were congenitally blind. This may be explained by the complexity of the task, the time needed to create an allocentric representation, and discrepancy in the tactile experiences between the congenitally and late blind groups. A number of implications for further research are outlined.

Reference Frame Preferences in Haptics Differ for the Blind and Sighted in the Horizontal but Not in the Vertical Plane

We investigated which reference frames are preferred when matching spatial language to the haptic domain. Sighted, low-vision, and blind participants were tested on a haptic-sentence-verification task where participants had to haptically explore different configurations of a ball and a shoe and judge the relation between them. Results from the spatial relation “above”, in the vertical plane, showed that various reference frames are available after haptic inspection of a configuration. Moreover, the pattern of results was similar for all three groups and resembled patterns found for the sighted on visual sentence-verification tasks. In contrast, when judging the spatial relation “in front”, in the horizontal plane, the blind showed a markedly different response pattern. The sighted and low-vision participants did not show a clear preference for either the absolute/relative or the intrinsic reference frame when these frames were dissociated. The blind, on the other hand, showed a clear preference for the intrinsic reference frame. In the absence of a dominant cue, such as gravity in the vertical plane, the blind might emphasise the functional relationship between the objects owing to enhanced experience with haptic exploration of objects.

Unimodal and crossmodal working memory representations of visual and kinesthetic movement trajectories

The present study investigated whether visual and kinesthetic stimuli are stored as multisensory or modality-specific representations in unimodal and crossmodal working memory tasks. To this end, angle-shaped movement trajectories were presented to 16 subjects in delayed matching-to-sample tasks either visually or kinesthetically during encoding and recognition. During the retention interval, a secondary visual or kinesthetic interference task was inserted either immediately or with a delay after encoding. The modality of the interference task interacted significantly with the encoding modality. After visual encoding, memory was more impaired by a visual than by a kinesthetic secondary task, while after kinesthetic encoding the pattern was reversed. The time when the secondary task had to be performed interacted with the encoding modality as well. For visual encoding, memory was more impaired, when the secondary task had to be performed at the beginning of the retention interval. In contrast, memory after kinesthetic encoding was more affected, when the secondary task was introduced later in the retention interval. The findings suggest that working memory traces are maintained in a modality-specific format characterized by distinct consolidation processes that take longer after kinesthetic than after visual encoding.