Time course of information processing in visual and haptic object classification

Processing of visual hapaxes in picture naming task: An event-related potential study

Object recognition and visual categorization are typically swift and seemingly effortless tasks that involve numerous underlying processes. In our investigation, we utilized a picture naming task to explore the processing of rarely encountered objects (visual hapaxes) in comparison to common objects. Our aim was to determine the stage at which these rare objects are classified as unnamable. Contrary to our expectations and in contrast to some prior research on event-related potentials (ERPs) with novel and atypical objects, no differences between conditions were observed in the late time windows corresponding to the P300 or N400 components. However, distinctive patterns between hapaxes and common objects surfaced in three early time windows, corresponding to the posterior N1 and P2 waves, as well as a widespread N2 wave. According to the ERP data, the differentiation between hapaxes and common objects occurs within the first 380 ms of the processing line, involving only limited and indirect top-down influence.

Haptic and visuo-haptic impairments for object recognition in children with autism spectrum disorder: focus on the sensory and multisensory processing dysfunctions

Dysfunctions in sensory processing are widely described in individuals with autism spectrum disorder (ASD), although little is known about the developmental course and the impact of these difficulties on the learning processes during the preschool and school ages of ASD children. Specifically, as regards the interplay between visual and haptic information in ASD during developmental age, knowledge is very scarce and controversial. In this study, we investigated unimodal (visual and haptic) and cross-modal (visuo-haptic) processing skills aimed at object recognition through a behavioural paradigm already used in children with typical development (TD), with cerebral palsy and with peripheral visual impairments. Thirty-five children with ASD (age range: 5-11 years) and thirty-five age-matched and gender-matched typically developing peers were recruited. The procedure required participants to perform an object-recognition task relying on only the visual modality (black-and-white photographs), only the haptic modality (manipulation of real objects) and visuo-haptic transfer of these two types of information. Results are consistent with the idea that visuo-haptic transfer may be significantly worse in ASD children than in TD peers, leading to significant impairment in multisensory interactions for object recognition facilitation. Furthermore, ASD children tended to show a specific deficit in haptic information processing, while a similar trend of maturation of visual modality between the two groups is reported. This study adds to the current literature by suggesting that ASD differences in multisensory processes also regard visuo-haptic abilities necessary to identify and recognise objects of daily life.

An ensemble deep-learning approach for single-trial EEG classification of vibration intensity

Objective. Single-trial electroencephalography (EEG) classification is a promising approach to evaluate the cognitive experience associated with haptic feedback. Convolutional neural networks (CNNs), which are among the most widely used deep learning techniques, have demonstrated their effectiveness in extracting EEG features for the classification of different cognitive functions, including the perception of vibration intensity that is often experienced during human-computer interaction. This paper proposes a novel CNN ensemble model to classify the vibration-intensity from a single trial EEG data that outperforms the state-of-the-art EEG models.Approach. The proposed ensemble model, named SE NexFusion, builds upon the observed complementary learning behaviors of the EEGNex and TCNet Fusion models, exhibited in learning personal as well generic neural features associated with vibration intensity. The proposed ensemble employs multi-branch feature encoders corroborated with squeeze-and-excitation units that enables rich-feature encoding while at the same time recalibrating the weightage of the obtained feature maps based on their discriminative power. The model takes in a single trial of raw EEG as an input and does not require complex EEG signal-preprocessing.Main results. The proposed model outperforms several state-of-the-art bench-marked EEG models by achieving an average accuracy of 60.7% and 61.6% under leave-one-subject-out and within-subject cross-validation (three-classes), respectively. We further validate the robustness of the model through Shapley values explainability method, where the most influential spatio-temporal features of the model are counter-checked with the neural correlates that encode vibration intensity.Significance. Results show that SE NexFusion outperforms other benchmarked EEG models in classifying the vibration intensity. Additionally, explainability analysis confirms the robustness of the model in attending to features associated with the neural correlates of vibration intensity.

Action and posture influence the retrieval of memory for objects

The embodied approach states that memory traces are retrieved, at least in part, through a sensorimotor simulation of the original events, i.e., during retrieval we use our body and its sensorimotor pathways to simulate what happened during encoding. Thus, body manipulations that are incongruent with the motor elements involved at encoding should modulate memory performance. To test this hypothesis, we devised two experiments. In Experiment 1, participants had to observe (observation task) or observe and also perform an action on (enactment task) a series of objects. At recognition, they recognised the enacted objects faster and more accurately than the observed ones. Crucially, in Experiment 2, we manipulated body posture during recognition: one group was asked to hold their hands/arms in front of them (non-interfering group), and the other group was asked to block their hands/arms behind their back (interfering group). The results on reaction times, but not those on accuracy, showed a critical interaction: while the noninterfering group recognised enacted objects faster than observed objects, this advantage disappeared for the interfering group. This suggests that adopting a posture inconsistent with action at encoding could influence the time needed to correctly recognise the objects, but not the accuracy of the recognition.

Long-term memory of haptic and visual information in older adults

The present study examined haptic and visual memory capacity for familiar objects through the application of an intentional free-recall task with three-time intervals in a sample of 78 healthy older adults without cognitive impairment. A wooden box and a turntable were used for the presentation of haptic and visual stimuli, respectively. The procedure consisted of two phases, a study phase that consisted of the presentation of stimuli, and a test phase (free-recall task) performed after one hour, one day or one week. The analysis of covariance (ANCOVA) indicated that there was a main effect only for the time intervals (F (2,71) = 12.511, p = .001, η² = 0.261), with a lower recall index for the interval of one week compared to the other intervals. We concluded that the memory capacity between the systems (haptic and visual) is similar for long retrieval intervals (hours to days).

Behavioral and electrophysiological correlates of interactions between grouping principles in touch: Evidence from psychophysical indirect tasks

In two experiments we investigated the behavioral and brain correlates of the interactions between spatial-proximity and texture-similarity grouping principles in touch. We designed two adaptations of the repetition discrimination task (RDT) previously used in vision. This task provides an indirect measure of grouping that does not require explicit attention to the grouping process. In Experiment 1, participants were presented with a row of elements alternating in texture except for one pair in which the same texture was repeated. The participants had to decide whether the repeated texture stimuli (similarity grouping) were smooth or rough, while the spatial proximity between targets and distractors was varied either to facilitate or hinder the response. In Experiment 2, participants indicated which cohort (proximity grouping) contained more elements, while texture-similarity within and between cohorts was modified. The results indicated additive effects of grouping cues in which proximity dominated the perceptual grouping process when the two principles acted together. In addition, the independent component analysis (ICA) performed on electrophysiological data revealed the implication of a widespread network of sensorimotor, prefrontal, parietal and occipital brain areas in both experiments.

Long-Term Memory for Haptically Explored Objects: Fidelity, Durability, Incidental Encoding, and Cross-Modal Transfer

The question of how many of our perceptual experiences are stored in long-term memory has received considerable attention. The present study examined long-term memory for haptic experiences. Blindfolded participants haptically explored 168 everyday objects (e.g., a pen) for 10 s each. In a blindfolded memory test, they indicated which of two objects from the same basic-level category (e.g., two different pens) had been touched before. As shown in Experiment 1 (N = 26), memory was nearly perfect when tested immediately after exploration (94%) and still high when tested after 1 week (85%). As shown in Experiment 2 (N = 43), when participants explored the objects without the intention to memorize them, memory in a 1-week delayed surprise test was still high (79%), even when assessed with a cross-modal visual memory test (73%). These results indicate that detailed, durable, long-term memory representations are stored as a natural product of haptic perception.

Alpha and beta band correlates of haptic perceptual grouping: Results from an orientation detection task

Behavioral and neurophysiological findings in vision suggest that perceptual grouping is not a unitary process and that different grouping principles have different processing requirements and neural correlates. The present study aims to examine whether the same occurs in the haptic modality using two grouping principles widely studied in vision, spatial proximity and texture similarity. We analyzed behavioral responses (accuracy and response times) and conducted an independent component analysis of brain oscillations in alpha and beta bands for haptic stimuli grouped by spatial proximity and texture similarity, using a speeded orientation detection task performed on a novel haptic device (MonHap). Behavioral results showed faster response times for patterns grouped by spatial proximity relative to texture similarity. Independent component clustering analysis revealed the activation of a bilateral network of sensorimotor and parietal areas while performing the task. We conclude that, as occurs in visual perception, grouping the elements of the haptic scene by means of their spatial proximity is faster than forming the same objects by means of texture similarity. In addition, haptic grouping seems to involve the activation of a network of widely distributed bilateral sensorimotor and parietal areas as reflected by the consistent event-related desynchronization found in alpha and beta bands.

The role of contour polarity, objectness, and regularities in haptic and visual perception

Regularities like symmetry (mirror reflection) and repetition (translation) play an important role in both visual and haptic (active touch) shape perception. Altering figure-ground factors to change what is perceived as an object influences regularity detection. For vision, symmetry is usually easier to detect within one object, whereas repetition is easier to detect across two objects. For haptics, we have not found this interaction between regularity type and objectness (Cecchetto & Lawson, Journal of Experimental Psychology: Human Perception and Performance, 43, 103-125, 2017; Lawson, Ajvani, & Cecchetto, Experimental Psychology, 63, 197-214, 2016). However, our studies used repetition stimuli with mismatched concavities, convexities, and luminance, and so had mismatched contour polarities. Such stimuli may be processed differently to stimuli with matching contour polarities. We investigated this possibility. For haptics, speeded symmetry and repetition detection for novel, planar shapes was similar. Performance deteriorated strikingly if contour polarity mismatched (keeping objectness constant), whilst there was a modest disadvantage for between-2objects:facing-sides compared to within-1object:outer-sides comparisons (keeping contour polarity constant). For the same task for vision, symmetry detection was similar to haptics (strong costs for mismatched contour polarity, weaker costs for between-2objects:facing-sides comparisons), but repetition detection was very different (weak costs for mismatched contour polarity, strong benefits for between-2objects:facing-sides comparisons). Thus, objectness was less influential than contour polarity for both haptic and visual symmetry detection, and for haptic repetition detection. However, for visual repetition detection, objectness effects reversed direction (within-1object:outer-sides comparisons were harder) and were stronger than contour polarity effects. This pattern of results suggests that regularity detection reflects information extraction as well as regularity distributions in the physical world.

Haptic recognition memory following short-term visual deprivation: Behavioral and neural correlates from ERPs and alpha band oscillations

In the current study, we investigated the effects of short-term visual deprivation (2 h) on a haptic recognition memory task with familiar objects. Behavioral data, as well as event-related potentials (ERPs) and induced event-related oscillations (EROs) were analyzed. At the behavioral level, deprived participants showed speeded reaction times to new stimuli. Analyses of ERPs indicated that starting from 1000 ms the recognition of old objects elicited enhanced positive amplitudes only for the visually deprived group. Visual deprivation also influenced EROs. In this sense, we observed reduced power in the lower-1 alpha band for the processing of new compared to old stimuli between 500 and 750 ms. Overall, our data showed improved haptic recognition memory after a short period of visual deprivation. These effects were thought to reflect a compensatory mechanism that might have developed as an adaptive strategy for dealing with the environment when visual information is not available.

Analyzing text recognition from tactually evoked EEG

Tactual exploration of objects produce specific patterns in the human brain and hence objects can be recognized by analyzing brain signals during tactile exploration. The present work aims at analyzing EEG signals online for recognition of embossed texts by tactual exploration. EEG signals are acquired from the parietal region over the somatosensory cortex of blindfolded healthy subjects while they tactually explored embossed texts, including symbols, numbers, and alphabets. Classifiers based on the principle of supervised learning are trained on the extracted EEG feature space, comprising three features, namely, adaptive autoregressive parameters, Hurst exponents, and power spectral density, to recognize the respective texts. The pre-trained classifiers are used to classify the EEG data to identify the texts online and the recognized text is displayed on the computer screen for communication. Online classifications of two, four, and six classes of embossed texts are achieved with overall average recognition rates of 76.62, 72.31, and 67.62% respectively and the computational time is less than 2 s in each case. The maximum information transfer rate and utility of the system performance over all experiments are 0.7187 and 2.0529 bits/s respectively. This work presents a study that shows the possibility to classify 3D letters using tactually evoked EEG. In future, it will help the BCI community to design stimuli for better tactile augmentation n also opens new directions of research to facilitate 3D letters for visually impaired persons. Further, 3D maps can be generated for aiding tactual BCI in teleoperation.

Effects of Line Separation and Exploration on the Visual and Haptic Detection of Symmetry and Repetition

Detection of regularities (e.g., symmetry, repetition) can be used to investigate object and shape perception. Symmetry and nearby lines may both signal that one object is present, so moving lines apart may disrupt symmetry detection, while repetition may signal that multiple objects are present. Participants discriminated symmetrical/irregular and repeated/irregular pairs of lines. For vision, as predicted, increased line separation disrupted symmetry detection more than repetition detection. For haptics, symmetry and repetition detection were similarly disrupted by increased line separation; also, symmetry was easier to detect than repetition for one-handed exploration and for body midline-aligned stimuli, whereas symmetry was harder to detect than repetition with two-handed exploration of stimuli oriented across the body. These effects of exploration and stimulus orientation show the influence of modality-specific processing rather than properties of the external world on regularity detection. These processes may, in turn, provide insights into the nature of objectness in vision and in touch.

Simultaneous Sketching Aids the Haptic Identification of Raised Line Drawings

Haptically identifying raised line drawings is difficult. We investigated whether a major component of this difficulty lies in acquiring, integrating, and maintaining shape information from touch. Wijntjes, van Lienen, Verstijnen, and Kappers reported that drawings which participants had failed to identify by touch alone could often subsequently be named if they were sketched. Thus, people sometimes needed to externalize haptically acquired information by making a sketch in order to be able to use it. We extended Wijntjes et al.'s task and found that sketching while touching improved drawing identification even more than sketching after touching, but only if people could see their sketches. Our results suggest that the slow, serial nature of information acquisition seriously hampers the haptic identification of raised line drawings relative to visually identifying line drawings. Simultaneous sketching may aid identification by reducing the burden on working memory and by helping to guide haptic exploration. This conclusion is consistent with the finding reported by Lawson and Bracken that 3-D objects are much easier to identify haptically than raised line drawings since, unlike for vision, simultaneously extracting global shape information is much easier haptically for 3-D stimuli than for line drawings.