1
|
Memeo M, Sandini G, Cocchi E, Brayda L. Blind people can actively manipulate virtual objects with a novel tactile device. Sci Rep 2023; 13:22845. [PMID: 38129483 PMCID: PMC10739710 DOI: 10.1038/s41598-023-49507-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 12/08/2023] [Indexed: 12/23/2023] Open
Abstract
Frequently in rehabilitation, visually impaired persons are passive agents of exercises with fixed environmental constraints. In fact, a printed tactile map, i.e. a particular picture with a specific spatial arrangement, can usually not be edited. Interaction with map content, instead, facilitates the learning of spatial skills because it exploits mental imagery, manipulation and strategic planning simultaneously. However, it has rarely been applied to maps, mainly because of technological limitations. This study aims to understand if visually impaired people can autonomously build objects that are completely virtual. Specifically, we investigated if a group of twelve blind persons, with a wide age range, could exploit mental imagery to interact with virtual content and actively manipulate it by means of a haptic device. The device is mouse-shaped and designed to jointly perceive, with one finger only, local tactile height and inclination cues of arbitrary scalar fields. Spatial information can be mentally constructed by integrating local tactile cues, given by the device, with global proprioceptive cues, given by hand and arm motion. The experiment consisted of a bi-manual task, in which one hand explored some basic virtual objects and the other hand acted on a keyboard to change the position of one object in real-time. The goal was to merge basic objects into more complex objects, like a puzzle. The experiment spanned different resolutions of the tactile information. We measured task accuracy, efficiency, usability and execution time. The average accuracy in solving the puzzle was 90.5%. Importantly, accuracy was linearly predicted by efficiency, measured as the number of moves needed to solve the task. Subjective parameters linked to usability and spatial resolutions did not predict accuracy; gender modulated the execution time, with men being faster than women. Overall, we show that building purely virtual tactile objects is possible in absence of vision and that the process is measurable and achievable in partial autonomy. Introducing virtual tactile graphics in rehabilitation protocols could facilitate the stimulation of mental imagery, a basic element for the ability to orient in space. The behavioural variable introduced in the current study can be calculated after each trial and therefore could be used to automatically measure and tailor protocols to specific user needs. In perspective, our experimental setup can inspire remote rehabilitation scenarios for visually impaired people.
Collapse
Affiliation(s)
- Mariacarla Memeo
- Robotics, Brain and Cognitive Sciences Department Now With Center for Human Technologies, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy
| | - Giulio Sandini
- Robotics, Brain and Cognitive Sciences Department, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy
| | - Elena Cocchi
- Istituto David Chiossone per Ciechi e Ipovedenti Onlus, Geona, Italy
| | - Luca Brayda
- Robotics, Brain and Cognitive Sciences Department, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy.
- Acoesis srl, Via Enrico Melen 83, Genoa, Italy.
- Nextage srl, Piazza della Vittoria 12, Genova, Italia.
| |
Collapse
|
2
|
Memeo M, Jacono M, Sandini G, Brayda L. Enabling visually impaired people to learn three-dimensional tactile graphics with a 3DOF haptic mouse. J Neuroeng Rehabil 2021; 18:146. [PMID: 34563218 PMCID: PMC8467032 DOI: 10.1186/s12984-021-00935-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Accepted: 09/07/2021] [Indexed: 11/13/2022] Open
Abstract
Background In this work, we present a novel sensory substitution system that enables to learn three dimensional digital information via touch when vision is unavailable. The system is based on a mouse-shaped device, designed to jointly perceive, with one finger only, local tactile height and inclination cues of arbitrary scalar fields. The device hosts a tactile actuator with three degrees of freedom: elevation, roll and pitch. The actuator approximates the tactile interaction with a plane tangential to the contact point between the finger and the field. Spatial information can therefore be mentally constructed by integrating local and global tactile cues: the actuator provides local cues, whereas proprioception associated with the mouse motion provides the global cues. Methods The efficacy of the system is measured by a virtual/real object-matching task. Twenty-four gender and age-matched participants (one blind and one blindfolded sighted group) matched a tactile dictionary of virtual objects with their 3D-printed solid version. The exploration of the virtual objects happened in three conditions, i.e., with isolated or combined height and inclination cues. We investigated the performance and the mental cost of approximating virtual objects in these tactile conditions. Results In both groups, elevation and inclination cues were sufficient to recognize the tactile dictionary, but their combination worked at best. The presence of elevation decreased a subjective estimate of mental effort. Interestingly, only visually impaired participants were aware of their performance and were able to predict it. Conclusions The proposed technology could facilitate the learning of science, engineering and mathematics in absence of vision, being also an industrial low-cost solution to make graphical user interfaces accessible for people with vision loss.
Collapse
Affiliation(s)
- Mariacarla Memeo
- Robotics, Brain and Cognitive Sciences Department Now with Cognition, Motion and Cognitive Science (CMON) Unit, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy.,University of Genoa, Genoa, Italy
| | - Marco Jacono
- Robotics, Brain and Cognitive Sciences Department, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy
| | - Giulio Sandini
- Robotics, Brain and Cognitive Sciences Department, Fondazione Istituto Italiano di Tecnologia, Via Enrico Melen 83, Genoa, Italy
| | - Luca Brayda
- Acoesis srl, Via Enrico Melen 83, Genoa, Italy.
| |
Collapse
|
3
|
Bettelani GC, Averta G, Catalano MG, Leporini B, Bianchi M. Design and Validation of the Readable Device: A Single-Cell Electromagnetic Refreshable Braille Display. IEEE TRANSACTIONS ON HAPTICS 2020; 13:239-245. [PMID: 32012027 DOI: 10.1109/toh.2020.2970929] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Blindness represents one of the major disabling societal causes, impacting the life of visually impaired people and their families. For what concerns the access to written information, one of the main tools used by blind people is the traditional Braille code. This is the reason why in the recent years, there has been a technological effort to develop refreshable Braille devices. These consist of multiple physical dots that dynamically change their configuration to reproduce different sequences of the letters in Braille code. Although promising, these approaches have many drawbacks, which are mainly related to costs, design complexity, portability, and power consumption. Of note, while many solutions have been proposed for multi-cell devices, the investigation of the potentialities of single-cell refreshable systems has received little attention so far. This investigation could offer effective and viable manners to overcome the aforementioned drawbacks, likely fostering a widespread adoption of such assistive technologies with end-users. In this article, we present the design and characterization of a new cost-effective single-cell Electromagnetic Refreshable Braille Display, the Readable system. We also report on tests performed with blindfolded and blind expert Braille code readers. Results demonstrate the effectiveness of our device in correctly reproducing alphanumeric content, opening promising perspectives in every-day life applications.
Collapse
|
4
|
Strese M, Hassen R, Noll A, Steinbach E. A Tactile Computer Mouse for the Display of Surface Material Properties. IEEE TRANSACTIONS ON HAPTICS 2019; 12:18-33. [PMID: 30106740 DOI: 10.1109/toh.2018.2864751] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
We present a novel input/output device to display the tactile properties of surface materials. The proposed Tactile Computer Mouse (TCM) is equipped with a series of actuators that can create perceptually relevant tactile cues to a user. The display capabilities of our TCM match the major tactile dimensions in human surface material perception, namely, hardness, friction, warmth, microscopic roughness, and macroscopic roughness. The TCM also preserves necessary interaction capabilities of a typical computer mouse. In addition to the TCM design, we introduce data acquisition procedures and concepts that are necessary to derive a parametric representation of a surface material and further demonstrate the corresponding rendering approach on the TCM. We conducted subjective experiments to determine tactile property ratings of real materials, perceived property ratings using the TCM, and how precisely subjects match the real materials to corresponding virtual material representations using the TCM in the absence of visual and audible clues. Our experimental results show that our TCM successfully displays the five fundamental tactile dimensions and that the twenty participants were able to perceive the TCM-produced virtual surface material tactile sensations with a recognition rate of 89.6 percent for ten different materials.
Collapse
|
5
|
Leo F, Tinti C, Chiesa S, Cavaglià R, Schmidt S, Cocchi E, Brayda L. Improving spatial working memory in blind and sighted youngsters using programmable tactile displays. SAGE Open Med 2018; 6:2050312118820028. [PMID: 30574309 PMCID: PMC6299321 DOI: 10.1177/2050312118820028] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2018] [Accepted: 11/21/2018] [Indexed: 11/28/2022] Open
Abstract
OBJECTIVE To investigate whether training with tactile matrices displayed with a programmable tactile display improves recalling performance of spatial images in blind, low-vision and sighted youngsters. To code and understand the behavioral underpinnings of learning two-dimensional tactile dispositions, in terms of spontaneous exploration strategies. METHODS Three groups of blind, low-vision and sighted youngsters between 6 and 18 years old performed four training sessions with a weekly schedule in which they were asked to memorize single or double spatial layouts, featured as two-dimensional matrices. RESULTS Results showed that all groups of participants significantly improved their recall performance compared to the first session baseline in the single-matrix task. No statistical difference in performance between groups emerged in this task. Instead, the learning effect in visually impaired participants is reduced in the double-matrix task, whereas it is still robust in blindfolded sighted controls. We also coded tactile exploration strategies in both tasks and their correlation with performance. Sighted youngsters, in particular, favored a proprioceptive exploration strategy. Finally, performance in the double-matrix task negatively correlated with using one hand and positively correlated with a proprioceptive strategy. CONCLUSION The results of our study indicate that blind persons do not easily process two separate spatial layouts. However, rehabilitation programs promoting bi-manual and proprioceptive approaches to tactile exploration might help improve spatial abilities. Finally, programmable tactile displays are an effective way to make spatial and graphical configurations accessible to visually impaired youngsters and they can be profitably exploited in rehabilitation.
Collapse
Affiliation(s)
- Fabrizio Leo
- Robotics, Brain and Cognitive Sciences department, Center for Human Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Carla Tinti
- Dipartimento di Psicologia, Università degli Studi di Torino, Turin, Italy
| | - Silvia Chiesa
- Dipartimento di Psicologia, Università degli Studi di Torino, Turin, Italy
| | - Roberta Cavaglià
- Dipartimento di Psicologia, Università degli Studi di Torino, Turin, Italy
| | - Susanna Schmidt
- Dipartimento di Psicologia, Università degli Studi di Torino, Turin, Italy
| | - Elena Cocchi
- Istituto David Chiossone per Ciechi e Ipovedenti Onlus, Genoa, Italy
| | - Luca Brayda
- Robotics, Brain and Cognitive Sciences department, Center for Human Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| |
Collapse
|
6
|
Brayda L, Leo F, Baccelliere C, Ferrari E, Vigini C. Updated Tactile Feedback with a Pin Array Matrix Helps Blind People to Reduce Self-Location Errors. MICROMACHINES 2018; 9:E351. [PMID: 30424284 PMCID: PMC6082250 DOI: 10.3390/mi9070351] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/04/2018] [Revised: 06/28/2018] [Accepted: 07/09/2018] [Indexed: 11/16/2022]
Abstract
Autonomous navigation in novel environments still represents a challenge for people with visual impairment (VI). Pin array matrices (PAM) are an effective way to display spatial information to VI people in educative/rehabilitative contexts, as they provide high flexibility and versatility. Here, we tested the effectiveness of a PAM in VI participants in an orientation and mobility task. They haptically explored a map showing a scaled representation of a real room on the PAM. The map further included a symbol indicating a virtual target position. Then, participants entered the room and attempted to reach the target three times. While a control group only reviewed the same, unchanged map on the PAM between trials, an experimental group also received an updated map representing, in addition, the position they previously reached in the room. The experimental group significantly improved across trials by having both reduced self-location errors and reduced completion time, unlike the control group. We found that learning spatial layouts through updated tactile feedback on programmable displays outperforms conventional procedures on static tactile maps. This could represent a powerful tool for navigation, both in rehabilitation and everyday life contexts, improving spatial abilities and promoting independent living for VI people.
Collapse
Affiliation(s)
- Luca Brayda
- Research Unit of Robotics, Brain and Cognitive Sciences, Fondazione Istituto Italiano di Tecnologia, Genoa 16153, Italy.
| | - Fabrizio Leo
- Research Unit of Robotics, Brain and Cognitive Sciences, Fondazione Istituto Italiano di Tecnologia, Genoa 16153, Italy.
| | - Caterina Baccelliere
- Research Unit of Robotics, Brain and Cognitive Sciences, Fondazione Istituto Italiano di Tecnologia, Genoa 16153, Italy.
| | - Elisabetta Ferrari
- Research Unit of Robotics, Brain and Cognitive Sciences, Fondazione Istituto Italiano di Tecnologia, Genoa 16153, Italy.
| | | |
Collapse
|
7
|
Tonelli A, Gori M, Brayda L. The Influence of Tactile Cognitive Maps on Auditory Space Perception in Sighted Persons. Front Psychol 2016; 7:1683. [PMID: 27847488 PMCID: PMC5088781 DOI: 10.3389/fpsyg.2016.01683] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2016] [Accepted: 10/13/2016] [Indexed: 11/13/2022] Open
Abstract
We have recently shown that vision is important to improve spatial auditory cognition. In this study, we investigate whether touch is as effective as vision to create a cognitive map of a soundscape. In particular, we tested whether the creation of a mental representation of a room, obtained through tactile exploration of a 3D model, can influence the perception of a complex auditory task in sighted people. We tested two groups of blindfolded sighted people – one experimental and one control group – in an auditory space bisection task. In the first group, the bisection task was performed three times: specifically, the participants explored with their hands the 3D tactile model of the room and were led along the perimeter of the room between the first and the second execution of the space bisection. Then, they were allowed to remove the blindfold for a few minutes and look at the room between the second and third execution of the space bisection. Instead, the control group repeated for two consecutive times the space bisection task without performing any environmental exploration in between. Considering the first execution as a baseline, we found an improvement in the precision after the tactile exploration of the 3D model. Interestingly, no additional gain was obtained when room observation followed the tactile exploration, suggesting that no additional gain was obtained by vision cues after spatial tactile cues were internalized. No improvement was found between the first and the second execution of the space bisection without environmental exploration in the control group, suggesting that the improvement was not due to task learning. Our results show that tactile information modulates the precision of an ongoing space auditory task as well as visual information. This suggests that cognitive maps elicited by touch may participate in cross-modal calibration and supra-modal representations of space that increase implicit knowledge about sound propagation.
Collapse
Affiliation(s)
- Alessia Tonelli
- Unit for Visually Impaired People, Science and Technology for Children and Adults, Istituto Italiano di TecnologiaGenova, Italy; Robotics, Brain and Cognitive Sciences Department, Istituto Italiano di TecnologiaGenova, Italy
| | - Monica Gori
- Unit for Visually Impaired People, Science and Technology for Children and Adults, Istituto Italiano di Tecnologia Genova, Italy
| | - Luca Brayda
- Robotics, Brain and Cognitive Sciences Department, Istituto Italiano di Tecnologia Genova, Italy
| |
Collapse
|
8
|
Leo F, Cocchi E, Brayda L. The Effect of Programmable Tactile Displays on Spatial Learning Skills in Children and Adolescents of Different Visual Disability. IEEE Trans Neural Syst Rehabil Eng 2016; 25:861-872. [PMID: 27775905 DOI: 10.1109/tnsre.2016.2619742] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Vision loss has severe impacts on physical, social and emotional well-being. The education of blind children poses issues as many scholar disciplines (e.g., geometry, mathematics) are normally taught by heavily relying on vision. Touch-based assistive technologies are potential tools to provide graphical contents to blind users, improving learning possibilities and social inclusion. Raised-lines drawings are still the golden standard, but stimuli cannot be reconfigured or adapted and the blind person constantly requires assistance. Although much research concerns technological development, little work concerned the assessment of programmable tactile graphics, in educative and rehabilitative contexts. Here we designed, on programmable tactile displays, tests aimed at assessing spatial memory skills and shapes recognition abilities. Tests involved a group of blind and a group of low vision children and adolescents in a four-week longitudinal schedule. After establishing subject-specific difficulty levels, we observed a significant enhancement of performance across sessions and for both groups. Learning effects were comparable to raised paper control tests: however, our setup required minimal external assistance. Overall, our results demonstrate that programmable maps are an effective way to display graphical contents in educative/rehabilitative contexts. They can be at least as effective as traditional paper tests yet providing superior flexibility and versatility.
Collapse
|