LIDAR Assist Spatial Sensing for the Visually Impaired and Performance Analysis

Range sensor-based assistive technology solutions for people with visual impairment: a review

PURPOSE

There are about 2.2 billion people with visual impairment all over the world. In order to improve the quality of life for visually impaired people, various devices have been developed that support the users to make them capable of performing everyday work. There have been considerable advancements in the development of assistive devices over the last few decades. This work aims to review the research work of past decade to explore the technologies used in the assistive devices for the mobility of the people with visual impairment. It focuses on range sensors based (RSB) solutions and to present a comprehensive comparison for researchers to improve the quality of the assistive devices.

METHODS

In order to improve life quality of about 2.2 billion people with visual impairment in the world, various assistive devices have been developed. This work aims to review the research work of past decade to explore the technologies used in the assistive devices for the mobility of the people with visual impairment. It focuses on RSB solutions and their comparison.

RESULTS

Various devices developed or the people with visual impairment, in last decade are described along with their workings. Authors also introduced, their own, newly developed assistive device for the people with visual impairment. The feedback from the people with visual impairment, about assistive technology, is also included in the paper.

CONCLUSIONS

This study will benefit researchers developing assistive devices to help for the mobility of the people with visual impairment. Through user feedback and evaluation of the assistive devices, the authors have come to the conclusion that performance, weight and cost of the assistive devices are always important considerations to make the assistive devices more popular among their users.Implication for rehabilitationAlthough assistive device cannot rehabilitate a visually impaired person, range sensor-based assistive device may have following implications:•Use of assistive device is growing, and performance, weight and cost of assistive devices are always important considerations so that proposed technology solution should widely be accepted and adopted by the visually impaired people.•The adaptability and acceptability of an assistive device by the visually impaired people must be considered during design phase.•Proposed assistive technological solutions should meet all the needed functions.•Performance of these devices should be assessed in application context so that these devices may help the visually impaired to perform their tasks independently.•At the end, a new light weight and low-cost device developed by authors, is also given, that could be used to assist the visually impaired people to move independently.

LidSonic for Visually Impaired: Green Machine Learning-Based Assistive Smart Glasses with Smart App and Arduino

Smart wearable technologies such as fitness trackers are creating many new opportunities to improve the quality of life for everyone. It is usually impossible for visually impaired people to orientate themselves in large spaces and navigate an unfamiliar area without external assistance. The design space for assistive technologies for the visually impaired is complex, involving many design parameters including reliability, transparent object detection, handsfree operations, high-speed real-time operations, low battery usage, low computation and memory requirements, ensuring that it is lightweight, and price affordability. State-of-the-art visually impaired devices lack maturity, and they do not fully meet user satisfaction, thus more effort is required to bring innovation to this field. In this work, we develop a pair of smart glasses called LidSonic that uses machine learning, LiDAR, and ultrasonic sensors to identify obstacles. The LidSonic system comprises an Arduino Uno device located in the smart glasses and a smartphone app that communicates data using Bluetooth. Arduino collects data, manages the sensors on smart glasses, detects objects using simple data processing, and provides buzzer warnings to visually impaired users. The smartphone app receives data from Arduino, detects and identifies objects in the spatial environment, and provides verbal feedback about the object to the user. Compared to image processing-based glasses, LidSonic requires much less processing time and energy to classify objects using simple LiDAR data containing 45-integer readings. We provide a detailed description of the system hardware and software design, and its evaluation using nine machine learning algorithms. The data for the training and validation of machine learning models are collected from real spatial environments. We developed the complete LidSonic system using off-the-shelf inexpensive sensors and a microcontroller board costing less than USD 80. The intention is to provide a design of an inexpensive, miniature, green device that can be built into, or mounted on, any pair of glasses or even a wheelchair to help the visually impaired. This work is expected to open new directions for smart glasses design using open software tools and off-the-shelf hardware.

A Smartphone-Based Mobility Assistant Using Depth Imaging for Visually Impaired and Blind

The means of assisting visually impaired and blind (VIB) people when travelling usually relies on other people. Assistive devices have been developed to assist in blind navigation, but many technologies require users to purchase more devices and they lack flexibility, thus making it inconvenient for VIB users. In this research, we made use of a mobile phone with a depth camera function for obstacle avoidance and object recognition. It includes a mobile application that is controlled using simple voice and gesture controls to assist in navigation. The proposed system gathers depth values from 23 coordinate points that are analyzed to determine whether an obstacle is present in the head area, torso area, or ground area, or is a full body obstacle. In order to provide a reliable warning system, the research detects outdoor objects within a distance of 1.6 m. Subsequently, the object detection function includes a unique interactable feature that enables interaction with the user and the device in finding indoor objects by providing an audio and vibration feedback, and users were able to locate their desired objects more than 80% of the time. In conclusion, a flexible and portable system was developed using a depth camera-enabled mobile phone for use in obstacle detection without the need to purchase additional hardware devices.

Dynamic Crosswalk Scene Understanding for the Visually Impaired

Independent mobility poses a great challenge to the visually impaired individuals. This paper proposes a novel system to understand dynamic crosswalk scenes, which detects the key objects, such as crosswalk, vehicle, and pedestrian, and identifies pedestrian traffic light status. The indication of where and when to cross the road is provided to the visually impaired based on the crosswalk scene understanding. Our proposed system is implemented on a head-mounted mobile device (SensingAI G1) equipped with an Intel RealSense camera and a cellphone, and provides surrounding scene information to visually impaired individuals through audio signal. To validate the performance of the proposed system, we propose a crosswalk scene understanding dataset which contains three sub-datasets: a pedestrian traffic light dataset with 7447 images, a dataset of key objects on the crossroad with 1006 images and a crosswalk dataset with 3336 images. Extensive experiments demonstrated that the proposed system was robust and outperformed the state-of-the-art approaches. The experiment conducted with the visually impaired subjects shows that the system is practical useful.

Human Posture Detection Method Based on Wearable Devices

The dynamic detection of human motion is important, which is widely applied in the fields of motion state capture and rehabilitation engineering. In this study, based on multimodal information of surface electromyography (sEMG) signals of upper limb and triaxial acceleration and plantar pressure signals of lower limb, the effective virtual driving control and gait recognition methods were proposed. The effective way of wearable human posture detection was also constructed. Firstly, the moving average window and threshold comparison were used to segment the sEMG signals of the upper limb. The standard deviation and singular values of wavelet coefficients were extracted as the features. After the training and classification by optimized support vector machine (SVM) algorithm, the real-time detection and analysis of three virtual driving actions were performed. The average identification accuracy was 90.90%. Secondly, the mean, standard deviation, variance, and wavelet energy spectrum of triaxial acceleration were extracted, and these parameters were combined with plantar pressure as the gait features. The optimized SVM was selected for the gait identification, and the average accuracy was 90.48%. The experimental results showed that, through different combinations of wearable sensors on the upper and lower limbs, the motion posture information could be dynamically detected, which could be used in the design of virtual rehabilitation system and walking auxiliary system.

Construction and Analysis of a Novel Wearable Assistive Device for a Visually Impaired Person

In this paper, we have given the design and development of a new wearable device that assists visually impaired individuals to travel independently and confidently. The newly proposed device is based on range-based sensors and would work effectively in both indoor and outdoor conditions. It is constructed in the form of two separate modules; one module is designed such that it can be attached to the waist belt of the user, and the other module is designed to wear it on ankle of the user. Both the modules communicate with each other using wireless communication and can cover the full front environment of the user. The information about the front environment is transmitted to the user, via headphone, by sending a set of voice instructions, stored in a memory card added in the belt module. In order to use the device in crowd mode, appropriate networking techniques were also implemented in the prototype such that the interference of two or more devices in the close vicinity can be avoided. In the end, effectiveness of the device is analyzed and proved by conducting experiments and obtaining statistical results.

Multimodal Navigation Systems for Users with Visual Impairments—A Review and Analysis

Multimodal interaction refers to situations where users are provided with multiple modes for interacting with systems. Researchers are working on multimodality solutions in several domains. The focus of this paper is within the domain of navigation systems for supporting users with visual impairments. Although several literature reviews have covered this domain, none have gone through the research synthesis of multimodal navigation systems. This paper provides a review and analysis of multimodal navigation solutions aimed at people with visual impairments. This review also puts forward recommendations for effective multimodal navigation systems. Moreover, this review also presents the challenges faced during the design, implementation and use of multimodal navigation systems. We call for more research to better understand the users’ evolving modality preferences during navigation.

Efficient Multi-Object Detection and Smart Navigation Using Artificial Intelligence for Visually Impaired People

Visually impaired people face numerous difficulties in their daily life, and technological interventions may assist them to meet these challenges. This paper proposes an artificial intelligence-based fully automatic assistive technology to recognize different objects, and auditory inputs are provided to the user in real time, which gives better understanding to the visually impaired person about their surroundings. A deep-learning model is trained with multiple images of objects that are highly relevant to the visually impaired person. Training images are augmented and manually annotated to bring more robustness to the trained model. In addition to computer vision-based techniques for object recognition, a distance-measuring sensor is integrated to make the device more comprehensive by recognizing obstacles while navigating from one place to another. The auditory information that is conveyed to the user after scene segmentation and obstacle identification is optimized to obtain more information in less time for faster processing of video frames. The average accuracy of this proposed method is 95.19% and 99.69% for object detection and recognition, respectively. The time complexity is low, allowing a user to perceive the surrounding scene in real time.