Deep Learning for Fall Detection: Three-Dimensional CNN Combined With LSTM on Video Kinematic Data

Leveraging natural cognitive systems in conjunction with ResNet50-BiGRU model and attention mechanism for enhanced medical image analysis and sports injury prediction

Introduction

In this study, we explore the potential benefits of integrating natural cognitive systems (medical professionals' expertise) and artificial cognitive systems (deep learning models) in the realms of medical image analysis and sports injury prediction. We focus on analyzing medical images of athletes to gain valuable insights into their health status.

Methods

To synergize the strengths of both natural and artificial cognitive systems, we employ the ResNet50-BiGRU model and introduce an attention mechanism. Our goal is to enhance the performance of medical image feature extraction and motion injury prediction. This integrated approach aims to achieve precise identification of anomalies in medical images, particularly related to muscle or bone damage.

Results

We evaluate the effectiveness of our method on four medical image datasets, specifically pertaining to skeletal and muscle injuries. We use performance indicators such as Peak Signal-to-Noise Ratio and Structural Similarity Index, confirming the robustness of our approach in sports injury analysis.

Discussion

Our research contributes significantly by providing an effective deep learning-driven method that harnesses both natural and artificial cognitive systems. By combining human expertise with advanced machine learning techniques, we offer a comprehensive understanding of athletes' health status. This approach holds potential implications for enhancing sports injury prevention, improving diagnostic accuracy, and tailoring personalized treatment plans for athletes, ultimately promoting better overall health and performance outcomes. Despite advancements in medical image analysis and sports injury prediction, existing systems often struggle to identify subtle anomalies and provide precise injury risk assessments, underscoring the necessity of a more integrated and comprehensive approach.

Fall detection based on dynamic key points incorporating preposed attention

Accidental falls pose a significant threat to the elderly population, and accurate fall detection from surveillance videos can significantly reduce the negative impact of falls. Although most fall detection algorithms based on video deep learning focus on training and detecting human posture or key points in pictures or videos, we have found that the human pose-based model and key points-based model can complement each other to improve fall detection accuracy. In this paper, we propose a preposed attention capture mechanism for images that will be fed into the training network, and a fall detection model based on this mechanism. We accomplish this by fusing the human dynamic key point information with the original human posture image. We first propose the concept of dynamic key points to account for incomplete pose key point information in the fall state. We then introduce an attention expectation that predicates the original attention mechanism of the depth model by automatically labeling dynamic key points. Finally, the depth model trained with human dynamic key points is used to correct the detection errors of the depth model with raw human pose images. Our experiments on the Fall Detection Dataset and the UP-Fall Detection Dataset demonstrate that our proposed fall detection algorithm can effectively improve the accuracy of fall detection and provide better support for elderly care.

Experimental Study: Deep Learning-Based Fall Monitoring among Older Adults with Skin-Wearable Electronics

Older adults are more vulnerable to falling due to normal changes due to aging, and their falls are a serious medical risk with high healthcare and societal costs. However, there is a lack of automatic fall detection systems for older adults. This paper reports (1) a wireless, flexible, skin-wearable electronic device for both accurate motion sensing and user comfort, and (2) a deep learning-based classification algorithm for reliable fall detection of older adults. The cost-effective skin-wearable motion monitoring device is designed and fabricated using thin copper films. It includes a six-axis motion sensor and is directly laminated on the skin without adhesives for the collection of accurate motion data. To study accurate fall detection using the proposed device, different deep learning models, body locations for the device placement, and input datasets are investigated using motion data based on various human activities. Our results indicate the optimal location to place the device is the chest, achieving accuracy of more than 98% for falls with motion data from older adults. Moreover, our results suggest a large motion dataset directly collected from older adults is essential to improve the accuracy of fall detection for the older adult population.

Deep Neural Network for the Detections of Fall and Physical Activities Using Foot Pressures and Inertial Sensing

Fall detection and physical activity (PA) classification are important health maintenance issues for the elderly and people with mobility dysfunctions. The literature review showed that most studies concerning fall detection and PA classification addressed these issues individually, and many were based on inertial sensing from the trunk and upper extremities. While shoes are common footwear in daily off-bed activities, most of the aforementioned studies did not focus much on shoe-based measurements. In this paper, we propose a novel footwear approach to detect falls and classify various types of PAs based on a convolutional neural network and recurrent neural network hybrid. The footwear-based detections using deep-learning technology were demonstrated to be efficient based on the data collected from 32 participants, each performing simulated falls and various types of PAs: fall detection with inertial measures had a higher F1-score than detection using foot pressures; the detections of dynamic PAs (jump, jog, walks) had higher F1-scores while using inertial measures, whereas the detections of static PAs (sit, stand) had higher F1-scores while using foot pressures; the combination of foot pressures and inertial measures was most efficient in detecting fall, static, and dynamic PAs.

Falls caused by balance disorders in the elderly with multiple systems involved: Pathogenic mechanisms and treatment strategies

Falls are the main contributor to both fatal and nonfatal injuries in elderly individuals as well as significant sources of morbidity and mortality, which are mostly induced by impaired balance control. The ability to keep balance is a remarkably complex process that allows for rapid and precise changes to prevent falls with multiple systems involved, such as musculoskeletal system, the central nervous system and sensory system. However, the exact pathogenesis of falls caused by balance disorders in the elderly has eluded researchers to date. In consideration of aging phenomenon aggravation and fall risks in the elderly, there is an urgent need to explore the pathogenesis and treatments of falls caused by balance disorders in the elderly. The present review discusses the epidemiology of falls in the elderly, potential pathogenic mechanisms underlying multiple systems involved in falls caused by balance disorders, including musculoskeletal system, the central nervous system and sensory system. Meanwhile, some common treatment strategies, such as physical exercise, new equipment based on artificial intelligence, pharmacologic treatments and fall prevention education are also reviewed. To fully understand the pathogenesis and treatment of falls caused by balance disorders, a need remains for future large-scale multi-center randomized controlled trials and in-depth mechanism studies.

Conceptual Structure and Current Trends in Artificial Intelligence, Machine Learning, and Deep Learning Research in Sports: A Bibliometric Review

Artificial intelligence and its subcategories of machine learning and deep learning are gaining increasing importance and attention in the context of sports research. This has also meant that the number of corresponding publications has become complex and unmanageably large in human terms. In the current state of the research field, there is a lack of bibliometric analysis, which would prove useful for obtaining insights into the large amounts of available literature. Therefore, the present work aims to identify important research issues, elucidate the conceptual structure of the research field, and unpack the evolutionary trends and the direction of hot topics regarding key themes in the research field of artificial intelligence in sports. Using the Scopus database, 1215 documents (reviews and articles) were selected. Bibliometric analysis was performed using VOSviewer and bibliometrix R package. The main findings are as follows: (a) the literature and research interest concerning AI and its subcategories is growing exponentially; (b) the top 20 most cited works comprise 32.52% of the total citations; (c) the top 10 journals are responsible for 28.64% of all published documents; (d) strong collaborative relationships are present, along with small, isolated collaboration networks of individual institutions; (e) the three most productive countries are China, the USA, and Germany; (f) different research themes can be characterized using author keywords with current trend topics, e.g., in the fields of biomechanics, injury prevention or prediction, new algorithms, and learning approaches. AI research activities in the fields of sports pedagogy, sports sociology, and sports economics seem to have played a subordinate role thus far. Overall, the findings of this study expand knowledge on the research situation as well as the development of research topics regarding the use of artificial intelligence in sports, and may guide researchers to identify currently relevant topics and gaps in the research.

Human Fall Detection Using 3D Multi-Stream Convolutional Neural Networks with Fusion

Human falls, especially for elderly people, can cause serious injuries that might lead to permanent disability. Approximately 20-30% of the aged people in the United States who experienced fall accidents suffer from head trauma, injuries, or bruises. Fall detection is becoming an important public healthcare problem. Timely and accurate fall incident detection could enable the instant delivery of medical services to the injured. New advances in vision-based technologies, including deep learning, have shown significant results in action recognition, where some focus on the detection of fall actions. In this paper, we propose an automatic human fall detection system using multi-stream convolutional neural networks with fusion. The system is based on a multi-level image-fusion approach of every 16 frames of an input video to highlight movement differences within this range. This results of four consecutive preprocessed images are fed to a new proposed and efficient lightweight multi-stream CNN model that is based on a four-branch architecture (4S-3DCNN) that classifies whether there is an incident of a human fall. The evaluation included the use of more than 6392 generated sequences from the Le2i fall detection dataset, which is a publicly available fall video dataset. The proposed method, using three-fold cross-validation to validate generalization and susceptibility to overfitting, achieved a 99.03%, 99.00%, 99.68%, and 99.00% accuracy, sensitivity, specificity, and precision, respectively. The experimental results prove that the proposed model outperforms state-of-the-art models, including GoogleNet, SqueezeNet, ResNet18, and DarkNet19, for fall incident detection.

In-Home Older Adults' Activity Pattern Monitoring Using Depth Sensors: A Review

The global population is aging due to many factors, including longer life expectancy through better healthcare, changing diet, physical activity, etc. We are also witnessing various frequent epidemics as well as pandemics. The existing healthcare system has failed to deliver the care and support needed to our older adults (seniors) during these frequent outbreaks. Sophisticated sensor-based in-home care systems may offer an effective solution to this global crisis. The monitoring system is the key component of any in-home care system. The evidence indicates that they are more useful when implemented in a non-intrusive manner through different visual and audio sensors. Artificial Intelligence (AI) and Computer Vision (CV) techniques may be ideal for this purpose. Since the RGB imagery-based CV technique may compromise privacy, people often hesitate to utilize in-home care systems which use this technology. Depth, thermal, and audio-based CV techniques could be meaningful substitutes here. Due to the need to monitor larger areas, this review article presents a systematic discussion on the state-of-the-art using depth sensors as primary data-capturing techniques. We mainly focused on fall detection and other health-related physical patterns. As gait parameters may help to detect these activities, we also considered depth sensor-based gait parameters separately. The article provides discussions on the topic in relation to the terminology, reviews, a survey of popular datasets, and future scopes.

An Ultrasonic-Based Sensor System for Elderly Fall Monitoring in a Smart Room

To reduce the risk of elderly people falling in a private room without relying on a closed-circuit television system that results in serious privacy and trust concerns, a fall monitoring system that protects the privacy and does not monitor a person’s activities is needed. An ultrasonic-based sensor system for elderly fall monitoring in a smart room is proposed in this study. An array of ultrasonic sensors, whose ranges are designed to cover the room space, are initially installed on a wall of the room, and the sensors are rotated to transmit and receive ultrasonic signals to measure the distances to a moving object while preventing ultrasonic signal interference. Distance changes measured by ultrasonic sensors are used as time-independent patterns to recognize when an elderly person falls. To evaluate the performance of the proposed system, a sensor system prototype using long short-term memory was constructed, and experiments with 25 participants were performed. An accuracy of approximately 98% was achieved in this experiment using the proposed method, which was a slight improvement over that of the conventional method.

Multimodal deep learning model on interim [18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma

OBJECTIVES

The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim ¹⁸F-fluoro-2-deoxyglucose ([¹⁸F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL.

METHODS

Initially, 205 DLBCL patients undergoing interim [¹⁸F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance.

RESULTS

The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability.

CONCLUSIONS

The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment.

KEY POINTS

• The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients. • Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance. • Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.

Fall Detection for Shipboard Seafarers Based on Optimized BlazePose and LSTM

Aiming to avoid personal injury caused by the failure of timely medical assistance following a fall by seafarer members working on ships, research on the detection of seafarer's falls and timely warnings to safety officers can reduce the loss and severe consequences of falls to seafarers. To improve the detection accuracy and real-time performance of the seafarer fall detection algorithm, a seafarer fall detection algorithm based on BlazePose-LSTM is proposed. This algorithm can automatically extract the human body key point information from the video image obtained by the vision sensor, analyze its internal data correlation characteristics, and realize the process from RGB camera image processing to seafarer fall detection. This fall detection algorithm extracts the human body key point information through the optimized BlazePose human body key point information extraction network. In this section, a new method for human bounding-box acquisition is proposed. In this study, a head detector based on the Vitruvian theory was used to replace the pre-trained SSD body detector in the BlazePose preheating module. Simultaneously, an offset vector is proposed to update the bounding box obtained. This method can reduce the frequency of repeated use of the head detection module. The algorithm then uses the long short-term memory neural network to detect seafarer falls. After extracting fall and related behavior data from the URFall public data set and FDD public data set to enrich the self-made data set, the experimental results show that the algorithm can achieve 100% accuracy and 98.5% specificity for the seafarer's falling behavior, indicating that the algorithm has reasonable practicability and strong generalization ability. The detection frame rate can reach 29 fps on a CPU, which can meet the effect of real-time detection. The proposed method can be deployed on common vision sensors.

Intelligent cataract surgery supervision and evaluation via deep learning

PURPOSE

To assess the performance of a deep learning (DL) algorithm for evaluating and supervising cataract extraction using phacoemulsification with intraocular lens (IOL) implantation based on cataract surgery (CS) videos.

MATERIALS AND METHODS

DeepSurgery was trained using 186 standard CS videos to recognize 12 CS steps and was validated in two datasets that contained 50 and 21 CS videos, respectively. A supervision test including 50 CS videos was used to assess the DeepSurgery guidance and alert function. In addition, a real-time test containing 54 CSs was used to compare the DeepSurgery grading performance to an expert panel and residents.

RESULTS

DeepSurgery achieved stable performance for all 12 recognition steps, including the duration between two pairs of adjacent steps in internal validation with an ACC of 95.06% and external validations with ACCs of 88.77% and 88.34%. DeepSurgery also recognized the chronology of surgical steps and alerted surgeons to order of incorrect steps. Six main steps are automatically and simultaneously quantified during the evaluation process (centesimal system). In a real-time comparative test, the DeepSurgery step recognition performance was robust (ACC of 90.30%). In addition, DeepSurgery and an expert panel achieved comparable performance when assessing the surgical steps (kappa ranged from 0.58 to 0.77).

CONCLUSIONS

DeepSurgery represents a potential approach to provide a real-time supervision and an objective surgical evaluation system for routine CS and to improve surgical outcomes.

Towards Secure and Intelligent Internet of Health Things: A Survey of Enabling Technologies and Applications

With the growth of computing and communication technologies, the information processing paradigm of the healthcare environment is evolving. The patient information is stored electronically, making it convenient to store and retrieve patient information remotely when needed. However, evolving the healthcare systems into smart healthcare environments comes with challenges and additional pressures. Internet of Things (IoT) connects things, such as computing devices, through wired or wireless mediums to form a network. There are numerous security vulnerabilities and risks in the existing IoT-based systems due to the lack of intrinsic security technologies. For example, patient medical data, data privacy, data sharing, and convenience are considered imperative for collecting and storing electronic health records (EHR). However, the traditional IoT-based EHR systems cannot deal with these paradigms because of inconsistent security policies and data access structures. Blockchain (BC) technology is a decentralized and distributed ledger that comes in handy in storing patient data and encountering data integrity and confidentiality challenges. Therefore, it is a viable solution for addressing existing IoT data security and privacy challenges. BC paves a tremendous path to revolutionize traditional IoT systems by enhancing data security, privacy, and transparency. The scientific community has shown a variety of healthcare applications based on artificial intelligence (AI) that improve health diagnosis and monitoring practices. Moreover, technology companies and startups are revolutionizing healthcare with AI and related technologies. This study illustrates the implication of integrated technologies based on BC, IoT, and AI to meet growing healthcare challenges. This research study examines the integration of BC technology with IoT and analyzes the advancements of these innovative paradigms in the healthcare sector. In addition, our research study presents a detailed survey on enabling technologies for the futuristic, intelligent, and secure internet of health things (IoHT). Furthermore, this study comprehensively studies the peculiarities of the IoHT environment and the security, performance, and progression of the enabling technologies. First, the research gaps are identified by mapping security and performance benefits inferred by the BC technologies. Secondly, practical issues related to the integration process of BC and IoT devices are discussed. Third, the healthcare applications integrating IoT, BC, and ML in healthcare environments are discussed. Finally, the research gaps, future directions, and limitations of the enabling technologies are discussed.

Construction of Correlation Analysis Model of College Students’ Sports Performance Based on Convolutional Neural Network

This paper proposes a network model recurrent fully connected network (RFC-Net) based on recurrent full convolution and polarization change. RFC-Net enriches the network by reconstructing and fine-tuning the fully convolutional network and adding recurrent convolutions to it. By studying the data mining technology of multidimensional association rules, based on the existing algorithms, this paper improves the shortcomings of the algorithms and realizes an efficient and practical method for data mining based on interdimensional multidimensional association rules. On the basis of mastering the actual student information, the effectiveness of the method is tested, and an employment analysis system based on association rules is established. Aiming at the fact that traditional grade prediction methods ignore the different influences of different behavioral characteristics on grades, and considering that behavioral data in different periods have different influences on student grades, the grade prediction problem is abstracted into a time series classification problem. The mechanism is combined with long short-term memory neural network to construct a performance prediction model based on Attention-BiLSTM. Experiments show that the prediction model proposed in this paper improves the accuracy and effectively improves the prediction quality compared with the logistic regression model with a better prediction effect in the traditional benchmark model and the long short-term memory neural network model without the introduction of the attention mechanism. Research shows that physical performance and academic performance are not contradictory. We must face up to the status of physical exercise in schools; as long as physical exercise is properly arranged, it can inspire students to form a spirit of unity, interaction, positivity, and perseverance in cultural studies.

Fall Detection Using Multi-Property Spatiotemporal Autoencoders in Maritime Environments

Man overboard is an emergency in which fast and efficient detection of the critical event is the key factor for the recovery of the victim. Its severity urges the utilization of intelligent video surveillance systems that monitor the ship’s perimeter in real time and trigger the relative alarms that initiate the rescue mission. In terms of deep learning analysis, since man overboard incidents occur rarely, they present a severe class imbalance problem, and thus, supervised classification methods are not suitable. To tackle this obstacle, we follow an alternative philosophy and present a novel deep learning framework that formulates man overboard identification as an anomaly detection task. The proposed system, in the absence of training data, utilizes a multi-property spatiotemporal convolutional autoencoder that is trained only on the normal situation. We explore the use of RGB video sequences to extract specific properties of the scene, such as gradient and saliency, and utilize the autoencoders to detect anomalies. To the best of our knowledge, this is the first time that man overboard detection is made in a fully unsupervised manner while jointly learning the spatiotemporal features from RGB video streams. The algorithm achieved 97.30% accuracy and a 96.01% F1-score, surpassing the other state-of-the-art approaches significantly.

IoT-Based Human Fall Detection System

Human falls are an issue that especially affects elderly people, resulting in permanent disabilities or even in the person’s death. Preventing human falls is a social desire, but it is almost impossible to achieve because it is not possible to ensure full prevention. A possible solution is the detection of human falls in near real-time so that help can quickly be provided. This has the potential to greatly reduce the severity of the fall in long-term health consequences. This work proposes a solution based on the internet of things devices installed in people’s homes. The proposed non-wearable solution is non-intrusive and can be deployed not only in homes but also in hospitals, rehabilitation facilities, and elderly homes. The solution uses a three-layered computation architecture composed of edge, fog, and cloud. A mathematical model using the Morlet wavelet and an artificial intelligence model using artificial neural networks are used for human fall classification; both approaches are compared. The results showed that the combination of both models is possible and brings benefits to the system, achieving an accuracy of 92.5% without false negatives.

Deep Vigilante: A deep learning network for real-world crime detection

 Identification/recognition of assault, fighting, shooting, and vandalism from video sequence using deep 2D and 3D convolutional neural networks (CNNs) is explored in this paper. Recent wave of extensive unrestricted urbanization has not only uplifted the standard of living, but has also threatened the safety of a common man leading to an extraordinary rise in crime rate. Although Closed-circuit television (CCTV) footage provides a monitoring framework, yet, it’s useless without an auto volume crime detection system. The system proposed in this work is an effort to eradicate volume crimes through accurate detection in real-time. Firstly, a fine-grained annotated dataset including instance and activity information has been developed for real-world volume crimes. Secondly, a comparison between 3D CNN and 2D CNN network has been presented to identify the malicious event from the video sequence. This is carried out to explore the significance of spatial and temporal information present in the video for event recognition. It has been observed that 2D CNN even with lesser parameters achieved a promising classification accuracy of 91.2%and Area under the curve (AUC) of 95.2%on four classes. The system also reduces false alarm rate in comparison to state-of-the-art approaches.

Imitating Emergencies: Generating Thermal Surveillance Fall Data Using Low-Cost Human-like Dolls

Outdoor fall detection, in the context of accidents, such as falling from heights or in water, is a research area that has not received as much attention as other automated surveillance areas. Gathering sufficient data for developing deep-learning models for such applications has also proven to be not a straight-forward task. Normally, footage of volunteer people falling is used for providing data, but that can be a complicated and dangerous process. In this paper, we propose an application for thermal images of a low-cost rubber doll falling in a harbor, for simulating real emergencies. We achieve thermal signatures similar to a human on different parts of the doll’s body. The change of these thermal signatures over time is measured, and its stability is verified. We demonstrate that, even with the size and weight differences of the doll, the produced videos of falls have a similar motion and appearance to what is expected from real people. We show that the captured thermal doll data can be used for the real-world application of pedestrian detection by running the captured data through a state-of-the-art object detector trained on real people. An average confidence score of 0.730 is achieved, compared to a confidence score of 0.761 when using footage of real people falling. The captured fall sequences using the doll can be used as a substitute to sequences of people.

Data portability for activities of daily living and fall detection in different environments using radar micro-doppler

The health status of an older or vulnerable person can be determined by looking into the additive effects of aging as well as any associated diseases. This status can lead the person to a situation of ‘unstable incapacity’ for normal aging and is determined by the decrease in response to the environment and to specific pathologies with apparent decrease of independence in activities of daily living (ADL). In this paper, we use micro-Doppler images obtained using a frequency-modulated continuous wave radar (FMCW) operating at 5.8 GHz with 400 MHz bandwidth as the sensor to perform assessment of this health status. The core idea is to develop a generalized system where the data obtained for ADL can be portable across different environments and groups of subjects, and critical events such as falls in mature individuals can be detected. In this context, we have conducted comprehensive experimental campaigns at nine different locations including four laboratory environments and five elderly care homes. A total of 99 subjects participated in the experiments where 1453 micro-Doppler signatures were recorded for six activities. Different machine learning, deep learning algorithms and transfer learning technique were used to classify the ADL. The support vector machine (SVM), K-nearest neighbor (KNN) and convolutional neural network (CNN) provided adequate classification accuracies for particular scenarios; however, the autoencoder neural network outperformed the mentioned classifiers by providing classification accuracy of ~ 88%. The proposed system for fall detection in elderly people can be deployed in care centers and is application for any indoor settings with various age group of people. For future work, we would focus on monitoring multiple older adults, concurrently in indoor settings using continuous radar sensor data stream which is limitation of the present system.

A convolution neural network approach for fall detection based on adaptive channel selection of UWB radar signals

According to the World Health Organization and other authorities, falls are one of the main causes of accidental injuries among the elderly population. Therefore, it is essential to detect and predict the fall activities of older persons in indoor environments such as homes, nursing, senior residential centers, and care facilities. Due to non-contact and signal confidentiality characteristics, radar equipment is widely used in indoor care, detection, and rescue. This paper proposes an adaptive channel selection algorithm to separate the activity signals from the background using an ultra-wideband radar and to generalize fused features of frequency- and time-domain images which will be sent to a lightweight convolutional neural network to detect and recognize fall activities. The experimental results show that the method is able to distinguish three types of fall activities (i.e., stand to fall, bow to fall, and squat to fall) and obtain a high recognition accuracy up to 95.7%.

Video-Based Inpatient Fall Risk Assessment: A Case Study

Inpatient falls are a serious safety issue in hospitals and healthcare facilities. Recent advances in video analytics for patient monitoring provide a non-intrusive avenue to reduce this risk through continuous activity monitoring. However, in- bed fall risk assessment systems have received less attention in the literature. The majority of prior studies have focused on fall event detection, and do not consider the circumstances that may indicate an imminent inpatient fall. Here, we propose a video-based system that can monitor the risk of a patient falling, and alert staff of unsafe behaviour to help prevent falls before they occur. We propose an approach that leverages recent advances in human localisation and skeleton pose estimation to extract spatial features from video frames recorded in a simulated environment. We demonstrate that body positions can be effectively recognised and provide useful evidence for fall risk assessment. This work highlights the benefits of video-based models for analysing behaviours of interest, and demonstrates how such a system could enable sufficient lead time for healthcare professionals to respond and address patient needs, which is necessary for the development of fall intervention programs.

A Class-Imbalanced Deep Learning Fall Detection Algorithm Using Wearable Sensors

Falling represents one of the most serious health risks for elderly people; it may cause irreversible injuries if the individual cannot obtain timely treatment after the fall happens. Therefore, timely and accurate fall detection algorithm research is extremely important. Recently, a number of researchers have focused on fall detection and made many achievements, and most of the relevant algorithm studies are based on ideal class-balanced datasets. However, in real-life applications, the possibilities of Activities of Daily Life (ADL) and fall events are different, so the data collected by wearable sensors suffers from class imbalance. The previously developed algorithms perform poorly on class-imbalanced data. In order to solve this problem, this paper proposes an algorithm that can effectively distinguish falls from a large amount of ADL signals. Compared with the state-of-the-art fall detection algorithms, the proposed method can achieve the highest score in multiple evaluation methods, with a sensitivity of 99.33%, a specificity of 91.86%, an F-Score of 98.44% and an AUC of 98.35%. The results prove that the proposed algorithm is effective on class-imbalanced data and more suitable for real-life application compared to previous works.

Artificial intelligence-based hybrid deep learning models for image classification: The first narrative review

BACKGROUND

Artificial intelligence (AI) has served humanity in many applications since its inception. Currently, it dominates the imaging field-in particular, image classification. The task of image classification became much easier with machine learning (ML) and subsequently got automated and more accurate by using deep learning (DL). By default, DL consists of a single architecture and is termed solo deep learning (SDL). When two or more DL architectures are fused, the result is termed a hybrid deep learning (HDL) model. The use of HDL models is becoming popular in several applications, but no review of these uses has been designed thus far. Therefore, this study provides the first narrative HDL review by considering all facets of image classification using AI.

APPROACH

Our review employs a PRISMA search strategy using Google Scholar, PubMed, IEEE, and Elsevier Science Direct, through which 127 relevant HDL studies were considered. Based on the computer vision evolution, HDLs were subsequently classified into three categories (spatial, temporal, and spatial-temporal). Each study was then analyzed based on several attributes, including continent, publisher, hybridization of two DL or ML, architecture layout, application type, data set type, dataset size, feature extraction methodology, connecting classifier, performance evaluation metrics, and risk-of-bias.

CONCLUSION

The HDL models have shown stable and superior performance by taking the best aspects of two or more solo DL or fusion of DL with ML models. Our findings indicate that HDL is being applied aggressively to several medical and non-medical applications. Furthermore, risk-of-bias is highly debatable for DL and HDL models.

A smartphone-based online system for fall detection with alert notifications and contextual information of real-life falls

BACKGROUND

Falls are a leading cause of accidental deaths and injuries worldwide. The risk of falling is especially high for individuals suffering from balance impairments. Retrospective surveys and studies of simulated falling in lab conditions are frequently used and are informative, but prospective information about real-life falls remains sparse. Such data are essential to address fall risks and develop fall detection and alert systems. Here we present the results of a prospective study investigating a proof-of-concept, smartphone-based, online system for fall detection and notification.

METHODS

The system uses the smartphone's accelerometer and gyroscope to monitor the participants' motion, and falls are detected using a regularized logistic regression. Data on falls and near-fall events (i.e., stumbles) is stored in a cloud server and fall-related variables are logged onto a web portal developed for data exploration, including the event time and weather, fall probability, and the faller's location and activity before the fall.

RESULTS

In total, 23 individuals with an elevated risk of falling carried the phones for 2070 days in which the model classified 14,904,000 events. The system detected 27 of the 37 falls that occurred (sensitivity = 73.0 %) and resulted in one false alarm every 46 days (specificity > 99.9 %, precision = 37.5 %). 42.2 % of the events falsely classified as falls were validated as stumbles.

CONCLUSIONS

The system's performance shows the potential of using smartphones for fall detection and notification in real-life. Apart from functioning as a practical fall monitoring instrument, this system may serve as a valuable research tool, enable future studies to scale their ability to capture fall-related data, and help researchers and clinicians to investigate real-falls.

Binary Sensors-Based Privacy-Preserved Activity Recognition of Elderly Living Alone Using an RNN

The recent growth of the elderly population has led to the requirement for constant home monitoring as solitary living becomes popular. This protects older people who live alone from unwanted instances such as falling or deterioration caused by some diseases. However, although wearable devices and camera-based systems can provide relatively precise information about human motion, they invade the privacy of the elderly. One way to detect the abnormal behavior of elderly residents under the condition of maintaining privacy is to equip the resident's house with an Internet of Things system based on a non-invasive binary motion sensor array. We propose to concatenate external features (previous activity and begin time-stamp) along with extracted features with a bi-directional long short-term memory (Bi-LSTM) neural network to recognize the activities of daily living with a higher accuracy. The concatenated features are classified by a fully connected neural network (FCNN). The proposed model was evaluated on open dataset from the Center for Advanced Studies in Adaptive Systems (CASAS) at Washington State University. The experimental results show that the proposed method outperformed state-of-the-art models with a margin of more than 6.25% of the F₁ score on the same dataset.

Real-time computer-aided diagnosis of focal pancreatic masses from endoscopic ultrasound imaging based on a hybrid convolutional and long short-term memory neural network model

Differential diagnosis of focal pancreatic masses is based on endoscopic ultrasound (EUS) guided fine needle aspiration biopsy (EUS-FNA/FNB). Several imaging techniques (i.e. gray-scale, color Doppler, contrast-enhancement and elastography) are used for differential diagnosis. However, diagnosis remains highly operator dependent. To address this problem, machine learning algorithms (MLA) can generate an automatic computer-aided diagnosis (CAD) by analyzing a large number of clinical images in real-time. We aimed to develop a MLA to characterize focal pancreatic masses during the EUS procedure. The study included 65 patients with focal pancreatic masses, with 20 EUS images selected from each patient (grayscale, color Doppler, arterial and venous phase contrast-enhancement and elastography). Images were classified based on cytopathology exam as: chronic pseudotumoral pancreatitis (CPP), neuroendocrine tumor (PNET) and ductal adenocarcinoma (PDAC). The MLA is based on a deep learning method which combines convolutional (CNN) and long short-term memory (LSTM) neural networks. 2688 images were used for training and 672 images for testing the deep learning models. The CNN was developed to identify the discriminative features of images, while a LSTM neural network was used to extract the dependencies between images. The model predicted the clinical diagnosis with an area under curve index of 0.98 and an overall accuracy of 98.26%. The negative (NPV) and positive (PPV) predictive values and the corresponding 95% confidential intervals (CI) are 96.7%, [94.5, 98.9] and 98.1%, [96.81, 99.4] for PDAC, 96.5%, [94.1, 98.8], and 99.7%, [99.3, 100] for CPP, and 98.9%, [97.5, 100] and 98.3%, [97.1, 99.4] for PNET. Following further validation on a independent test cohort, this method could become an efficient CAD tool to differentiate focal pancreatic masses in real-time.

The effects of mechanical noise bandwidth on balance across flat and compliant surfaces

Although the application of sub-sensory mechanical noise to the soles of the feet has been shown to enhance balance, there has been no study on how the bandwidth of the noise affects balance. Here, we report a single-blind randomized controlled study on the effects of a narrow and wide bandwidth mechanical noise on healthy young subjects’ sway during quiet standing on firm and compliant surfaces. For the firm surface, there was no improvement in balance for both bandwidths—this may be because the young subjects could already balance near-optimally or optimally on the surface by themselves. For the compliant surface, balance improved with the introduction of wide but not narrow bandwidth noise, and balance is improved for wide compared to narrow bandwidth noise. This could be explained using a simple model, which suggests that adding noise to a sub-threshold pressure stimulus results in markedly different frequency of nerve impulse transmitted to the brain for the narrow and wide bandwidth noise—the frequency is negligible for the former but significantly higher for the latter. Our results suggest that if a person’s standing balance is not optimal (for example, due to aging), it could be improved by applying a wide bandwidth noise to the feet.

High Accuracy WiFi-Based Human Activity Classification System with Time-Frequency Diagram CNN Method for Different Places

Older people are very likely to fall, which is a significant threat to the health. However, falls are preventable and are not necessarily an inevitable part of aging. Many different fall detection systems have been developed to help people avoid falling. However, traditional systems based on wearable devices or image recognition-based have many disadvantages, such as user-unfriendly, privacy issues. Recently, WiFi-based fall detection systems try to solve the above problems. However, there is a common problem of reduced accuracy. Since the system is trained at the original signal collecting/training place, however, the application is at a different place. The proposed solution only extracts the features of the changed signal, which is caused by a specific human action. To implement this, we used Channel State Information (CSI) to train Convolutional Neural Networks (CNNs) and further classify the action. We have designed a prototype to test the performance of our proposed method. Our simulation results show an average accuracy of same place and different place is 93.2% and 90.3%, respectively.

Movement Tube Detection Network Integrating 3D CNN and Object Detection Framework to Detect Fall

Unlike most of the existing neural network-based fall detection methods, which only detect fall at the time range, the algorithm proposed in this paper detect fall in both spatial and temporal dimension. A movement tube detection network integrating 3D CNN and object detection framework such as SSD is proposed to detect human fall with constrained movement tubes. The constrained movement tube, which encapsulates the person with a sequence of bounding boxes, has the merits of encapsulating the person closely and avoiding peripheral interference. A 3D convolutional neural network is used to encode the motion and appearance features of a video clip, which are fed into the tube anchors generation layer, softmax classification, and movement tube regression layer. The movement tube regression layer fine tunes the tube anchors to the constrained movement tubes. A large-scale spatio-temporal (LSST) fall dataset is constructed using self-collected data to evaluate the fall detection in both spatial and temporal dimensions. LSST has three characteristics of large scale, annotation, and posture and viewpoint diversities. Furthermore, the comparative experiments on a public dataset demonstrate that the proposed algorithm achieved sensitivity, specificity an accuracy of 100%, 97.04%, and 97.23%, respectively, outperforms the existing methods.

Fall Detection With UWB Radars and CNN-LSTM Architecture

Fall detection is a major challenge for researchers. Indeed, a fall can cause injuries such as femoral neck fracture, brain hemorrhage, or skin burns, leading to significant pain. However, in some cases, trauma caused by an undetected fall can get worse with the time and conducts to painful end of life or even death. One solution is to detect falls efficiently to alert somebody (e.g., nurses) as quickly as possible. To respond to this need, we propose to detect falls in a real apartment of 40 square meters by exploiting three ultra-wideband radars and a deep neural network model. The deep neural network is composed of a convolutional neural network stacked with a long-short term memory network and a fully connected neural network to identify falls. In other words, the problem addressed in this paper is a binary classification attempting to differentiate fall and non-fall events. As it can be noticed in real cases, the falls can have different forms. Hence, the data to train and test the classification model have been generated with falls (four types) simulated by 10 participants in three locations in the apartment. Finally, the train and test stages have been achieved according to three strategies, including the leave-one-subject-out method. This latter method allows for obtaining the performances of the proposed system in a generalization context. The results are very promising since we reach almost 90% of accuracy.

MC-LSTM: Real-Time 3D Human Action Detection System for Intelligent Healthcare Applications

Due to the movement expressiveness and privacy assurance of human skeleton data, 3D skeleton-based action inference is becoming popular in healthcare applications. These scenarios call for more advanced performance in application-specific algorithms and efficient hardware support. Warnings on health emergencies sensitive to response speed require low latency output and action early detection capabilities. Medical monitoring that works in an always-on edge platform needs the system processor to have extreme energy efficiency. Therefore, in this paper, we propose the MC-LSTM, a functional and versatile 3D skeleton-based action detection system, for the above demands. Our system achieves state-of-the-art accuracy on trimmed and untrimmed cases of general-purpose and medical-specific datasets with early-detection features. Further, the MC-LSTM accelerator supports parallel inference on up to 64 input channels. The implementation on Xilinx ZCU104 reaches a throughput of 18 658 Frames-Per-Second (FPS) and an inference latency of 3.5 ms with the batch size of 64. Accordingly, the power consumption is 3.6 W for the whole FPGA+ARM system, which is 37.8x and 10.4x more energy-efficient than the high-end Titan X GPU and i7-9700 CPU, respectively. Meanwhile, our accelerator also keeps a 4  ∼ 5x energy efficiency advantage against the low-power high-performance Firefly-RK3399 board carrying an ARM Cortex-A72+A53 CPU. We further synthesize an 8-bit quantized version on the same hardware, providing a 48.8% increase in energy efficiency under the same throughput.

Classification of Indoor Human Fall Events Using Deep Learning

Human fall identification can play a significant role in generating sensor based alarm systems, assisting physical therapists not only to reduce after fall effects but also to save human lives. Usually, elderly people suffer from various kinds of diseases and fall action is a very frequently occurring circumstance at this time for them. In this regard, this paper represents an architecture to classify fall events from others indoor natural activities of human beings. Video frame generator is applied to extract frame from video clips. Initially, a two dimensional convolutional neural network (2DCNN) model is proposed to extract features from video frames. Afterward, gated recurrent unit (GRU) network finds the temporal dependency of human movement. Binary cross-entropy loss function is calculated to update the attributes of the network like weights, learning rate to minimize the losses. Finally, sigmoid classifier is used for binary classification to detect human fall events. Experimental result shows that the proposed model obtains an accuracy of 99%, which outperforms other state-of-the-art models.

Fall Detection from Electrocardiogram (ECG) Signals and Classification by Deep Transfer Learning

Fall is a prominent issue due to its severe consequences both physically and mentally. Fall detection and prevention is a critical area of research because it can help elderly people to depend less on caregivers and allow them to live and move more independently. Using electrocardiograms (ECG) signals independently for fall detection and activity classification is a novel approach used in this paper. An algorithm has been proposed which uses pre-trained convolutional neural networks AlexNet and GoogLeNet as a classifier between the fall and no fall scenarios using electrocardiogram signals. The ECGs for both falling and no falling cases were obtained as part of the study using eight volunteers. The signals are pre-processed using an elliptical filter for signal noises such as baseline wander and power-line interface. As feature extractors, frequency-time representations (scalograms) were obtained by applying a continuous wavelet transform on the filtered ECG signals. These scalograms were used as inputs to the neural network and a significant validation accuracy of 98.08% was achieved in the first model. The trained model is able to distinguish ECGs with a fall activity from an ECG with a no fall activity with an accuracy of 98.02%. For the verification of the robustness of the proposed algorithm, our experimental dataset was augmented by adding two different publicly available datasets to it. The second model can classify fall, daily activities and no activities with an accuracy of 98.44%. These models were developed by transfer learning from the domain of real images to the medical images. In comparison to traditional deep learning approaches, the transfer learning not only avoids “reinventing the wheel,” but also presents a lightweight solution to otherwise computationally heavy problems.

A Two-Stage Fall Recognition Algorithm Based on Human Posture Features

Falls are seriously threatening the health of elderly. In order to reduce the potential danger caused by falls, this paper proposes a two-stage fall recognition algorithm based on human posture features. For preprocessing, we construct the new key features: deflection angles and spine ratio to describe the changes of human posture based on the human skeleton extracted by OpenPose. In the first stage, based on the variables: tendency symbol and steady symbol integrated by the scattered key features, we divide the human body state into three states: stable state, fluctuating state, and disordered state. By analyzing whether the body is in a stable state, the ADL (activities of daily living) actions with high stability can be preliminarily excluded. In the second stage: to further identify the confusing ADL actions and the fall actions, we innovatively design a time-continuous recognition algorithm. When human body is constantly in an unstable state, the human posture features: compare value γ, energy value ε, state score τ are proposed to form a feature vector, and support vector machine (SVM), K nearest neighbors (KNN), decision tree (DT), random forest (RF) are utilized for classification. Experiment results demonstrate that SVM with linear kernel function can distinguish falling actions best and our approach achieved a detection accuracy of 97.34%, precision of 98.50%, and the recall, F1 score are 97.33%, 97.91% respectively. Compared with previous state-of-art algorithms, our algorithm can achieve the highest recognition accuracy. It proves that our fall detection method is effective.

Falling Detection Research Based on Elderly Behavior Infrared Video Image Contours Ellipse Fitting

Throughout the world, the proportion of the elders in the total population is increasing dramatically, and home-based care has become the most important form of old-age care. Falling is the most common cause of accidents among the elders at home that poses a huge threat to their health and lives. In order to protect the privacy of the elders an accidental falling detection algorithm for the elders in the home has been proposed in this paper. First, contour-based infrared motion video images are used instead of high-definition cameras to collect the elderly behaviors to protect their privacy. Second, ellipse fitting is performed on the infrared video images of the five behaviors including standing, sitting, squatting, bending and falling. The five geometric characteristic variables of the contour-fitting ellipses including the number of ellipses, centroid positions, ellipsoidal areas, horizontal inclinations and long-short axis ratios of the images, have been extracted. Next, an LSTM model is established using the above variables as inputs for feature extraction and classification. Finally, infrared video images of different types of active behaviors of the elders aged from 50 to 70 years have been selected as IFD database for classification detection. Sixty percent of the IFD images are used as training datasets, and 40% of the IFD images are used as test datasets, and compared with the classification detection of URFD datasets which contains optical RGB HD video images of the different behaviors. The experimental results show the effectiveness of the algorithm proposed in this paper which combines the contour ellipse fitting of the infrared video images and the LSTM feature extraction. The average correct classification rate of the normal and falling down behaviors of the elders is above 95%, which is comparable to the optical RGB datasets. The precision of behavior recognition can effectively protect the privacy of the elders, and provide protection for the accidental falling detection of the elders living alone.

A Survey on Recent Advances in Wearable Fall Detection Systems

With advances in medicine and healthcare systems, the average life expectancy of human beings has increased to more than 80 yrs. As a result, the demographic old-age dependency ratio (people aged 65 or above relative to those aged 15–64) is expected to increase, by 2060, from ∼28% to ∼50% in the European Union and from ∼33% to ∼45% in Asia (Ageing Report European Economy, 2015). Therefore, the percentage of people who need additional care is also expected to increase. For instance, per studies conducted by the National Program for Health Care of the Elderly (NPHCE), elderly population in India will increase to 12% of the national population by 2025 with 8%–10% requiring utmost care. Geriatric healthcare has gained a lot of prominence in recent years, with specific focus on fall detection systems (FDSs) because of their impact on public lives. According to a World Health Organization report, the frequency of falls increases with increase in age and frailty. Older people living in nursing homes fall more often than those living in the community and 40% of them experience recurrent falls (World Health Organization, 2007). Machine learning (ML) has found its application in geriatric healthcare systems, especially in FDSs. In this paper, we examine the requirements of a typical FDS. Then we present a survey of the recent work in the area of fall detection systems, with focus on the application of machine learning. We also analyze the challenges in FDS systems based on the literature survey.

Hardware/Software Co-design of Fractal Features based Fall Detection System

Falls are a leading cause of death in older adults and result in high levels of mortality, morbidity and immobility. Fall Detection Systems (FDS) are imperative for timely medical aid and have been known to reduce death rate by 80%. We propose a novel wearable sensor FDS which exploits fractal dynamics of fall accelerometer signals. Fractal dynamics can be used as an irregularity measure of signals and our work shows that it is a key discriminant for classification of falls from other activities of life. We design, implement and evaluate a hardware feature accelerator for computation of fractal features through multi-level wavelet transform on a reconfigurable embedded System on Chip, Zynq device for evaluating wearable accelerometer sensors. The proposed FDS utilises a hardware/software co-design approach with hardware accelerator for fractal features and software implementation of Linear Discriminant Analysis on an embedded ARM core for high accuracy and energy efficiency. The proposed system achieves 99.38% fall detection accuracy, 7.3× speed-up and 6.53× improvements in power consumption, compared to the software only execution with an overall performance per Watt advantage of 47.6×, while consuming low reconfigurable resources at 28.67%.