Human Activity Recognition through Recurrent Neural Networks for Human–Robot Interaction in Agriculture

Human-Robot Interaction in Agriculture: A Systematic Review

In the pursuit of optimizing the efficiency, flexibility, and adaptability of agricultural practices, human-robot interaction (HRI) has emerged in agriculture. Enabled by the ongoing advancement in information and communication technologies, this approach aspires to overcome the challenges originating from the inherent complex agricultural environments. Τhis paper systematically reviews the scholarly literature to capture the current progress and trends in this promising field as well as identify future research directions. It can be inferred that there is a growing interest in this field, which relies on combining perspectives from several disciplines to obtain a holistic understanding. The subject of the selected papers is mainly synergistic target detection, while simulation was the main methodology. Furthermore, melons, grapes, and strawberries were the crops with the highest interest for HRI applications. Finally, collaboration and cooperation were the most preferred interaction modes, with various levels of automation being examined. On all occasions, the synergy of humans and robots demonstrated the best results in terms of system performance, physical workload of workers, and time needed to execute the performed tasks. However, despite the associated progress, there is still a long way to go towards establishing viable, functional, and safe human-robot interactive systems.

Human Activity Prediction Based on Forecasted IMU Activity Signals by Sequence-to-Sequence Deep Neural Networks

Human Activity Recognition (HAR) has gained significant attention due to its broad range of applications, such as healthcare, industrial work safety, activity assistance, and driver monitoring. Most prior HAR systems are based on recorded sensor data (i.e., past information) recognizing human activities. In fact, HAR works based on future sensor data to predict human activities are rare. Human Activity Prediction (HAP) can benefit in multiple applications, such as fall detection or exercise routines, to prevent injuries. This work presents a novel HAP system based on forecasted activity data of Inertial Measurement Units (IMU). Our HAP system consists of a deep learning forecaster of IMU activity signals and a deep learning classifier to recognize future activities. Our deep learning forecaster model is based on a Sequence-to-Sequence structure with attention and positional encoding layers. Then, a pre-trained deep learning Bi-LSTM classifier is used to classify future activities based on the forecasted IMU data. We have tested our HAP system for five daily activities with two tri-axial IMU sensors. The forecasted signals show an average correlation of 91.6% to the actual measured signals of the five activities. The proposed HAP system achieves an average accuracy of 97.96% in predicting future activities.

A Probabilistic Model of Human Activity Recognition with Loose Clothing

Human activity recognition has become an attractive research area with the development of on-body wearable sensing technology. Textiles-based sensors have recently been used for activity recognition. With the latest electronic textile technology, sensors can be incorporated into garments so that users can enjoy long-term human motion recording worn comfortably. However, recent empirical findings suggest, surprisingly, that clothing-attached sensors can actually achieve higher activity recognition accuracy than rigid-attached sensors, particularly when predicting from short time windows. This work presents a probabilistic model that explains improved responsiveness and accuracy with fabric sensing from the increased statistical distance between movements recorded. The accuracy of the comfortable fabric-attached sensor can be increased by 67% more than rigid-attached sensors when the window size is 0.5s. Simulated and real human motion capture experiments with several participants confirm the model's predictions, demonstrating that this counterintuitive effect is accurately captured.

Safety of Automated Agricultural Machineries: A Systematic Literature Review

Automated agricultural machinery has advanced significantly in the previous ten years; however, the ability of such robots to operate safely will be critical to their commercialization. This study provides a holistic evaluation of the work carried out so far in the field of automated agricultural machines’ safety, as well as a framework for future research considerations. Previous automated agricultural machines’ safety-related studies are analyzed and grouped into three categories: (1) environmental perception, (2) risk assessment as well as risk mitigation, and (3) human factors as well as ergonomics. The key findings are as follows: (1) The usage of single perception, multiple perception sensors, developing datasets of agricultural environments, different algorithms, and external solutions to improve sensor performance were all explored as options to improve automated agricultural machines’ safety. (2) Current risk assessment methods cannot be efficient when dealing with new technology, such as automated agricultural machines, due to a lack of pre-existing knowledge. Full compliance with the guidelines provided by the current International Organization for Standardization (ISO 18497) cannot ensure automated agricultural machines’ safety. A regulatory framework and being able to test the functionalities of automated agricultural machines within a reliable software environment are efficient ways to mitigate risks. (3) Knowing foreseeable human activity is critical to ensure safe human–robot interaction.

An Energy-Efficient Method for Recurrent Neural Network Inference in Edge Cloud Computing

Recurrent neural networks (RNNs) are widely used to process sequence-related tasks such as natural language processing. Edge cloud computing systems are in an asymmetric structure, where task managers allocate tasks to the asymmetric edge and cloud computing systems based on computation requirements. In such a computing system, cloud servers have no energy limitations, since they have unlimited energy resources. Edge computing systems, however, are resource-constrained, and the energy consumption is thus expensive, which requires an energy-efficient method for RNN job processing. In this paper, we propose a low-overhead, energy-aware runtime manager to process tasks in edge cloud computing. The RNN task latency is defined as the quality of service (QoS) requirement. Based on the QoS requirements, the runtime manager dynamically assigns RNN inference tasks to edge and cloud computing systems and performs energy optimization on edge systems using dynamic voltage and frequency scaling (DVFS) techniques. Experimental results on a real edge cloud system indicate that in edge systems, our method can reduce the energy up to 45% compared with the state-of-the-art approach.

Data Fusion in Agriculture: Resolving Ambiguities and Closing Data Gaps

Acquiring useful data from agricultural areas has always been somewhat of a challenge, as these are often expansive, remote, and vulnerable to weather events. Despite these challenges, as technologies evolve and prices drop, a surge of new data are being collected. Although a wealth of data are being collected at different scales (i.e., proximal, aerial, satellite, ancillary data), this has been geographically unequal, causing certain areas to be virtually devoid of useful data to help face their specific challenges. However, even in areas with available resources and good infrastructure, data and knowledge gaps are still prevalent, because agricultural environments are mostly uncontrolled and there are vast numbers of factors that need to be taken into account and properly measured for a full characterization of a given area. As a result, data from a single sensor type are frequently unable to provide unambiguous answers, even with very effective algorithms, and even if the problem at hand is well defined and limited in scope. Fusing the information contained in different sensors and in data from different types is one possible solution that has been explored for some decades. The idea behind data fusion involves exploring complementarities and synergies of different kinds of data in order to extract more reliable and useful information about the areas being analyzed. While some success has been achieved, there are still many challenges that prevent a more widespread adoption of this type of approach. This is particularly true for the highly complex environments found in agricultural areas. In this article, we provide a comprehensive overview on the data fusion applied to agricultural problems; we present the main successes, highlight the main challenges that remain, and suggest possible directions for future research.

Wearable Sensor-Based Human Activity Recognition with Transformer Model

Computing devices that can recognize various human activities or movements can be used to assist people in healthcare, sports, or human-robot interaction. Readily available data for this purpose can be obtained from the accelerometer and the gyroscope built into everyday smartphones. Effective classification of real-time activity data is, therefore, actively pursued using various machine learning methods. In this study, the transformer model, a deep learning neural network model developed primarily for the natural language processing and vision tasks, was adapted for a time-series analysis of motion signals. The self-attention mechanism inherent in the transformer, which expresses individual dependencies between signal values within a time series, can match the performance of state-of-the-art convolutional neural networks with long short-term memory. The performance of the proposed adapted transformer method was tested on the largest available public dataset of smartphone motion sensor data covering a wide range of activities, and obtained an average identification accuracy of 99.2% as compared with 89.67% achieved on the same data by a conventional machine learning method. The results suggest the expected future relevance of the transformer model for human activity recognition.

Exploring Data-Driven Components of Socially Intelligent AI through Cooperative Game Paradigms

The development of new approaches for creating more “life-like” artificial intelligence (AI) capable of natural social interaction is of interest to a number of scientific fields, from virtual reality to human–robot interaction to natural language speech systems. Yet how such “Social AI” agents might be manifested remains an open question. Previous research has shown that both behavioral factors related to the artificial agent itself as well as contextual factors beyond the agent (i.e., interaction context) play a critical role in how people perceive interactions with interactive technology. As such, there is a need for customizable agents and customizable environments that allow us to explore both sides in a simultaneous manner. To that end, we describe here the development of a cooperative game environment and Social AI using a data-driven approach, which allows us to simultaneously manipulate different components of the social interaction (both behavioral and contextual). We conducted multiple human–human and human–AI interaction experiments to better understand the components necessary for creation of a Social AI virtual avatar capable of autonomously speaking and interacting with humans in multiple languages during cooperative gameplay (in this case, a social survival video game) in context-relevant ways.

An Information Gain-Based Model and an Attention-Based RNN for Wearable Human Activity Recognition

Wearable sensor-based HAR (human activity recognition) is a popular human activity perception method. However, due to the lack of a unified human activity model, the number and positions of sensors in the existing wearable HAR systems are not the same, which affects the promotion and application. In this paper, an information gain-based human activity model is established, and an attention-based recurrent neural network (namely Attention-RNN) for human activity recognition is designed. Besides, the attention-RNN, which combines bidirectional long short-term memory (BiLSTM) with attention mechanism, was tested on the UCI opportunity challenge dataset. Experiments prove that the proposed human activity model provides guidance for the deployment location of sensors and provides a basis for the selection of the number of sensors, which can reduce the number of sensors used to achieve the same classification effect. In addition, experiments show that the proposed Attention-RNN achieves F1 scores of 0.898 and 0.911 in the ML (Modes of Locomotion) task and GR (Gesture Recognition) task, respectively.

Orchard Mapping with Deep Learning Semantic Segmentation

This study aimed to propose an approach for orchard trees segmentation using aerial images based on a deep learning convolutional neural network variant, namely the U-net network. The purpose was the automated detection and localization of the canopy of orchard trees under various conditions (i.e., different seasons, different tree ages, different levels of weed coverage). The implemented dataset was composed of images from three different walnut orchards. The achieved variability of the dataset resulted in obtaining images that fell under seven different use cases. The best-trained model achieved 91%, 90%, and 87% accuracy for training, validation, and testing, respectively. The trained model was also tested on never-before-seen orthomosaic images or orchards based on two methods (oversampling and undersampling) in order to tackle issues with out-of-the-field boundary transparent pixels from the image. Even though the training dataset did not contain orthomosaic images, it achieved performance levels that reached up to 99%, demonstrating the robustness of the proposed approach.

Machine Learning in Agriculture: A Comprehensive Updated Review

The digital transformation of agriculture has evolved various aspects of management into artificial intelligent systems for the sake of making value from the ever-increasing data originated from numerous sources. A subset of artificial intelligence, namely machine learning, has a considerable potential to handle numerous challenges in the establishment of knowledge-based farming systems. The present study aims at shedding light on machine learning in agriculture by thoroughly reviewing the recent scholarly literature based on keywords' combinations of "machine learning" along with "crop management", "water management", "soil management", and "livestock management", and in accordance with PRISMA guidelines. Only journal papers were considered eligible that were published within 2018-2020. The results indicated that this topic pertains to different disciplines that favour convergence research at the international level. Furthermore, crop management was observed to be at the centre of attention. A plethora of machine learning algorithms were used, with those belonging to Artificial Neural Networks being more efficient. In addition, maize and wheat as well as cattle and sheep were the most investigated crops and animals, respectively. Finally, a variety of sensors, attached on satellites and unmanned ground and aerial vehicles, have been utilized as a means of getting reliable input data for the data analyses. It is anticipated that this study will constitute a beneficial guide to all stakeholders towards enhancing awareness of the potential advantages of using machine learning in agriculture and contributing to a more systematic research on this topic.