Human Motion Understanding for Selecting Action Timing in Collaborative Human-Robot Interaction

Deep Learning-Based Action Detection in Untrimmed Videos: A Survey

Understanding human behavior and activity facilitates advancement of numerous real-world applications, and is critical for video analysis. Despite the progress of action recognition algorithms in trimmed videos, the majority of real-world videos are lengthy and untrimmed with sparse segments of interest. The task of temporal activity detection in untrimmed videos aims to localize the temporal boundary of actions and classify the action categories. Temporal activity detection task has been investigated in full and limited supervision settings depending on the availability of action annotations. This article provides an extensive overview of deep learning-based algorithms to tackle temporal action detection in untrimmed videos with different supervision levels including fully-supervised, weakly-supervised, unsupervised, self-supervised, and semi-supervised. In addition, this article reviews advances in spatio-temporal action detection where actions are localized in both temporal and spatial dimensions. Action detection in online setting is also reviewed where the goal is to detect actions in each frame without considering any future context in a live video stream. Moreover, the commonly used action detection benchmark datasets and evaluation metrics are described, and the performance of the state-of-the-art methods are compared. Finally, real-world applications of temporal action detection in untrimmed videos and a set of future directions are discussed.

MoTiS Parameters for Expressive Multi-Robot Systems: Relative Motion, Timing, and Spacing

Multi-robot systems are moving into human spaces, such as working with people in factories (Bacula et al., in: Companion of the 2020 ACM/IEEE international conference on human–robot interaction, pp 119–121, 2020) or in emergency support (Wagner in Front Robot AI 8, 2021; Baxter et al., in: Autonomous robots and agents, Springer, pp 9–16, 2007) and it is crucial to consider how robots can communicate with the humans in the space. Our work evaluates a parameter framework to allow multi-robot groups of x, y, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\theta $$\end{document}θ robots to effectively communicate using expressive motion. While expressive motion has been extensively studied in single robots (Knight et al., in: 2016 IEEE international conference on intelligent robots and systems (IROS), IEEE, 2016; Bacula and LaViers in Int J Soc Robot, 1–16, 2020; Dragan et al., in: 2013 8th ACM/IEEE international conference on human–robot interaction (HRI), IEEE, pp 301–308, 2013; Kirby et al., in: The 18th IEEE international symposium on robot and human interactive communication, 2009, RO-MAN 2009, IEEE, pp 607–612, 2009), moving to multi-robots creates new challenges as the state space expands and becomes more complex. We evaluate a hierarchical framework of six parameters to generate multi-robot expressive motion consisting of: (1) relative direction, (2) coherence, (3) relative speed, (4) relative start time, (5) proximity, and (6) geometry. We conducted six independent online studies to explore each parameter, finding that four out of six of the parameters had significant impact on people’s perception of the multi-robot group. Additional takeaways of our studies clarify what humans interpret as a robot group, when the group is perceived positively versus negatively, and the critical role of architectural floor plan in interpreting robot intent.

On the Use of Efficient Projection Kernels for Motion-Based Visual Saliency Estimation

In this paper, we investigate the potential of a family of efficient filters—the Gray-Code Kernels (GCKs)—for addressing visual saliency estimation with a focus on motion information. Our implementation relies on the use of 3D kernels applied to overlapping blocks of frames and is able to gather meaningful spatio-temporal information with a very light computation. We introduce an attention module that reasons the use of pooling strategies, combined in an unsupervised way to derive a saliency map highlighting the presence of motion in the scene. A coarse segmentation map can also be obtained. In the experimental analysis, we evaluate our method on publicly available datasets and show that it is able to effectively and efficiently identify the portion of the image where the motion is occurring, providing tolerance to a variety of scene conditions and complexities.

Analysis of Timing and Effect of Visual Cue on Turn-Taking in Human-Robot Interaction

This paper presents a communication robot system with a simple LED display for representing timing for turn-taking in human-robot interaction. Human-like conversation with non-verbal information, such as gestures, facial expressions, tone of voice, and eye contact enables more natural communication. If the robots could use such verbal and non-verbal communication skills, it would establish a social relation between a robot and human. Timing and time interval for turn-taking in human communication are important non-verbal cues to efficiently convey messages and to share opinions with each other. In this study, we present some experimental results to discuss the effect of response timing for turn-taking in communication between a person and a robot.

The MoCA dataset, kinematic and multi-view visual streams of fine-grained cooking actions

MoCA is a bi-modal dataset in which we collect Motion Capture data and video sequences acquired from multiple views, including an ego-like viewpoint, of upper body actions in a cooking scenario. It has been collected with the specific purpose of investigating view-invariant action properties in both biological and artificial systems. Besides that, it represents an ideal test bed for research in a number of fields - including cognitive science and artificial vision - and application domains - as motor control and robotics. Compared to other benchmarks available, MoCA provides a unique compromise for research communities leveraging very different approaches to data gathering: from one extreme of action recognition in the wild - the standard practice nowadays in the fields of Computer Vision and Machine Learning - to motion analysis in very controlled scenarios - as for motor control in biomedical applications. In this work we introduce the dataset and its peculiarities, and discuss a baseline analysis as well as examples of applications for which the dataset is well suited.