Optical Flow-Based 3D Human Motion Estimation from Monocular Video

Subject-specific trunk segmental masses prediction for musculoskeletal models using artificial neural networks

Accurate determination of body segment parameters is crucial for studying human movement and joint forces using musculoskeletal (MSK) models. However, existing methods for predicting segment mass have limited generalizability and sensitivity to body shapes. With recent advancements in machine learning, this study proposed a novel artificial neural network-based method for computing subject-specific trunk segment mass and center of mass (CoM) using only anthropometric measurements. We first developed, trained, and validated two artificial neural networks that used anthropometric measurements as input to predict body shape (ANN1) and tissue mass (ANN2). Then, we calculated trunk segmental mass for two volunteers using the predicted body shape and tissue mass. The body shape model (ANN1) was tested on 279 subjects, and maximum deviation between the predicted body shape and the original was 28 mm. The tissue mass model (ANN2) was evaluated on 223 subjects, which when compared to ground truth data, had a mean error of less than 0.51% in the head, trunk, legs, and arms. We also compared the two volunteer's trunk segment mass with experimental data and found similar trend and magnitude. Our findings suggested that the proposed method could serve as an effective and convenient tool for predicting trunk mass.

Disentangled Human Body Embedding Based on Deep Hierarchical Neural Network

Human bodies exhibit various shapes for different identities or poses, but the body shape has certain similarities in structure and thus can be embedded in a low-dimensional space. This article presents an autoencoder-like network architecture to learn disentangled shape and pose embedding specifically for the 3D human body. This is inspired by recent progress of deformation-based latent representation learning. To improve the reconstruction accuracy, we propose a hierarchical reconstruction pipeline for the disentangling process and construct a large dataset of human body models with consistent connectivity for the learning of the neural network. Our learned embedding can not only achieve superior reconstruction accuracy but also provide great flexibility in 3D human body generation via interpolation, bilinear interpolation, and latent space sampling. The results from extensive experiments demonstrate the powerfulness of our learned 3D human body embedding in various applications.

Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera

We propose the first real-time system for the egocentric estimation of 3D human body pose in a wide range of unconstrained everyday activities. This setting has a unique set of challenges, such as mobility of the hardware setup, and robustness to long capture sessions with fast recovery from tracking failures. We tackle these challenges based on a novel lightweight setup that converts a standard baseball cap to a device for high-quality pose estimation based on a single cap-mounted fisheye camera. From the captured egocentric live stream, our CNN based 3D pose estimation approach runs at 60 Hz on a consumer-level GPU. In addition to the lightweight hardware setup, our other main contributions are: 1) a large ground truth training corpus of top-down fisheye images and 2) a disentangled 3D pose estimation approach that takes the unique properties of the egocentric viewpoint into account. As shown by our evaluation, we achieve lower 3D joint error as well as better 2D overlay than the existing baselines.