Fusion Poser: 3D Human Pose Estimation Using Sparse IMUs and Head Trackers in Real Time

Three-Dimensional Human Posture Recognition by Extremity Angle Estimation with Minimal IMU Sensor

Recently, posture recognition technology has advanced rapidly. Herein, we present a novel posture angle calculation system utilizing a single inertial measurement unit and a spatial geometric equation to accurately identify the three-dimensional (3D) motion angles and postures of both the upper and lower limbs of the human body. This wearable system facilitates continuous monitoring of body movements without the spatial limitations or occlusion issues associated with camera-based methods. This posture-recognition system has many benefits. Providing precise posture change information helps users assess the accuracy of their movements, prevent sports injuries, and enhance sports performance. This system employs a single inertial sensor, coupled with a filtering mechanism, to calculate the sensor's trajectory and coordinates in 3D space. Subsequently, the spatial geometry equation devised herein accurately computed the joint angles for changing body postures. To validate its effectiveness, the joint angles estimated from the proposed system were compared with those from dual inertial sensors and image recognition technology. The joint angle discrepancies for this system were within 10° and 5° when compared with dual inertial sensors and image recognition technology, respectively. Such reliability and accuracy of the proposed angle estimation system make it a valuable reference for assessing joint angles.

Human Joint Angle Estimation Using Deep Learning-Based Three-Dimensional Human Pose Estimation for Application in a Real Environment

Human pose estimation (HPE) is a technique used in computer vision and artificial intelligence to detect and track human body parts and poses using images or videos. Widely used in augmented reality, animation, fitness applications, and surveillance, HPE methods that employ monocular cameras are highly versatile and applicable to standard videos and CCTV footage. These methods have evolved from two-dimensional (2D) to three-dimensional (3D) pose estimation. However, in real-world environments, current 3D HPE methods trained on laboratory-based motion capture data encounter challenges, such as limited training data, depth ambiguity, left/right switching, and issues with occlusions. In this study, four 3D HPE methods were compared based on their strengths and weaknesses using real-world videos. Joint position correction techniques were proposed to eliminate and correct anomalies such as left/right inversion and false detections of joint positions in daily life motions. Joint angle trajectories were obtained for intuitive and informative human activity recognition using an optimization method based on a 3D humanoid simulator, with the joint position corrected by the proposed technique as the input. The efficacy of the proposed method was verified by applying it to three types of freehand gymnastic exercises and comparing the joint angle trajectories during motion.

Markerless Motion Tracking With Noisy Video and IMU Data

OBJECTIVE

Marker-based motion capture, considered the gold standard in human motion analysis, is expensive and requires trained personnel. Advances in inertial sensing and computer vision offer new opportunities to obtain research-grade assessments in clinics and natural environments. A challenge that discourages clinical adoption, however, is the need for careful sensor-to-body alignment, which slows the data collection process in clinics and is prone to errors when patients take the sensors home.

METHODS

We propose deep learning models to estimate human movement with noisy data from videos (VideoNet), inertial sensors (IMUNet), and a combination of the two (FusionNet), obviating the need for careful calibration. The video and inertial sensing data used to train the models were generated synthetically from a marker-based motion capture dataset of a broad range of activities and augmented to account for sensor-misplacement and camera-occlusion errors. The models were tested using real data that included walking, jogging, squatting, sit-to-stand, and other activities.

RESULTS

On calibrated data, IMUNet was as accurate as state-of-the-art models, while VideoNet and FusionNet reduced mean ± std root-mean-squared errors by 7.6 ± 5.4 ° and 5.9 ± 3.3 °, respectively. Importantly, all the newly proposed models were less sensitive to noise than existing approaches, reducing errors by up to 14.0 ± 5.3 ° for sensor-misplacement errors of up to 30.0 ± 13.7 ° and by up to 7.4 ± 5.5 ° for joint-center-estimation errors of up to 101.1 ± 11.2 mm, across joints.

CONCLUSION

These tools offer clinicians and patients the opportunity to estimate movement with research-grade accuracy, without the need for time-consuming calibration steps or the high costs associated with commercial products such as Theia3D or Xsens, helping democratize the diagnosis, prognosis, and treatment of neuromusculoskeletal conditions.

SSA Net: Small Scale-Aware Enhancement Network for Human Pose Estimation

In the field of human pose estimation, heatmap-based methods have emerged as the dominant approach, and numerous studies have achieved remarkable performance based on this technique. However, the inherent drawbacks of heatmaps lead to serious performance degradation in methods based on heatmaps for smaller-scale persons. While some researchers have attempted to tackle this issue by improving the performance of small-scale persons, their efforts have been hampered by the continued reliance on heatmap-based methods. To address this issue, this paper proposes the SSA Net, which aims to enhance the detection accuracy of small-scale persons as much as possible while maintaining a balanced perception of persons at other scales. SSA Net utilizes HRNetW48 as a feature extractor and leverages the TDAA module to enhance small-scale perception. Furthermore, it abandons heatmap-based methods and instead adopts coordinate vector regression to represent keypoints. Notably, SSA Net achieved an AP of 77.4% on the COCO Validation dataset, which is superior to other heatmap-based methods. Additionally, it achieved highly competitive results on the Tiny Validation and MPII datasets as well.

The Performance of Inertial Measurement Unit Sensors on Various Hardware Platforms for Binaural Head-Tracking Applications

Binaural synthesis with head tracking is often used in spatial audio systems. The devices used for head tracking must provide data on the orientation of the listener's head. These data need to be highly accurate, and they need to be provided as fast and as frequently as possible. Therefore, head-tracking devices need to be equipped with high-quality inertial measurement unit (IMU) sensors. Since IMUs readily include triaxial accelerometers, gyroscopes, and magnetometers, it is crucial that all of these sensors perform well, as the head orientation is calculated from all sensor outputs. This paper discusses the challenges encountered in the process of the performance assessment of IMUs through appropriate measurements. Three distinct hardware platforms were investigated: five IMU sensors either connected to Arduino-based embedded systems or being an integral part of one, five smartphones across a broad range of overall quality with integrated IMUs, and a commercial virtual reality unit that utilizes a headset with integrated IMUs. An innovative measurement method is presented and proposed for comparing the performance of sensors on all three platforms. The results of the measurements performed using the proposed method show that all three investigated platforms are adequate for the acquisition of the data required for calculating the orientation of a device as the input to the binaural synthesis process. Some limitations that have been observed during the measurements, regarding data acquisition and transfer, are discussed.