Modeling of Deformable Objects for Robotic Manipulation: A Tutorial and Review

Self-Adaptive Perception of Object's Deformability with Multiple Deformation Attributes Utilizing Biomimetic Mechanoreceptors

The perception of object's deformability in unstructured interactions relies on both kinesthetic and cutaneous cues to adapt the uncertainties of an object. However, the existing tactile sensors cannot provide adequate cutaneous cues to self-adaptively estimate the material softness, especially in non-standard contact scenarios where the interacting object deviates from the assumption of an elastic half-infinite body. This paper proposes an innovative design of a tactile sensor that integrates the capabilities of two slow-adapting mechanoreceptors within a soft medium, allowing self-decoupled sensing of local pressure and strain at specific locations within the contact interface. By leveraging these localized cutaneous cues, the sensor can accurately and self-adaptively measure the material softness of an object, accommodating variations in thicknesses and applied forces. Furthermore, when combined with a kinesthetic cue from the robot, the sensor can enhance tactile expression by the synergy of two relevant deformation attributes, including material softness and compliance. It is demonstrated that the biomimetic fusion of tactile information can fully comprehend the deformability of an object, hence facilitating robotic decision-making and dexterous manipulation.

Design of a locust leg-like compliant constant-force mechanism supporting large-scale damage-free manipulation

Precision manipulation is plays an increasingly crucial role in bioengineering fields such as cell injection. Due to the specificity of the operational process, which is highly susceptible and damageable by the actuated force, millimeter-level nondestructive operations are gaining more and more attention. With this, a symmetrical compliant constant-force mechanism (CCFM) is developed to provide stable and large motion stroke for damage-free precision manipulation in this paper. The mechanism design is inspired by the legs of the locust, which flexes and folds when the locust jumps. In terms of structure design, double biomimetic diamond beams are used to generate positive and negative stiffness. A crossbeam is added to the internal diamond mechanism, which flexes during movement to provide negative stiffness, while the external diamond mechanism without additional constraint provides positive stiffness. The theoretical model of this CCFM is established to analyze its force-displacement relationship, which is verified by performing finite element analysis simulations and experimental studies. Meanwhile, a parametric study is conducted to investigate the influence of the dominant design variable of the CCFM. Finally, the test results show that the CCFM can generate motion range up to 5 mm with a constant output force ∼15.2 N. The developed CCFM has potential applications in the field of manipulation techniques of cell engineering and robotics in the future.

Data-Driven Robotic Manipulation of Cloth-like Deformable Objects: The Present, Challenges and Future Prospects

Manipulating cloth-like deformable objects (CDOs) is a long-standing problem in the robotics community. CDOs are flexible (non-rigid) objects that do not show a detectable level of compression strength while two points on the article are pushed towards each other and include objects such as ropes (1D), fabrics (2D) and bags (3D). In general, CDOs' many degrees of freedom (DoF) introduce severe self-occlusion and complex state-action dynamics as significant obstacles to perception and manipulation systems. These challenges exacerbate existing issues of modern robotic control methods such as imitation learning (IL) and reinforcement learning (RL). This review focuses on the application details of data-driven control methods on four major task families in this domain: cloth shaping, knot tying/untying, dressing and bag manipulation. Furthermore, we identify specific inductive biases in these four domains that present challenges for more general IL and RL algorithms.

Cloth manipulation based on category classification and landmark detection

Cloth manipulation remains a challenging problem for the robotic community. Recently, there has been an increased interest in applying deep learning techniques to problems in the fashion industry. As a result, large annotated data sets for cloth category classification and landmark detection were created. In this work, we leverage these advances in deep learning to perform cloth manipulation. We propose a full cloth manipulation framework that, performs category classification and landmark detection based on an image of a garment, followed by a manipulation strategy. The process is performed iteratively to achieve a stretching task where the goal is to bring a crumbled cloth into a stretched out position. We extensively evaluate our learning pipeline and show a detailed evaluation of our framework on different types of garments in a total of 140 recorded and available experiments. Finally, we demonstrate the benefits of training a network on augmented fashion data over using a small robotic-specific data set.

Learning garment manipulation policies toward robot-assisted dressing

Assistive robots have the potential to support people with disabilities in a variety of activities of daily living, such as dressing. People who have completely lost their upper limb movement functionality may benefit from robot-assisted dressing, which involves complex deformable garment manipulation. Here, we report a dressing pipeline intended for these people and experimentally validate it on a medical training manikin. The pipeline is composed of the robot grasping a hospital gown hung on a rail, fully unfolding the gown, navigating around a bed, and lifting up the user's arms in sequence to finally dress the user. To automate this pipeline, we address two fundamental challenges: first, learning manipulation policies to bring the garment from an uncertain state into a configuration that facilitates robust dressing; second, transferring the deformable object manipulation policies learned in simulation to real world to leverage cost-effective data generation. We tackle the first challenge by proposing an active pre-grasp manipulation approach that learns to isolate the garment grasping area before grasping. The approach combines prehensile and nonprehensile actions and thus alleviates grasping-only behavioral uncertainties. For the second challenge, we bridge the sim-to-real gap of deformable object policy transfer by approximating the simulator to real-world garment physics. A contrastive neural network is introduced to compare pairs of real and simulated garment observations, measure their physical similarity, and account for simulator parameters inaccuracies. The proposed method enables a dual-arm robot to put back-opening hospital gowns onto a medical manikin with a success rate of more than 90%.

Action Generative Networks Planning for Deformable Object with Raw Observations

Synthesizing plans for a deformable object to transit from initial observations to goal observations, both of which are represented by high-dimensional data (namely “raw” data), is challenging due to the difficulty of learning abstract state representations of raw data and transition models of continuous states and continuous actions. Even though there have been some approaches making remarkable progress regarding the planning problem, they often neglect actions between observations and are unable to generate action sequences from initial observations to goal observations. In this paper, we propose a novel algorithm framework, namely AGN. We first learn a state-abstractor model to abstract states from raw observations, a state-generator model to generate raw observations from states, a heuristic model to predict actions to be executed in current states, and a transition model to transform current states to next states after executing specific actions. After that, we directly generate plans for a deformable object by performing the four models. We evaluate our approach in continuous domains and show that our approach is effective with comparison to state-of-the-art algorithms.