Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review

Movement-Based Prosthesis Control with Angular Trajectory Is Getting Closer to Natural Arm Coordination

Traditional myoelectric controls of trans-humeral prostheses fail to provide intuitive coordination of the necessary degrees of freedom. We previously showed that by using artificial neural network predictions to reconstruct distal joints, based on the shoulder posture and movement goals (i.e., position and orientation of the targeted object), participants were able to position and orient an avatar hand to grasp objects with natural arm performances. However, this control involved rapid and unintended prosthesis movements at each modification of the movement goal, impractical for real-life scenarios. Here, we eliminate this abrupt change using novel methods based on an angular trajectory, determined from the speed of stump movement and the gap between the current and the 'goal' distal configurations. These new controls are tested offline and online (i.e., involving participants-in-the-loop) and compared to performances obtained with a natural control. Despite a slight increase in movement time, the new controls allowed twelve valid participants and six participants with trans-humeral limb loss to reach objects at various positions and orientations without prior training. Furthermore, no usability or workload degradation was perceived by participants with upper limb disabilities. The good performances achieved highlight the potential acceptability and effectiveness of those controls for our target population.

Target recognition and segmentation in turbid water using data from non-turbid conditions: a unified approach and experimental validation

Semantic segmentation of targets in underwater images within turbid water environments presents significant challenges, hindered by factors such as environmental variability, difficulties in acquiring datasets, imprecise data annotation, and the poor robustness of conventional methods. This paper addresses this issue by proposing a novel joint method using deep learning to effectively perform semantic segmentation tasks in turbid environments, with the practical case of efficiently collecting polymetallic nodules in deep-sea while minimizing damage to the seabed environment. Our approach includes a novel data expansion technique and a modified U-net based model. Drawing on the underwater image formation model, we introduce noise to clear water images to simulate images captured under varying degrees of turbidity, thus providing an alternative to the required data. Furthermore, traditional U-net-based modified models have shown limitations in enhancing performance in such tasks. Based on the primary factors underlying image degradation, we propose a new model which incorporates an improved dual-channel encoder. Our method significantly advances the fine segmentation of underwater images in turbid media, and experimental validation demonstrates its effectiveness and superiority under different turbidity conditions. The study provides new technical means for deep-sea resource development, holding broad application prospects and scientific value.

Object-Centric Scene Representations Using Active Inference

Representing a scene and its constituent objects from raw sensory data is a core ability for enabling robots to interact with their environment. In this letter, we propose a novel approach for scene understanding, leveraging an object-centric generative model that enables an agent to infer object category and pose in an allocentric reference frame using active inference, a neuro-inspired framework for action and perception. For evaluating the behavior of an active vision agent, we also propose a new benchmark where, given a target viewpoint of a particular object, the agent needs to find the best matching viewpoint given a workspace with randomly positioned objects in 3D. We demonstrate that our active inference agent is able to balance epistemic foraging and goal-driven behavior, and quantitatively outperforms both supervised and reinforcement learning baselines by more than a factor of two in terms of success rate.

A Survey of 6DoF Object Pose Estimation Methods for Different Application Scenarios

Recently, 6DoF object pose estimation has become increasingly important for a broad range of applications in the fields of virtual reality, augmented reality, autonomous driving, and robotic operations. This task involves extracting the target area from the input data and subsequently determining the position and orientation of the objects. In recent years, many new advances have been made in pose estimation. However, existing reviews have the problem of only summarizing category-level or instance-level methods, and not comprehensively summarizing deep learning methods. This paper will provide a comprehensive review of the latest progress in 6D pose estimation to help researchers better understanding this area. In this study, the current methods about 6DoF object pose estimation are mainly categorized into two groups: instance-level and category-level groups, based on whether it is necessary to acquire the CAD model of the object. Recent advancements about learning-based 6DoF pose estimation methods are comprehensively reviewed. The study systematically explores the innovations and applicable scenarios of various methods. It provides an overview of widely used datasets, task metrics, and diverse application scenarios. Furthermore, state-of-the-art methods are compared across publicly accessible datasets, taking into account differences in input data types. Finally, we summarize the challenges of current tasks, methods for different applications, and future development directions.

A neuromorphic dataset for tabletop object segmentation in indoor cluttered environment

Event-based cameras are commonly leveraged to mitigate issues such as motion blur, low dynamic range, and limited time sampling, which plague conventional cameras. However, a lack of dedicated event-based datasets for benchmarking segmentation algorithms, especially those offering critical depth information for occluded scenes, has been observed. In response, this paper introduces a novel Event-based Segmentation Dataset (ESD), a high-quality event 3D spatial-temporal dataset designed for indoor object segmentation within cluttered environments. ESD encompasses 145 sequences featuring 14,166 manually annotated RGB frames, along with a substantial event count of 21.88 million and 20.80 million events from two stereo-configured event-based cameras. Notably, this densely annotated 3D spatial-temporal event-based segmentation benchmark for tabletop objects represents a pioneering initiative, providing event-wise depth, and annotated instance labels, in addition to corresponding RGBD frames. By releasing ESD, our aim is to offer the research community a challenging segmentation benchmark of exceptional quality.

Hunter: Exploring High-Order Consistency for Point Cloud Registration With Severe Outliers

After decades of investigation, point cloud registration is still a challenging task in practice, especially when the correspondences are contaminated by a large number of outliers. It may result in a rapidly decreasing probability of generating a hypothesis close to the true transformation, leading to the failure of point cloud registration. To tackle this problem, we propose a transformation estimation method, named Hunter, for robust point cloud registration with severe outliers. The core of Hunter is to design a global-to-local exploration scheme to robustly find the correct correspondences. The global exploration aims to exploit guided sampling to generate promising initial alignments. To this end, a hypergraph-based consistency reasoning module is introduced to learn the high-order consistency among correct correspondences, which is able to yield a more distinct inlier cluster that facilitates the generation of all-inlier hypotheses. Moreover, we propose a preference-based local exploration module that exploits the preference information of top- k promising hypotheses to find a better transformation. This module can efficiently obtain multiple reliable transformation hypotheses by using a multi-initialization searching strategy. Finally, we present a distance-angle based hypothesis selection criterion to choose the most reliable transformation, which can avoid selecting symmetrically aligned false transformations. Experimental results on simulated, indoor, and outdoor datasets, demonstrate that Hunter can achieve significant superiority over the state-of-the-art methods, including both learning-based and traditional methods (as shown in Fig. 1). Moreover, experimental results also indicate that Hunter can achieve more stable performance compared with all other methods with severe outliers.

Secure Grasping Detection of Objects in Stacked Scenes Based on Single-Frame RGB Images

Secure grasping of objects in complex scenes is the foundation of many tasks. It is important for robots to autonomously determine the optimal grasp based on visual information, which requires reasoning about the stacking relationship of objects and detecting the grasp position. This paper proposes a multi-task secure grasping detection model, which consists of the grasping relationship network (GrRN) and the oriented rectangles detection network CSL-YOLO, which uses circular smooth label (CSL). GrRN uses DETR to solve set prediction problems in object detection, enabling end-to-end detection of grasping relationships. CSL-YOLO uses classification to predict the angle of oriented rectangles, and solves the angle distance problem caused by classification. Experiments on the Visual Manipulate Relationship Dataset (VMRD) and the grasping detection dataset Cornell demonstrate that our method outperforms existing methods and exhibits good applicability on robot platforms.

A Variable Photo-Model Method for Object Pose and Size Estimation with Stereo Vision in a Complex Home Scene

Model-based stereo vision methods can estimate the 6D poses of rigid objects. They can help robots to achieve a target grip in complex home environments. This study presents a novel approach, called the variable photo-model method, to estimate the pose and size of an unknown object using a single photo of the same category. By employing a pre-trained You Only Look Once (YOLO) v4 weight for object detection and 2D model generation in the photo, the method converts the segmented 2D photo-model into 3D flat photo-models assuming different sizes and poses. Through perspective projection and model matching, the method finds the best match between the model and the actual object in the captured stereo images. The matching fitness function is optimized using a genetic algorithm (GA). Unlike data-driven approaches, this approach does not require multiple photos or pre-training time for single object pose recognition, making it more versatile. Indoor experiments demonstrate the effectiveness of the variable photo-model method in estimating the pose and size of the target objects within the same class. The findings of this study have practical implications for object detection prior to robotic grasping, particularly due to its ease of application and the limited data required.

CNN-Based Vehicle Bottom Face Quadrilateral Detection Using Surveillance Cameras for Intelligent Transportation Systems

In intelligent transportation systems, it is essential to estimate the vehicle position accurately. To this end, it is preferred to detect vehicles as a bottom face quadrilateral (BFQ) rather than an axis-aligned bounding box. Although there have been some methods for detecting the vehicle BFQ using vehicle-mounted cameras, few studies have been conducted using surveillance cameras. Therefore, this paper conducts a comparative study on various approaches for detecting the vehicle BFQ in surveillance camera environments. Three approaches were selected for comparison, including corner-based, position/size/angle-based, and line-based. For comparison, this paper suggests a way to implement the vehicle BFQ detectors by simply adding extra heads to one of the most widely used real-time object detectors, YOLO. In experiments, it was shown that the vehicle BFQ can be adequately detected by using the suggested implementation, and the three approaches were quantitatively evaluated, compared, and analyzed.

A Survey on Deep Learning Based Segmentation, Detection and Classification for 3D Point Clouds

The computer vision, graphics, and machine learning research groups have given a significant amount of focus to 3D object recognition (segmentation, detection, and classification). Deep learning approaches have lately emerged as the preferred method for 3D segmentation problems as a result of their outstanding performance in 2D computer vision. As a result, many innovative approaches have been proposed and validated on multiple benchmark datasets. This study offers an in-depth assessment of the latest developments in deep learning-based 3D object recognition. We discuss the most well-known 3D object recognition models, along with evaluations of their distinctive qualities.

Learning-based robotic grasping: A review

As personalization technology increasingly orchestrates individualized shopping or marketing experiences in industries such as logistics, fast-moving consumer goods, and food delivery, these sectors require flexible solutions that can automate object grasping for unknown or unseen objects without much modification or downtime. Most solutions in the market are based on traditional object recognition and are, therefore, not suitable for grasping unknown objects with varying shapes and textures. Adequate learning policies enable robotic grasping to accommodate high-mix and low-volume manufacturing scenarios. In this paper, we review the recent development of learning-based robotic grasping techniques from a corpus of over 150 papers. In addition to addressing the current achievements from researchers all over the world, we also point out the gaps and challenges faced in AI-enabled grasping, which hinder robotization in the aforementioned industries. In addition to 3D object segmentation and learning-based grasping benchmarks, we have also performed a comprehensive market survey regarding tactile sensors and robot skin. Furthermore, we reviewed the latest literature on how sensor feedback can be trained by a learning model to provide valid inputs for grasping stability. Finally, learning-based soft gripping is evaluated as soft grippers can accommodate objects of various sizes and shapes and can even handle fragile objects. In general, robotic grasping can achieve higher flexibility and adaptability, when equipped with learning algorithms.

Machine learning unifies flexibility and efficiency of spinodal structure generation for stochastic biomaterial design

Porous biomaterials design for bone repair is still largely limited to regular structures (e.g. rod-based lattices), due to their easy parameterization and high controllability. The capability of designing stochastic structure can redefine the boundary of our explorable structure-property space for synthesizing next-generation biomaterials. We hereby propose a convolutional neural network (CNN) approach for efficient generation and design of spinodal structure-an intriguing structure with stochastic yet interconnected, smooth, and constant pore channel conducive to bio-transport. Our CNN-based approach simultaneously possesses the tremendous flexibility of physics-based model in generating various spinodal structures (e.g. periodic, anisotropic, gradient, and arbitrarily large ones) and comparable computational efficiency to mathematical approximation model. We thus successfully design spinodal bone structures with target anisotropic elasticity via high-throughput screening, and directly generate large spinodal orthopedic implants with desired gradient porosity. This work significantly advances stochastic biomaterials development by offering an optimal solution to spinodal structure generation and design.

6D Object Localization in Car-Assembly Industrial Environment

In this work, a visual object detection and localization workflow integrated into a robotic platform is presented for the 6D pose estimation of objects with challenging characteristics in terms of weak texture, surface properties and symmetries. The workflow is used as part of a module for object pose estimation deployed to a mobile robotic platform that exploits the Robot Operating System (ROS) as middleware. The objects of interest aim to support robot grasping in the context of human-robot collaboration during car door assembly in industrial manufacturing environments. In addition to the special object properties, these environments are inherently characterised by cluttered background and unfavorable illumination conditions. For the purpose of this specific application, two different datasets were collected and annotated for training a learning-based method that extracts the object pose from a single frame. The first dataset was acquired in controlled laboratory conditions and the second in the actual indoor industrial environment. Different models were trained based on the individual datasets and a combination of them were further evaluated in a number of test sequences from the actual industrial environment. The qualitative and quantitative results demonstrate the potential of the presented method in relevant industrial applications.

Visuo-haptic object perception for robots: an overview

The object perception capabilities of humans are impressive, and this becomes even more evident when trying to develop solutions with a similar proficiency in autonomous robots. While there have been notable advancements in the technologies for artificial vision and touch, the effective integration of these two sensory modalities in robotic applications still needs to be improved, and several open challenges exist. Taking inspiration from how humans combine visual and haptic perception to perceive object properties and drive the execution of manual tasks, this article summarises the current state of the art of visuo-haptic object perception in robots. Firstly, the biological basis of human multimodal object perception is outlined. Then, the latest advances in sensing technologies and data collection strategies for robots are discussed. Next, an overview of the main computational techniques is presented, highlighting the main challenges of multimodal machine learning and presenting a few representative articles in the areas of robotic object recognition, peripersonal space representation and manipulation. Finally, informed by the latest advancements and open challenges, this article outlines promising new research directions.

Design and Evaluation of Capacitive Smart Transducer for a Forestry Crane Gripper

Stable grasps are essential for robots handling objects. This is especially true for "robotized" large industrial machines as heavy and bulky objects that are unintentionally dropped by the machine can lead to substantial damages and pose a significant safety risk. Consequently, adding a proximity and tactile sensing to such large industrial machinery can help to mitigate this problem. In this paper, we present a sensing system for proximity/tactile sensing in gripper claws of a forestry crane. In order to avoid difficulties with respect to the installation of cables (in particular in retrofitting of existing machinery), the sensors are truly wireless and can be powered using energy harvesting, leading to autarkic, i.e., self-contained, sensors. The sensing elements are connected to a measurement system which transmits the measurement data to the crane automation computer via Bluetooth low energy (BLE) compliant to IEEE 1451.0 (TEDs) specification for eased logical system integration. We demonstrate that the sensor system can be fully integrated in the grasper and that it can withstand the challenging environmental conditions. We present experimental evaluation of detection in various grasping scenarios such as grasping at an angle, corner grasping, improper closure of the gripper and proper grasp for logs of three different sizes. Results indicate the ability to detect and differentiate between good and poor grasping configurations.

A Comprehensive Study of 3-D Vision-Based Robot Manipulation

Robot manipulation, for example, pick-and-place manipulation, is broadly used for intelligent manufacturing with industrial robots, ocean engineering with underwater robots, service robots, or even healthcare with medical robots. Most traditional robot manipulations adopt 2-D vision systems with plane hypotheses and can only generate 3-DOF (degrees of freedom) pose accordingly. To mimic human intelligence and endow the robot with more flexible working capabilities, 3-D vision-based robot manipulation has been studied. However, this task is still challenging in the open world especially for general object recognition and pose estimation with occlusion in cluttered backgrounds and human-like flexible manipulation. In this article, we propose a comprehensive analysis of recent progress about the 3-D vision for robot manipulation, including 3-D data acquisition and representation, robot-vision calibration, 3-D object detection/recognition, 6-DOF pose estimation, grasping estimation, and motion planning. We then present some public datasets, evaluation criteria, comparisons, and challenges. Finally, the related application domains of robot manipulation are given, and some future directions and open problems are studied as well.

Refined Voting and Scene Feature Fusion for 3D Object Detection in Point Clouds

An essential task for 3D visual world understanding is 3D object detection in lidar point clouds. To predict directly bounding box parameters from point clouds, existing voting-based methods use Hough voting to obtain the centroid of each object. However, it may be difficult for the inaccurately voted centers to regress boxes accurately, leading to the generation of redundant bounding boxes. For objects in indoor scenes, there are several co-occurrence patterns for objects in indoor scenes. Concurrently, semantic relations between object layouts and scenes can be used as prior context to guide object detection. We propose a simple, yet effective network, RSFF-Net, which adds refined voting and scene feature fusion for indoor 3D object detection. The RSFF-Net consists of three modules: geometric function, refined voting, and scene constraint. First, a geometric function module is used to capture the geometric features of the nearest object of the voted points. Then, the coarse votes are revoted by a refined voting module, which is based on the fused feature between the coarse votes and geometric features. Finally, a scene constraint module is used to add the association information between candidate objects and scenes. RSFF-Net achieves competitive results on indoor 3D object detection benchmarks: ScanNet V2 and SUN RGB-D.

A Comprehensive Review of Vision-Based Robotic Applications: Current State, Components, Approaches, Barriers, and Potential Solutions

Being an emerging technology, robotic manipulation has encountered tremendous advancements due to technological developments starting from using sensors to artificial intelligence. Over the decades, robotic manipulation has advanced in terms of the versatility and flexibility of mobile robot platforms. Thus, robots are now capable of interacting with the world around them. To interact with the real world, robots require various sensory inputs from their surroundings, and the use of vision is rapidly increasing nowadays, as vision is unquestionably a rich source of information for a robotic system. In recent years, robotic manipulators have made significant progress towards achieving human-like abilities. There is still a large gap between human and robot dexterity, especially when it comes to executing complex and long-lasting manipulations. This paper comprehensively investigates the state-of-the-art development of vision-based robotic application, which includes the current state, components, and approaches used along with the algorithms with respect to the control and application of robots. Furthermore, a comprehensive analysis of those vision-based applied algorithms, their effectiveness, and their complexity has been enlightened here. To conclude, there is a discussion over the constraints while performing the research and potential solutions to develop a robust and accurate vision-based robot manipulation.

Research on Intelligent Robot Point Cloud Grasping in Internet of Things

The development of Internet of Things (IoT) technology has enabled intelligent robots to have more sensing and decision-making capabilities, broadening the application areas of robots. Grasping operation is one of the basic tasks of intelligent robots, and vision-based robot grasping technology can enable robots to perform dexterous grasping. Compared with 2D images, 3D point clouds based on objects can generate more reasonable and stable grasping poses. In this paper, we propose a new algorithm structure based on the PointNet network to process object point cloud information. First, we use the T-Net network to align the point cloud to ensure its rotation invariance; then we use a multilayer perceptron to extract point cloud characteristics and use the symmetric function to get global features, while adding the point cloud characteristics attention mechanism to make the network more focused on the object local point cloud. Finally, a grasp quality evaluation network is proposed to evaluate the quality of the generated candidate grasp positions, and the grasp with the highest score is obtained. A grasping dataset is generated based on the YCB dataset to train the proposed network, which achieves excellent classification accuracy. The actual grasping experiments are carried out using the Baxter robot and compared with the existing methods; the proposed method achieves good grasping effect.

Embodied Object Representation Learning and Recognition

Scene understanding and decomposition is a crucial challenge for intelligent systems, whether it is for object manipulation, navigation, or any other task. Although current machine and deep learning approaches for object detection and classification obtain high accuracy, they typically do not leverage interaction with the world and are limited to a set of objects seen during training. Humans on the other hand learn to recognize and classify different objects by actively engaging with them on first encounter. Moreover, recent theories in neuroscience suggest that cortical columns in the neocortex play an important role in this process, by building predictive models about objects in their reference frame. In this article, we present an enactive embodied agent that implements such a generative model for object interaction. For each object category, our system instantiates a deep neural network, called Cortical Column Network (CCN), that represents the object in its own reference frame by learning a generative model that predicts the expected transform in pixel space, given an action. The model parameters are optimized through the active inference paradigm, i.e., the minimization of variational free energy. When provided with a visual observation, an ensemble of CCNs each vote on their belief of observing that specific object category, yielding a potential object classification. In case the likelihood on the selected category is too low, the object is detected as an unknown category, and the agent has the ability to instantiate a novel CCN for this category. We validate our system in an simulated environment, where it needs to learn to discern multiple objects from the YCB dataset. We show that classification accuracy improves as an embodied agent can gather more evidence, and that it is able to learn about novel, previously unseen objects. Finally, we show that an agent driven through active inference can choose their actions to reach a preferred observation.

A Survey of 6D Object Detection Based on 3D Models for Industrial Applications

Six-dimensional object detection of rigid objects is a problem especially relevant for quality control and robotic manipulation in industrial contexts. This work is a survey of the state of the art of 6D object detection with these use cases in mind, specifically focusing on algorithms trained only with 3D models or renderings thereof. Our first contribution is a listing of requirements typically encountered in industrial applications. The second contribution is a collection of quantitative evaluation results for several different 6D object detection methods trained with synthetic data and the comparison and analysis thereof. We identify the top methods for individual requirements that industrial applications have for object detectors, but find that a lack of comparable data prevents large-scale comparison over multiple aspects.

Heuristic EPnP-Based Pose Estimation for Underground Machine Tracking

Pose estimation is one of the most complicated and compromising problems for underground mining machine tracking, and it is particularly important for hydraulic support autonomous following mining machine (AFM) policy-making system. In this paper, a low-cost infrared vision-based system through an Efficient Perspective-n-Point (EPnP) algorithm is proposed. To improve efficiency and simplify computation, a traditional EPnP algorithm is modified through a nature-inspired heuristic optimization algorithm. The optimized algorithm is integrated into the AFM policy-making system to estimate the relative pose (R-Pose) estimation between hydraulic support and the mining machine’s shearer drum. Simple yet effective numerical simulations and industrial experiments were carried out to validate the proposed method. The pose estimation error was ≤1% under normal lighting and illuminance, and ≤2% in a simulated underground environment, which was accurate enough to meet the needs of practical applications. Both numerical simulation and industrial experiment proved the superiority of the approach.

Evaluation of Vision-Based Hand Tool Tracking Methods for Quality Assessment and Training in Human-Centered Industry 4.0

Smart industrial workstations for the training and evaluation of workers are an innovative approach to face the problems of manufacturing quality assessment and fast training. However, such products do not implement algorithms that are able to accurately track the pose of a hand tool that might also be partially occluded by the operator’s hands. In the best case, the already proposed systems roughly track the position of the operator’s hand center assuming that a certain task has been performed if the hand center position is close enough to a specified area. The problem of the pose estimation of 3D objects, including the hand tool, is an open and debated problem. The methods that lead to high accuracies are time consuming and require a 3D model of the object to detect, which is why they cannot be adopted for a real-time training system. The rise in deep learning has stimulated the search for better-performing vision-based solutions. Nevertheless, the problem of hand tool pose estimation for assembly and training procedures appears to not have been extensively investigated. In this study, four different vision-based methods based, respectively, on ArUco markers, OpenPose, Azure Kinect Body Tracking and the YOLO network have been proposed in order to estimate the position of a specific point of interest of the tool that has to be tracked in real-time during an assembly or maintenance procedure. The proposed approaches have been tested on a real scenario with four users handling a power drill simulating three different conditions during an assembly procedure. The performance of the methods has been evaluated and compared with the HTC Vive tracking system as a benchmark. Then, the advantages and drawbacks in terms of the accuracy and invasiveness of the method have been discussed. The authors can state that OpenPose is the most robust proposal arising from the study. The authors will investigate the OpenPose performance in more depth in further studies. The framework appears to be very interesting regarding its integration into a smart workstation for quality assessment and training.

An Improved Estimation Algorithm of Space Targets Pose Based on Multi-Modal Feature Fusion

The traditional estimation methods of space targets pose are based on artificial features to match the transformation relationship between the image and the object model. With the explosion of deep learning technology, the approach based on deep neural networks (DNN) has significantly improved the performance of pose estimation. However, the current methods still have problems such as complex calculation, low accuracy, and poor real-time performance. Therefore, a new pose estimation algorithm is proposed in this paper. Firstly, the mask image of the target is obtained by the instance segmentation algorithm, and its point cloud image is obtained based on a depth map combined with camera parameters. Finally, the correlation among points is established to realize the prediction of pose based on multi-modal feature fusion. Experimental results in the YCB-Video dataset show that the proposed algorithm can recognize complex images at a speed of about 24 images per second with an accuracy of more than 80%. In conclusion, the proposed algorithm can realize fast pose estimation for complex stacked objects and has strong stability for different objects.

Vision-Based Robotic Arm Control Algorithm Using Deep Reinforcement Learning for Autonomous Objects Grasping

While working side-by-side, humans and robots complete each other nowadays, and we may say that they work hand in hand. This study aims to evolve the grasping task by reaching the intended object based on deep reinforcement learning. Thereby, in this paper, we propose a deep deterministic policy gradient approach that can be applied to a numerous-degrees-of-freedom robotic arm towards autonomous objects grasping according to their classification and a given task. In this study, this approach is realized by a five-degrees-of-freedom robotic arm that reaches the targeted object using the inverse kinematics method. You Only Look Once v5 is employed for object detection, and backward projection is used to detect the three-dimensional position of the target. After computing the angles of the joints at the detected position by inverse kinematics, the robot’s arm is moved towards the target object’s emplacement thanks to the algorithm. Our approach provides a neural inverse kinematics solution that increases overall performance, and its simulation results reveal its advantages compared to the traditional one. The robot’s end grip joint can reach the targeted location by calculating the angle of every joint with an acceptable range of error. However, the accuracy of the angle and the posture are satisfied. Experiments reveal the performance of our proposal compared to the state-of-the-art approaches in vision-based grasp tasks. This is a new approach to grasp an object by referring to inverse kinematics. This method is not only easier than the standard one but is also more meaningful for multi-degrees of freedom robots.

Aiding Grasp Synthesis for Novel Objects Using Heuristic-Based and Data-Driven Active Vision Methods

In this work, we present several heuristic-based and data-driven active vision strategies for viewpoint optimization of an arm-mounted depth camera to aid robotic grasping. These strategies aim to efficiently collect data to boost the performance of an underlying grasp synthesis algorithm. We created an open-source benchmarking platform in simulation (https://github.com/galenbr/2021ActiveVision), and provide an extensive study for assessing the performance of the proposed methods as well as comparing them against various baseline strategies. We also provide an experimental study with a real-world two finger parallel jaw gripper setup by utilizing an existing grasp planning benchmark in the literature. With these analyses, we were able to quantitatively demonstrate the versatility of heuristic methods that prioritize certain types of exploration, and qualitatively show their robustness to both novel objects and the transition from simulation to the real world. We identified scenarios in which our methods did not perform well and objectively difficult scenarios, and present a discussion on which avenues for future research show promise.

Robotics Dexterous Grasping: The Methods Based on Point Cloud and Deep Learning

Dexterous manipulation, especially dexterous grasping, is a primitive and crucial ability of robots that allows the implementation of performing human-like behaviors. Deploying the ability on robots enables them to assist and substitute human to accomplish more complex tasks in daily life and industrial production. A comprehensive review of the methods based on point cloud and deep learning for robotics dexterous grasping from three perspectives is given in this paper. As a new category schemes of the mainstream methods, the proposed generation-evaluation framework is the core concept of the classification. The other two classifications based on learning modes and applications are also briefly described afterwards. This review aims to afford a guideline for robotics dexterous grasping researchers and developers.

Outline viewpoint feature histogram: An improved point cloud descriptor for recognition and grasping of workpieces

We propose an improved point cloud global descriptor for recognition and grasping of similar workpieces. In the industry, different types of workpieces need to be recognized precisely in some intelligent systems. Deep learning requires a lot of preparation work, and it is difficult to adapt to the variety of workpieces. Furthermore, traditional descriptors based on point pairs cannot meet the requirements of identification. To solve this problem, the Outline Viewpoint Feature Histogram (Outline-VFH) descriptor remains part of the recognition ability of the Viewpoint Feature Histogram (VFH) descriptor and contains an extra outline description, which is established based on the oriented bounding box theory. To validate the effectiveness of the proposed descriptor, experiments were conducted on public dataset and some physical workpieces. The results show that the Outline-VFH is much better than VFH and some other descriptors on recognition and has great potential in vision-based robot grasping applications.