1
|
Qi J, Zhou P, Ran G, Gao H, Wang P, Li D, Gao Y, Navarro-Alarcon D. Model predictive manipulation of compliant objects with multi-objective optimizer and adversarial network for occlusion compensation. ISA TRANSACTIONS 2024; 150:359-373. [PMID: 38797650 DOI: 10.1016/j.isatra.2024.05.015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 05/08/2024] [Accepted: 05/08/2024] [Indexed: 05/29/2024]
Abstract
BACKGROUND The manipulation of compliant objects by robotic systems remains a challenging task, largely due to their variable shapes and the complex, high-dimensional nature of their interaction dynamics. Traditional robotic manipulation strategies struggle with the accurate modeling and control necessary to handle such materials, especially in the presence of visual occlusions that frequently occur in dynamic environments. Meanwhile, for most unstructured environments, robots are required to have autonomous interactions with their surroundings. METHODS To solve the shape manipulation of compliant objects in an unstructured environment, we begin by exploring the regression-based algorithm of representing the high-dimensional configuration space of deformable objects in a compressed form that enables efficient and effective manipulation. Simultaneously, we address the issue of visual occlusions by proposing the integration of an adversarial network, enabling guiding the shaping task even with partial observations of the object. Afterwards, we propose a receding-time estimator to coordinate the robot action with the computed shape features while satisfying various performance criteria. Finally, model predictive controller is utilized to compute the robot's shaping motions subject to safety constraints. Detailed experiments are presented to evaluate the proposed manipulation framework. SIGNIFICANT FINDINGS Our MPC framework utilizes the compressed representation and occlusion-compensated information to predict the object's behavior, while the multi-objective optimizer ensures that the resulting control actions meet multiple performance criteria. Through rigorous experimental validation, our approach demonstrates superior manipulation capabilities in scenarios with visual obstructions, outperforming existing methods in terms of precision and operational reliability. The findings highlight the potential of our integrated approach to significantly enhance the manipulation of compliant objects in real-world robotic applications.
Collapse
Affiliation(s)
- Jiaming Qi
- Centre for Transformative Garment Production, The Hong Kong University, NT, Hong Kong.
| | - Peng Zhou
- Centre for Transformative Garment Production, The Hong Kong University, NT, Hong Kong.
| | - Guangtao Ran
- Harbin Institute of Technology, Department of Control Science and Engineering, Heilongjiang, China.
| | - Han Gao
- The School of Automation, Beijing Institution of Technology, Beijing, China.
| | - Pengyu Wang
- Department of Aerospace Engineering, Korea Advanced Institute of Science and Technology, Republic of Korea.
| | - Dongyu Li
- Beihang University, School of Cyber Science and Technology, Beijing, China.
| | - Yufeng Gao
- Harbin Institute of Technology, Department of Control Science and Engineering, Heilongjiang, China.
| | - David Navarro-Alarcon
- The Hong Kong Polytechnic University, Department of Mechanical Engineering, Kowloon, Hong Kong.
| |
Collapse
|
2
|
Misaros M, Stan OP, Donca IC, Miclea LC. Autonomous Robots for Services-State of the Art, Challenges, and Research Areas. SENSORS (BASEL, SWITZERLAND) 2023; 23:4962. [PMID: 37430875 DOI: 10.3390/s23104962] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/18/2023] [Revised: 05/14/2023] [Accepted: 05/17/2023] [Indexed: 07/12/2023]
Abstract
It has been almost half a century since the first interest in autonomous robots was shown, and research is still continuing to improve their ability to make perfectly conscious decisions from a user safety point of view. These autonomous robots are now at a fairly advanced level, which means that their adoption rate in social environments is also increasing. This article reviews the current state of development of this technology and highlights the evolution of interest in it. We analyze and discuss specific areas of its use, for example, its functionality and current level of development. Finally, challenges related to the current level of research and new methods that are still being developed for the wider adoption of these autonomous robots are highlighted.
Collapse
Affiliation(s)
- Marius Misaros
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania
| | - Ovidiu-Petru Stan
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania
| | - Ionut-Catalin Donca
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania
| | - Liviu-Cristian Miclea
- Department of Automation, Faculty of Automation and Computer Science, Technical University of Cluj-Napoca, 400114 Cluj-Napoca, Romania
| |
Collapse
|
3
|
Fiorini P, Goldberg KY, Liu Y, Taylor RH. Concepts and Trends n Autonomy for Robot-Assisted Surgery. PROCEEDINGS OF THE IEEE. INSTITUTE OF ELECTRICAL AND ELECTRONICS ENGINEERS 2022; 110:993-1011. [PMID: 35911127 PMCID: PMC7613181 DOI: 10.1109/jproc.2022.3176828] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
Abstract
Surgical robots have been widely adopted with over 4000 robots being used in practice daily. However, these are telerobots that are fully controlled by skilled human surgeons. Introducing "surgeon-assist"-some forms of autonomy-has the potential to reduce tedium and increase consistency, analogous to driver-assist functions for lanekeeping, cruise control, and parking. This article examines the scientific and technical backgrounds of robotic autonomy in surgery and some ethical, social, and legal implications. We describe several autonomous surgical tasks that have been automated in laboratory settings, and research concepts and trends.
Collapse
Affiliation(s)
- Paolo Fiorini
- Department of Computer Science, University of Verona, 37134 Verona, Italy
| | - Ken Y. Goldberg
- Department of Industrial Engineering and Operations Research and the Department of Electrical Engineering and Computer Science, University of California at Berkeley, Berkeley, CA 94720 USA
| | - Yunhui Liu
- Department of Mechanical and Automation Engineering, T Stone Robotics Institute, The Chinese University of Hong Kong, Hong Kong, China
| | - Russell H. Taylor
- Department of Computer Science, the Department of Mechanical Engineering, the Department of Radiology, the Department of Surgery, and the Department of Otolaryngology, Head-and-Neck Surgery, Johns Hopkins University, Baltimore, MD 21218 USA, and also with the Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA
| |
Collapse
|
4
|
Nazari AA, Zareinia K, Janabi-Sharifi F. Visual servoing of continuum robots: Methods, challenges, and prospects. Int J Med Robot 2022; 18:e2384. [PMID: 35199451 DOI: 10.1002/rcs.2384] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/14/2021] [Revised: 02/16/2022] [Accepted: 02/19/2022] [Indexed: 01/05/2023]
Abstract
BACKGROUND Recent advancements in continuum robotics have accentuated developing efficient and stable controllers to handle shape deformation and compliance. The control of continuum robots (CRs) using physical sensors attached to the robot, particularly in confined spaces, is difficult due to their limited accuracy in three-dimensional deflections and challenging localisation. Therefore, using non-contact imaging sensors finds noticeable importance, particularly in medical scenarios. Accordingly, given the need for direct control of the robot tip and notable uncertainties in the kinematics and dynamics of CRs, many papers have focussed on the visual servoing (VS) of CRs in recent years. METHODS The significance of this research towards safe human-robot interaction has fuelled our survey on the previous methods, current challenges, and future opportunities. RESULTS Beginning with actuation modalities and modelling approaches, the paper investigates VS methods in medical and non-medical scenarios. CONCLUSIONS Finally, challenges and prospects of VS for CRs are discussed, followed by concluding remarks.
Collapse
Affiliation(s)
- Ali A Nazari
- Department of Mechanical and Industrial Engineering, Ryerson University, Toronto, Ontario, Canada
| | - Kourosh Zareinia
- Department of Mechanical and Industrial Engineering, Ryerson University, Toronto, Ontario, Canada
| | - Farrokh Janabi-Sharifi
- Department of Mechanical and Industrial Engineering, Ryerson University, Toronto, Ontario, Canada
| |
Collapse
|
5
|
Shetab-Bushehri M, Aranda M, Mezouar Y, Ozgur E. As-Rigid-as-Possible Shape Servoing. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3145960] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
6
|
Sui C, He K, Lyu C, Liu YH. Accurate 3D Reconstruction of Dynamic Objects by Spatial-Temporal Multiplexing and Motion-Induced Error Elimination. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2022; 31:2106-2121. [PMID: 35167454 DOI: 10.1109/tip.2022.3150297] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Three-dimensional (3D) reconstruction of dynamic objects has broad applications, including object recognition and robotic manipulation. However, achieving high-accuracy reconstruction and robustness to motion simultaneously is a challenging task. In this paper, we present a novel method for 3D reconstruction of dynamic objectS, whose main features are as follows. Firstly, a structured-light multiplexing method is developed that only requires 3 patterns to achieve high-accuracy encoding. Fewer projected patterns require shorter image acquisition time, thus, the object motion is reduced in each reconstruction cycle. The three patterns, i.e. spatial-temporally encoded patterns, are generated by embedding a specifically designed spatial-coded texture map into the temporal-encoded three-step phase-shifting fringes. A temporal codeword and three spatial codewords are extracted from the composite patterns using a proposed extraction algorithm. The two types of codewords are utilized separately in stereo matching: the temporal codeword ensures the high accuracy, while the spatial codewords are responsible for removing phase ambiguity. Secondly, we aim to eliminate the reconstruction error induced by motion between frames abbreviated as motion induced error (MiE). Instead of assuming the object to be static when acquiring the 3 images, we derive the motion of projection pixels among frames. Using the extracted spatial codewords, correspondences between different frames are found, i.e. pixels with the same codewords are traceable in the image sequences. Therefore, we can obtain the phase map at each image-acquisition moment without being affected by the object motion. Then the object surfaces corresponding to all the images can be recovered. Experimental results validate the high reconstruction accuracy and precision of the proposed method for dynamic objects with different motion speeds. Comparative experiments show that the presented method demonstrates superior performance with various types of motion, including translation in different directions and deformation.
Collapse
|
7
|
A Switching Image-Based Visual Servoing Method for Cooperative Continuum Robots. J INTELL ROBOT SYST 2021. [DOI: 10.1007/s10846-021-01435-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
8
|
Cabibihan JJ, Alhaddad AY, Gulrez T, Yoon WJ. Influence of Visual and Haptic Feedback on the Detection of Threshold Forces in a Surgical Grasping Task. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3068934] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
9
|
Huang J, Cai Y, Chu X, Taylor RH, Au KWS. Non-Fixed Contact Manipulation Control Framework for Deformable Objects With Active Contact Adjustment. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3062302] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
10
|
Alambeigi F, Pedram SA, Speyer JL, Rosen J, Iordachita I, Taylor RH, Armand M. SCADE: Simultaneous Sensor Calibration and Deformation Estimation of FBG-Equipped Unmodeled Continuum Manipulators. IEEE T ROBOT 2020; 36:222-239. [PMID: 32661460 PMCID: PMC7357879 DOI: 10.1109/tro.2019.2946726] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
In this article, we present a novel stochastic algorithm called simultaneous sensor calibration and deformation estimation (SCADE) to address the problem of modeling deformation behavior of a generic continuum manipulator (CM) in free and obstructed environments. In SCADE, using a novel mathematical formulation, we introduce a priori model-independent filtering algorithm to fuse the continuous and inaccurate measurements of an embedded sensor (e.g., magnetic or piezoelectric sensors) with an intermittent but accurate data of an external imaging system (e.g., optical trackers or cameras). The main motivation of this article is the crucial need of obtaining an accurate shape/position estimation of a CM utilized in a surgical intervention. In these robotic procedures, the CM is typically equipped with an embedded sensing unit (ESU) while an external imaging modality (e.g., ultrasound or a fluoroscopy machine) is also available in the surgical site. The results of two different set of prior experiments in free and obstructed environments were used to evaluate the efficacy of SCADE algorithm. The experiments were performed with a CM specifically designed for orthopaedic interventions equipped with an inaccurate Fiber Bragg Grating (FBG) ESU and overhead camera. The results demonstrated the successful performance of the SCADE algorithm in simultaneous estimation of unknown deformation behavior of the utilized unmodeled CM together with realizing the time-varying drift of the poor-calibrated FBG sensing unit. Moreover, the results showed the phenomenal out-performance of the SCADE algorithm in estimation of the CM's tip position as compared to FBG-based position estimations.
Collapse
Affiliation(s)
- Farshid Alambeigi
- Walker Department of Mechanical Engineering, University of Texas at Austin, Austin, TX 78712 USA
| | - Sahba Aghajani Pedram
- Department of Mechanical and Aerospace Engineering, University of California-Los Angeles, Los Angeles, CA 90024 USA
| | - Jason L Speyer
- Department of Mechanical and Aerospace Engineering, University of California-Los Angeles, Los Angeles, CA 90024 USA
| | - Jacob Rosen
- Department of Mechanical and Aerospace Engineering, University of California-Los Angeles, Los Angeles, CA 90024 USA
| | - Iulian Iordachita
- Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA
| | - Russell H Taylor
- The Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA
| | - Mehran Armand
- The Laboratory for Computational Sensing and Robotics, Johns Hopkins University, Baltimore, MD 21218 USA
| |
Collapse
|