Magnetic Motion Control and Planning of Untethered Soft Grippers using Ultrasound Image Feedback

Advancements in Soft Robotics: A Comprehensive Review on Actuation Methods, Materials, and Applications

The flexibility and adaptability of soft robots enable them to perform various tasks in changing environments, such as flower picking, fruit harvesting, in vivo targeted treatment, and information feedback. However, these fulfilled functions are discrepant, based on the varied working environments, driving methods, and materials. To further understand the working principle and research emphasis of soft robots, this paper summarized the current research status of soft robots from the aspects of actuating methods (e.g., humidity, temperature, PH, electricity, pressure, magnetic field, light, biological, and hybrid drive), materials (like hydrogels, shape-memory materials, and other flexible materials) and application areas (camouflage, medical devices, electrical equipment, and grippers, etc.). Finally, we provided some opinions on the technical difficulties and challenges of soft robots to comprehensively comprehend soft robots, lucubrate their applications, and improve the quality of our lives.

High-resolution and high-speed 3D tracking of microrobots using a fluorescent light field microscope

Background

Imaging and tracking are crucial for microrobots which navigate through complex 3D environments. Fluorescent imaging (FI) by microscope offers a high-resolution and high-sensitive imaging method to study the property of microrobots. However, conventional microscope suffers from shallow depth of field (DOF) and lacks 3D imaging capability.

Methods

We proposed a high-resolution and high-speed 3D tracking method for microrobots based on a fluorescent light field microscope (FLFM). We designed the FLFM system according to the size of a representative helical microrobot (150 μm body length, 50 μm screw diameter), and studied the system's performance. We also proposed a 3D tracking algorithm for microrobots using digital refocusing.

Results

We validated the method by simulations and built an FLFM system to perform the tracking experiments of microrobots with representative size. Our 3D tracking method achieves a 30 fps data acquisition rate, 10 μm lateral resolution and approximately 40 μm axial resolution over a volume of 1,200×1,200×326 μm³. Results indicate that the accuracy of the method can reach about 9 μm.

Conclusions

Compared with the FI by a conventional microscope, the FLFM-based method gains wider DOF and 3D imaging capability with a single-shot image. The tracking method succeeds in providing the trajectory of the microrobot with a good lateral resolution.

Capsule robot pose and mechanism state detection in ultrasound using attention-based hierarchical deep learning

Ingestible robotic capsules with locomotion capabilities and on-board sampling mechanism have great potential for non-invasive diagnostic and interventional use in the gastrointestinal tract. Real-time tracking of capsule location and operational state is necessary for clinical application, yet remains a significant challenge. To this end, we propose an approach that can simultaneously determine the mechanism state and in-plane 2D pose of millimeter capsule robots in an anatomically representative environment using ultrasound imaging. Our work proposes an attention-based hierarchical deep learning approach and adapts the success of transfer learning towards solving the multi-task tracking problem with limited dataset. To train the neural networks, we generate a representative dataset of a robotic capsule within ex-vivo porcine stomachs. Experimental results show that the accuracy of capsule state classification is 97%, and the mean estimation errors for orientation and centroid position are 2.0 degrees and 0.24 mm (1.7% of the capsule's body length) on the hold-out test set. Accurate detection of the capsule while manipulated by an external magnet in a porcine stomach and colon is also demonstrated. The results suggest our proposed method has the potential for advancing the wireless capsule-based technologies by providing accurate detection of capsule robots in clinical scenarios.

Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning

Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time varying magnetic field generated from a three-axis array of electromagnets. The reinforcement learning agent learned successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulated the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.

Visualization of micro-agents and surroundings by real-time multicolor fluorescence microscopy

Optical microscopy techniques are a popular choice for visualizing micro-agents. They generate images with relatively high spatiotemporal resolution but do not reveal encoded information for distinguishing micro-agents and surroundings. This study presents multicolor fluorescence microscopy for rendering color-coded identification of mobile micro-agents and dynamic surroundings by spectral unmixing. We report multicolor microscopy performance by visualizing the attachment of single and cluster micro-agents to cancer spheroids formed with HeLa cells as a proof-of-concept for targeted drug delivery demonstration. A microfluidic chip is developed to immobilize a single spheroid for the attachment, provide a stable environment for multicolor microscopy, and create a 3D tumor model. In order to confirm that multicolor microscopy is able to visualize micro-agents in vascularized environments, in vitro vasculature network formed with endothelial cells and ex ovo chicken chorioallantoic membrane are employed as experimental models. Full visualization of our models is achieved by sequential excitation of the fluorophores in a round-robin manner and synchronous individual image acquisition from three-different spectrum bands. We experimentally demonstrate that multicolor microscopy spectrally decomposes micro-agents, organic bodies (cancer spheroids and vasculatures), and surrounding media utilizing fluorophores with well-separated spectrum characteristics and allows image acquisition with 1280 [Formula: see text] 1024 pixels up to 15 frames per second. Our results display that real-time multicolor microscopy provides increased understanding by color-coded visualization regarding the tracking of micro-agents, morphology of organic bodies, and clear distinction of surrounding media.

Endoscopy-assisted magnetic navigation of biohybrid soft microrobots with rapid endoluminal delivery and imaging

High-precision delivery of microrobots at the whole-body scale is of considerable importance for efforts toward targeted therapeutic intervention. However, vision-based control of microrobots, to deep and narrow spaces inside the body, remains a challenge. Here, we report a soft and resilient magnetic cell microrobot with high biocompatibility that can interface with the human body and adapt to the complex surroundings while navigating inside the body. We achieve time-efficient delivery of soft microrobots using an integrated platform called endoscopy-assisted magnetic actuation with dual imaging system (EMADIS). EMADIS enables rapid deployment across multiple organ/tissue barriers at the whole-body scale and high-precision delivery of soft and biohybrid microrobots in real time to tiny regions with depth up to meter scale through natural orifice, which are commonly inaccessible and even invisible by conventional endoscope and medical robots. The precise delivery of magnetic stem cell spheroid microrobots (MSCSMs) by the EMADIS transesophageal into the bile duct with a total distance of about 100 centimeters can be completed within 8 minutes. The integration strategy offers a full clinical imaging technique-based therapeutic/intervention system, which broadens the accessibility of hitherto hard-to-access regions, by means of soft microrobots.

Trends in Micro-/Nanorobotics: Materials Development, Actuation, Localization, and System Integration for Biomedical Applications

Micro-/nanorobots (m-bots) have attracted significant interest due to their suitability for applications in biomedical engineering and environmental remediation. Particularly, their applications in in vivo diagnosis and intervention have been the focus of extensive research in recent years with various clinical imaging techniques being applied for localization and tracking. The successful integration of well-designed m-bots with surface functionalization, remote actuation systems, and imaging techniques becomes the crucial step toward biomedical applications, especially for the in vivo uses. This review thus addresses four different aspects of biomedical m-bots: design/fabrication, functionalization, actuation, and localization. The biomedical applications of the m-bots in diagnosis, sensing, microsurgery, targeted drug/cell delivery, thrombus ablation, and wound healing are reviewed from these viewpoints. The developed biomedical m-bot systems are comprehensively compared and evaluated based on their characteristics. The current challenges and the directions of future research in this field are summarized.

Real-Time Magnetic Navigation of a Rotating Colloidal Microswarm Under Ultrasound Guidance

OBJECTIVE

Untethered microrobots hold great promise for applications in biomedical field including targeted delivery, biosensing, and microsurgery. A major challenge of using microrobots to perform in vivo tasks is the real-time localization and motion control using medical imaging technologies. Here we report real-time magnetic navigation of a paramagnetic nanoparticle-based microswarm under ultrasound guidance.

METHODS

A three-axis Helmholtz electromagnetic coil system integrated with an ultrasound imaging system is developed for generation, actuation, and closed-loop control of the microswarm. The magnetite nanoparticle-based microswarm is generated and navigated using rotating magnetic fields. In order to localize the microswarm in real time, the dynamic imaging contrast has been analyzed and exploited in image process to increase the signal-to-noise ratio. Moreover, imaging of the microswarm at different depths are experimentally studied and analyzed, and the minimal dose of nanoparticles for localizing a microswarm at different depths is ex vivo investigated. For real-time navigating the microswarm in a confined environment, a PI control scheme is designed.

RESULTS

Image differencing-based processing increases the signal-to-noise ratio, and the microswarm can be ex vivo localized at depth of 2.2-7.8 cm. Experimental results show that the microswarm is able to be real-time navigated along a planned path in a channel, and the average steady-state error is 0.27 mm (  ∼ 33.7% of the body length).

SIGNIFICANCE

The colloidal microswarm is real-time localized and navigated using ultrasound feedback, which shows great potential for biomedical applications that require real-time noninvasive tracking.

Ultrasound-Guided Wireless Tubular Robotic Anchoring System

Untethered miniature robots have significant poten-tial and promise in diverse minimally invasive medical applications inside the human body. For drug delivery and physical contra-ception applications inside tubular structures, it is desirable to have a miniature anchoring robot with self-locking mechanism at a target tubular region. Moreover, the behavior of this robot should be tracked and feedback-controlled by a medical imaging-based system. While such a system is unavailable, we report a reversible untethered anchoring robot design based on remote magnetic actuation. The current robot prototype's dimension is 7.5 mm in diameter, 17.8 mm in length, and made of soft polyurethane elastomer, photopolymer, and two tiny permanent magnets. Its relaxation and anchoring states can be maintained in a stable manner without supplying any control and actuation input. To control the robot's locomotion, we implement a two-dimensional (2D) ultrasound imaging-based tracking and control system, which automatically sweeps locally and updates the robot's position. With such a system, we demonstrate that the robot can be controlled to follow a pre-defined 1D path with the maximal position error of 0.53 ± 0.05 mm inside a tubular phantom, where the reversible anchoring could be achieved under the monitoring of ultrasound imaging.

Stimuli-Responsive Soft Untethered Grippers for Drug Delivery and Robotic Surgery

Untethered microtools that can be precisely navigated into deep in vivo locations are important for clinical procedures pertinent to minimally invasive surgery and targeted drug delivery. In this mini-review, untethered soft grippers are discussed, with an emphasis on a class of autonomous stimuli-responsive gripping soft tools that can be used to excise tissues and release drugs in a controlled manner. The grippers are composed of polymers and hydrogels and are thus compliant to soft tissues. They can be navigated using magnetic fields and controlled by robotic path-planning strategies to carry out tasks like pick-and-place of microspheres and biological materials either with user assistance, or in a fully autonomous manner. It is envisioned that the use of these untethered soft grippers will translate from laboratory experiments to clinical scenarios and the challenges that need to be overcome to make this transition are discussed.