Visual Servoing of Wheeled Mobile Robots Without Desired Images

This paper proposes a novel monocular visual servoing strategy, which can drive a wheeled mobile robot to the desired pose without a prerecorded desired image. Compared with existing methods that adopt the teaching pattern for visual regulation, this scheme can still work well in the situation that the desired image has not been previously acquired. Thus, with the aid of this method, it is more convenient for mobile robots to execute visual servoing tasks. Specifically, to deal with nonexistence of the desired image, the reference frame is craftily defined by taking advantage of visual targets and the planar motion constraint, and the pose estimation algorithm is designed for the mobile robot with respect to the reference frame. Then, an adaptive visual regulation controller is developed to drive the mobile robot to the intermediate frame, where the parameter updating law is constructed for the unknown feature height based on the concurrent learning framework. Stability analysis shows that regulation errors and height identification error can converge simultaneously. Afterwards, the mobile robot is driven to the metric desired pose with the identified feature height. Both simulation and experimental results are provided to validate the performance of this strategy.

The Actuation Mechanism of 3D Printed Flexure-Based Robotic Microtweezers

We report on the design and the modeling of a three-dimensional (3D) printed flexure-based actuation mechanism for robotic microtweezers, the main body of which is a single piece of nylon. Our design aims to fill a void in sample manipulation between two classes of widely used instruments: nano-scale and macro-scale robotic manipulators. The key component is a uniquely designed cam flexure system, which linearly translates the bending of a piezoelectric bimorph actuator into angular displacement. The 3D printing made it possible to realize the fabrication of the cam with a specifically calculated curve, which would otherwise be costly using conventional milling techniques. We first characterized 3D printed nylon by studying sets of simple cantilevers, which provided fundamental characteristics that could be used for further designs. The finite element method analysis based on the obtained material data matched well with the experimental data. The tweezers showed angular displacement from 0° to 10° linearly to the deflection of the piezo actuator (0–1.74 mm) with the linearity error of 0.1°. Resonant frequency of the system with/without working tweezer tips was discovered as 101 Hz and 127 Hz, respectively. Our design provides simple and low-cost construction of a versatile manipulator system for samples in the micro/meso-scale (0.1–1 mm).

Achieving Automated Organelle Biopsy on Small Single Cells Using a Cell Surgery Robotic System

Single cell surgery such as manipulation or removal of subcellular components or/and organelles from single cells is increasingly used for the study of diseases and their causes in precision medicine. This paper presents a robotic surgery system to achieve automated organelle biopsy of single cells with dimensions of less than 20 μm in diameter. The complexity of spatial detection of the organelle position is reduced by patterning the cells using a microfluidic chip device. A sliding mode nonlinear controller is developed to enable extraction of organelles, such as the mitochondria and the nucleus, from single cells with high precision. An image processing algorithm is also developed to automatically detect the position of the desired organelle. The effectiveness of the proposed robotic surgery system is demonstrated experimentally with automated extraction of mitochondria and nucleus from human acute promyelocytic leukemia cells and human fibroblast cells. Extraction is followed by biological tests to indicate the functionality of biopsied mitochondria as well as the cell viability after removal of mitochondria. The results presented here have revealed that the proposed approach of automated organelle biopsy on single small cells is feasible.

Automatic Manipulation of Magnetically Actuated Helical Microswimmers in Static Environments

Electromagnetically actuated microswimmers have been widely used in various biomedical applications due to their minor invasive traits and their easy access to confined environments. In order to guide the microswimmers autonomously towards a target, an obstacle-free path must be computed using path planning algorithms, meanwhile a motion controller must be formulated. However, automatic manipulations of magnetically actuated microswimmers are underdeveloped and still are challenging topics. In this paper, we develop an automatic manipulation system for magnetically actuated helical microswimmers in static environments, which mainly consists of a mapper, a path planner, and a motion controller. First, the mapper processes the captured image by morphological transformations and then labels the free space and the obstacle space. Second, the path planner explores the obstacle-free space to find a feasible path from the start to the goal by a global planning algorithm. Last, the motion controller guides the helical microswimmers along the desired path by a closed-loop algorithm. Experiments are conducted to verify the effectiveness of the proposed automatic manipulation. Furthermore, our proposed approach presents the first step towards applications of microswimmers for targeted medical treatments, such as micromanipulation, targeted therapy, and targeted drug delivery.

Improving the throughput of automated holographic optical tweezers

The purpose of this work is to introduce three improvements to automated holographic-optical-tweezers systems that increase the number and speed of particles that can be manipulated simultaneously. First, we address path planning by solving a bottleneck assignment problem, which can reduce total move time by up to 30% when compared with traditional assignment problem solutions. Next, we demonstrate a new strategy to identify and remove undesired (e.g., misshapen or agglomerated) particles. Finally, we employ a controller that combines both closed- and open-loop automation steps, which can increase the overall loop rate and average particle speeds while also utilizing necessary process monitoring checks to ensure that particles reach their destinations. Using these improvements, we show fast reconfiguration of 100 microspheres simultaneously with a closed-loop control rate of 6, and 10 Hz by employing both closed- and open-loop steps. We also demonstrate the closed-loop assembly of a large pattern in a continuously flowing microchannel-based particle-delivery system. The proposed approach provides a promising path toward automatic and scalable assembly of microgranular structures.

Manipulation of Biological Cells Using a Robot-Aided Optical Tweezers System

This article reviews the autonomous manipulation strategies of biological cells utilizing optical tweezers, mainly including optical direct and indirect manipulation strategies. The typical and latest achievements in the optical manipulation of cells are presented, and the existing challenges for autonomous optical manipulation of biological cells are also introduced. Moreover, the integrations of optical tweezers with other manipulation tools are presented, which broadens the applications of optical tweezers in the biomedical manipulation areas and will also foster new developments in cell-based physiology and pathology studies, such as cell migration, single cell surgery, and preimplantation genetic diagnosis (PGD).

Out-of-Plane Rotation Control of Biological Cells With a Robot-Tweezers Manipulation System for Orientation-Based Cell Surgery

In many cell surgery applications, cell must be oriented properly such that the microsurgery tool can access the target components with minimum damage to the cell. In this paper, a scheme for out of image plane orientation control of suspended biological cells using robotic controlled optical tweezers is presented for orientation-based cell surgery. Based on our previous work on planar cell rotation using optical tweezers, the dynamic model of cell out-of-plane orientation control is formulated by using the T-matrix approach. Vision-based algorithms are developed to extract the cell out of image plane orientation angles, based on 2-D image slices obtained under an optical microscope. A robust feedback controller is then proposed to achieve cell out-of-plane rotation. Experiments of automated out of image plane rotational control for cell nucleus extraction surgery are performed to demonstrate the effectiveness of the proposed approach. This approach advances robot-aided single cell manipulation and produces impactful benefits to cell surgery applications such as nucleus transplantation and organelle biopsy in precision medicine.

Human experience–inspired path planning for robots

In this article, we present a human experience–inspired path planning algorithm for service robots. In addition to considering the path distance and smoothness, we emphasize the safety of robot navigation. Specifically, we build a speed field in accordance with several human driving experiences, like slowing down or detouring at a narrow aisle, and keeping a safe distance to the obstacles. Based on this speed field, the path curvatures, path distance, and steering speed are all integrated to form an energy function, which can be efficiently solved by the A* algorithm to seek the optimal path by resorting to an admissible heuristic function estimated from the energy function. Moreover, a simple yet effective fast path smoothing algorithm is proposed so as to ease the robots steering. Several examples are presented, demonstrating the effectiveness of our human experience–inspired path planning method.

Passivity-based control framework for task-space bilateral teleoperation with parametric uncertainty over unreliable networks

Bilateral teleoperation systems developed in joint-space or in task-space without taking into account parameter uncertainties and unreliable communication have limited practical applications. In order to ensure stability, improve tracking performance, and enhance applicability, a novel task-space control framework for bilateral teleoperation with kinematic/dynamic uncertainties and time delays/packet losses is studied. In this paper, we have demonstrated that with the proposed control algorithms, the teleoperation system is stable and position tracking is guaranteed when the system is subjected to parametric uncertainties and communication delays. With the transformation of scattering variables, a packet modulation, called Passivity-Based Packet Modulation (PBPM), is proposed to cope with data losses, incurred in transmission of data over unreliable network. Moreover, numerical simulations and experiments are also presented to validate the efficiency of the developed control framework for task-space bilateral teleoperation.

Autonomous exploration of mobile robots through deep neural networks

The exploration problem of mobile robots aims to allow mobile robots to explore an unknown environment. We describe an indoor exploration algorithm for mobile robots using a hierarchical structure that fuses several convolutional neural network layers with decision-making process. The whole system is trained end to end by taking only visual information (RGB-D information) as input and generates a sequence of main moving direction as output so that the robot achieves autonomous exploration ability. The robot is a TurtleBot with a Kinect mounted on it. The model is trained and tested in a real world environment. And the training data set is provided for download. The outputs of the test data are compared with the human decision. We use Gaussian process latent variable model to visualize the feature map of last convolutional layer, which proves the effectiveness of this deep convolution neural network mode. We also present a novel and lightweight deep-learning library libcnn especially for deep-learning processing of robotics tasks.

A Hierarchical Auction-Based Mechanism for Real-Time Resource Allocation in Cloud Robotic Systems

Cloud computing enables users to share computing resources on-demand. The cloud computing framework cannot be directly mapped to cloud robotic systems with ad hoc networks since cloud robotic systems have additional constraints such as limited bandwidth and dynamic structure. However, most multirobotic applications with cooperative control adopt this decentralized approach to avoid a single point of failure. Robots need to continuously update intensive data to execute tasks in a coordinated manner, which implies real-time requirements. Thus, a resource allocation strategy is required, especially in such resource-constrained environments. This paper proposes a hierarchical auction-based mechanism, namely link quality matrix (LQM) auction, which is suitable for ad hoc networks by introducing a link quality indicator. The proposed algorithm produces a fast and robust method that is accurate and scalable. It reduces both global communication and unnecessary repeated computation. The proposed method is designed for firm real-time resource retrieval for physical multirobot systems. A joint surveillance scenario empirically validates the proposed mechanism by assessing several practical metrics. The results show that the proposed LQM auction outperforms state-of-the-art algorithms for resource allocation.

Microfluidics cell sample preparation for analysis: Advances in efficient cell enrichment and precise single cell capture

Single cell analysis has received increasing attention recently in both academia and clinics, and there is an urgent need for effective upstream cell sample preparation. Two extremely challenging tasks in cell sample preparation-high-efficiency cell enrichment and precise single cell capture-have now entered into an era full of exciting technological advances, which are mostly enabled by microfluidics. In this review, we summarize the category of technologies that provide new solutions and creative insights into the two tasks of cell manipulation, with a focus on the latest development in the recent five years by highlighting the representative works. By doing so, we aim both to outline the framework and to showcase example applications of each task. In most cases for cell enrichment, we take circulating tumor cells (CTCs) as the target cells because of their research and clinical importance in cancer. For single cell capture, we review related technologies for many kinds of target cells because the technologies are supposed to be more universal to all cells rather than CTCs. Most of the mentioned technologies can be used for both cell enrichment and precise single cell capture. Each technology has its own advantages and specific challenges, which provide opportunities for researchers in their own area. Overall, these technologies have shown great promise and now evolve into real clinical applications.

Real-time object tracking system based on field-programmable gate array and convolution neural network

Vision-based object tracking has lots of applications in robotics, like surveillance, navigation, motion capturing, and so on. However, the existing object tracking systems still suffer from the challenging problem of high computation consumption in the image processing algorithms. The problem can prevent current systems from being used in many robotic applications which have limitations of payload and power, for example, micro air vehicles. In these applications, the central processing unit- or graphics processing unit-based computers are not good choices due to the high weight and power consumption. To address the problem, this article proposed a real-time object tracking system based on field-programmable gate array, convolution neural network, and visual servo technology. The time-consuming image processing algorithms, such as distortion correction, color space convertor, and Sobel edge, Harris corner features detector, and convolution neural network were redesigned using the programmable gates in field-programmable gate array. Based on the field-programmable gate array-based image processing, an image-based visual servo controller was designed to drive a two degree of freedom manipulator to track the target in real time. Finally, experiments on the proposed system were performed to illustrate the effectiveness of the real-time object tracking system.

Unified Generic Geometric-Decompositions for Consensus or Flocking Systems of Cooperative Agents and Fast Recalculations of Decomposed Subsystems Under Topology-Adjustments

This paper considers a unified geometric projection approach for: 1) decomposing a general system of cooperative agents coupled via Laplacian matrices or stochastic matrices and 2) deriving a centroid-subsystem and many shape-subsystems, where each shape-subsystem has the distinct properties (e.g., preservation of formation and stability of the original system, sufficiently simple structures and explicit formation evolution of agents, and decoupling from the centroid-subsystem) which will facilitate subsequent analyses. Particularly, this paper provides an additional merit of the approach: considering adjustments of coupling topologies of agents which frequently occur in system design (e.g., to add or remove an edge, to move an edge to a new place, and to change the weight of an edge), the corresponding new shape-subsystems can be derived by a few simple computations merely from the old shape-subsystems and without referring to the original system, which will provide further convenience for analysis and flexibility of choice. Finally, such fast recalculations of new subsystems under topology adjustments are provided with examples.

Robotic Online Path Planning on Point Cloud

This paper deals with the path-planning problem for mobile wheeled- or tracked-robot which drive in 2.5-D environments, where the traversable surface is usually considered as a 2-D-manifold embedded in a 3-D ambient space. Specially, we aim at solving the 2.5-D navigation problem using raw point cloud as input. The proposed method is independent of traditional surface parametrization or reconstruction methods, such as a meshing process, which generally has high-computational complexity. Instead, we utilize the output of 3-D tensor voting framework on the raw point clouds. The computation of tensor voting is accelerated by optimized implementation on graphics computation unit. Based on the tensor voting results, a novel local Riemannian metric is defined using the saliency components, which helps the modeling of the latent traversable surface. Using the proposed metric, we prove that the geodesic in the 3-D tensor space leads to rational path-planning results by experiments. Compared to traditional methods, the results reveal the advantages of the proposed method in terms of smoothing the robot maneuver while considering the minimum travel distance.

Direct measurement of cell protrusion force utilizing a robot-aided cell manipulation system with optical tweezers for cell migration control

Cell migration refers to the directional cell movement in response to a chemoattractant gradient, a key process that occurs in a wide variety of biological phenomena. Cell protrusion force is generated by the actin polymerization of a cell, which drives the cell to move toward the stimulus as induced by the chemoattractant gradient. This paper presents a new methodology for the direct measurement of cell protrusion force utilizing a robot-aided optical tweezer system. The functionalized beads that are robotically trapped and placed near the cell serve as both cell migration stimulators and protrusion force probes. The force generated by the actin polymerization of the cell propels the bead to move away from the trapping center when the cell comes in contact with the bead. Such a deviation can be determined and used to calculate the trapping force, which is equal to the protrusion force at a balanced position. With the quantitative measurement of the protrusion, we find that the protrusion force of a live cell in response to a chemoattractant within the range of hundreds of piconewtons. We further probe the protrusion force distribution at the cell leading edge and find that the highest protrusion force appears at the cell migration direction. These measurements can help us characterize the mechanism of cell migration and lay a solid foundation for further proactive control of cell movement.

Towards Mobile Microrobot Swarms for Additive Micromanufacturing

In this paper, a novel approach to achieving the independent control of multiple magnetic microrobots is presented. The approach utilizes a specialized substrate consisting of a fine grid of planar, MEMS-fabricated micro coils of the same size as the microrobots (≤ 500 μm). The coils can be used to generate real magnetic potentials and, therefore, attractive and repulsive forces in the workspace to control the trajectories of the microrobots. Initial work on modelling the coil and microrobot behavior is reported along with simulation results for navigating one and two microrobots along independent desired trajectories. Qualitative results from a scaled-up printed circuit board version of the specialized substrate operating on permanent magnets are presented and offer proof-of-concept results for the approach. These tests also provide insights for practical implementations of such a system, which are similarly reported. The ultimate goal of this work is to use swarms of independently controlled microrobots in advanced, additive manufacturing applications.