Kinect-based system for automated control of terrestrial insect biobots

A Navigation Algorithm to Enable Sustainable Control of Insect-Computer Hybrid Robot with Stimulus Signal Regulator and Habituation-Breaking Function

The insect-computer hybrid soft robots are receiving increasing attention due to their excellent motor capabilities, small size, and low power consumption. However, the effective control of insects is limited to minutes since the response from insects is reduced as the number of stimulus signal increase. This phenomenon is known as habituation, which causes the loss of control of robots and hinders their application in practical tasks such as search and rescue missions that require several hours. It has been shown that constantly switching the pattern of stimulus signals can slow down the onset of habituation. Moreover, when habituation occurs, applying a different stimulus signal can break the habituation. Based on this, we have designed a navigation algorithm that can extend the control time of insects to several hours. The algorithm is composed of a stimulation decision-making core responsible for deciding on the type of stimulus signal (left, right, acceleration), a stimulation parameters adjustment (SPA) core responsible for adjusting the stimulus signal voltage constantly to delay the occurrence of habituation, and a reactivation function (RF), as a different stimulus signal from the normal stimulus signal, is used to break insects' habituation to the normal stimulus signal. Experiments have shown that our SPA regulator and RF can significantly extend the control time of insects. Navigation experiments demonstrating effective control of the insects for up to 3 h verified the effectiveness of the navigation algorithm, which strikes a balance between control accuracy and control time.

Feedback control of automatic navigation for cyborg cockroach without external motion capture system

Due to their size and locomotion ability, cockroaches are favorable as hybrid robot platforms in search and rescue (SAR) missions. However, cockroaches most likely approach the corner area and stay for an uncertain time. This natural behavior will hinder the utilization of cyborg cockroaches in SAR missions under rubble, unstructured, and unknown areas. Therefore, we proposed onboard automatic obstacle avoidance and human detection that can run on the wireless backpack stimulator without an external motion capture system. A low-power and small-size Time of Flight (ToF) sensor was selected as a distance measurement sensor, while a low-resolution thermopile array sensor was applied for human presence detection. The implemented feedback control based on IMU and ToF sensors has successfully navigated the cyborg cockroach to avoid obstacles and escape from the sharp corners in the laboratory unstructured area without stopping or being trapped. It could also recognize the human presence when the human was in front of it in real-time. Due to its performance, the random forest classifier was implemented as an embedded human detection system. It could achieve the highest accuracy at a distance of around 25 cm (92.5%) and the lowest accuracy at about 100 cm (70%).

The Autonomous Pipeline Navigation of a Cockroach Bio-Robot with Enhanced Walking Stimuli

Tens of crawling bio-robots with cockroaches as the mobile platform have been developed with various functions. Compared with artificial crawling robots of the same size, they revealed better flexibility, larger payload, and stronger endurance. These features made bio-robots ideal for pipeline inspection scenarios because the advancements in locomotion mechanisms and efficient power systems are still hurdles for current artificial systems. In this study, we controlled the bio-robot to crawl in the confined dark pipeline and achieved autonomous motion control with the help of an onboard sensing system. Specifically, a micro-camera was mounted on the electronic backpack of the cockroach for image collection, and an IMU sensor was used to compute its body orientation. The electronic backpack transmitted images to the host computer for junction recognition and distance estimation. Meanwhile, the insect's habituation to electrical stimulation has long been an uncertain factor in the control of bio-robots. Here, a synergistic stimulation strategy was proposed to markedly reduce the habituation and increase the number of effective turning controls to over 100 times. It is also found that both the increase of payload and the application of stimulations could promote the metabolic rate by monitoring carbon dioxide release. With the integration of synergistic stimulation and autonomous control, we demonstrated the fully autonomous pipeline navigation with our cockroach bio-robot, which realized the cycle number of approximately 10 in a roll. This research provides a novel technology that has the potential for practical applications in the future.

The long-acting walking control of a cockroach bio-bot for vision-aided pipeline navigation

Many small bionic crawling robots have been invented for search and rescue missions in narrow spaces. However, their locomotion capability is far from that of insects of the same size. Transforming a cockroach into a bio-bot has been a hot topic in the past decade. Herein, we modified this insect to perform surveillance work in dark confined environments. The synergistic electrical stimulation for turning control was proposed by alternating electrical stimulation of the cerci and antennae every 5 trials. The result showed that this method was able to control cockroaches turning steadily 117 times. An electronic backpack was designed, which was capable of transmitting images in real time, and a light emitting diode (LED) was installed on the backpack providing a light source for the camera. Thus, a vision-aided navigation system was formed for dark confined environments, e.g. pipelines. With a host computer software, the operator controlled the bio-bot to pass through a completely dark and closed pipeline. The electronic backpack and the host computer were connected via transmission control protocol (TCP), which allows the operator to manipulate the bio-robot remotely. This technology can be applied in pipeline surveillance in the future.

Rattractor-Instant guidance of a rat into a virtual cage using the deep brain stimulation

We developed "Rattractor" (rat attractor), a system to apply electrical stimuli to the deep brain of a rat as it stays in a specified region or a virtual cage to demonstrate an instant electrophysiological feedback guidance for animals. Two wire electrodes were implanted in the brains of nine rats. The electrodes targeted the medial forebrain bundle (MFB), which is a part of the reward system in the deep brain. Following the recovery period, the rats were placed in a plain field where they could move freely, but wired to a stimulation circuit. An image sensor installed over the field detected the subject's position, which triggered the stimulator such that the rat remained within the virtual cage. We conducted a behavioral experiment to evaluate the sojourn ratio of rats residing in the region. Thereafter, a histological analysis of the rat brain was performed to confirm the position of the stimulation sites in the brain. Seven rats survived the surgery and the recovery period without technical failures such as connector breaks. We observed that three of them tended to stay in the virtual cage during stimulation, and this effect was maintained for two weeks. Histological analysis revealed that the electrode tips were correctly placed in the MFB region of the rats. The other four subjects showed no apparent preference for the virtual cage. In these rats, we did not find electrode tips in the MFB, or could not determine their positions. Almost half of the rats tended to remain inside the virtual cage when position-related reward stimuli were triggered in the MFB region. Notably, our system did not require previous training or sequential interventions to affect the behavioral preferences of subjects. This process is similar to the situation in which sheep are chased by a shepherd dog in the desired direction.

CameraRoach: A WiFi- and Camera-Enabled Cyborg Cockroach for Search and Rescue

We describe here our design and implementation of a cyborg insect, called CameraRoach, with onboard camera feedback that can be navigated via remote control providing a first-person view. The camera pack is mounted on the Madagascar hissing cockroach, which is small enough to fit into crevices but also can carry a printed circuit boards with power, communication, and sensor components (visual camera). For navigating the cockroach, we implemented a unique electronic backpack neural stimulator, which allows the cockroach to be maneuvered on a desired path with a joystick. A high-resolution wireless camera, also included in the backpack, sends live images via a WiFi (Wireless Fidelity) network. We present the results of an evaluation experiment with the CameraRoach and compare it with the other state of the art systems like the Beetle-Cam.

ISOFLURANE ANESTHESIA IN THE MADAGASCAR HISSING COCKROACH (GROMPHADORHINA PORTENTOSA)

This study investigated the efficacy of isoflurane in Madagascar hissing cockroaches (Gromphadorhina portentosa). Cockroaches (n = 12) were placed in a 2-L anesthesia chamber with isoflurane initiated at 5% at 2 L/m. Movement of antennae, mandibles, and legs and righting reflex were assessed, with full induction achieved when all responses had ceased for 2 m. Cockroach movement became ataxic on average (±SD) 3.5 ± 0.9 m after isoflurane treatment, and induction occurred on average 18.7 ± 4.4 m after treatment. Loss of righting reflex was the most consistent indication of full induction. Cockroaches fully recovered on average 16.2 ± 5.6 m after removal of isoflurane inhalant. Induction was uneventful, and no mortality or obvious morbidity was observed in treated cockroaches up to 30 d posttreatment. The authors conclude that the use of inhalant isoflurane is both safe and effective for anesthesia of Madagascar hissing cockroaches.

An automated behavior analysis system for freely moving rodents using depth image

A rodent behavior analysis system is presented, capable of automated tracking, pose estimation, and recognition of nine behaviors in freely moving animals. The system tracks three key points on the rodent body (nose, center of body, and base of tail) to estimate its pose and head rotation angle in real time. A support vector machine (SVM)-based model, including label optimization steps, is trained to classify on a frame-by-frame basis: resting, walking, bending, grooming, sniffing, rearing supported, rearing unsupported, micro-movements, and "other" behaviors. Compared to conventional red-green-blue (RGB) camera-based methods, the proposed system operates on 3D depth images provided by the Kinect infrared (IR) camera, enabling stable performance regardless of lighting conditions and animal color contrast with the background. This is particularly beneficial for monitoring nocturnal animals' behavior. 3D features are designed to be extracted directly from the depth stream and combined with contour-based 2D features to further improve recognition accuracies. The system is validated on three freely behaving rats for 168 min in total. The behavior recognition model achieved a cross-validation accuracy of 86.8% on the rat used for training and accuracies of 82.1 and 83% on the other two "testing" rats. The automated head angle estimation aided by behavior recognition resulted in 0.76 correlation with human expert annotation. Graphical abstract Top view of a rat freely behaving in a standard homecage, captured by Kinect-v2 sensors. The depth image is used for constructing a 3D topography of the animal for pose estimation, behavior recognition, and head angle calculation. Results of the processed data are displayed on the user interface in various forms.

A smart homecage system with 3D tracking for long-term behavioral experiments

A wirelessly-powered homecage system, called the EnerCage-HC, that is equipped with multi-coil wireless power transfer, closed-loop power control, optical behavioral tracking, and a graphic user interface (GUI) is presented for long-term electrophysiology experiments. The EnerCage-HC system can wirelessly power a mobile unit attached to a small animal subject and also track its behavior in real-time as it is housed inside a standard homecage. The EnerCage-HC system is equipped with one central and four overlapping slanted wire-wound coils (WWCs) with optimal geometries to form 3-and 4-coil power transmission links while operating at 13.56 MHz. Utilizing multi-coil links increases the power transfer efficiency (PTE) compared to conventional 2-coil links and also reduces the number of power amplifiers (PAs) to only one, which significantly reduces the system complexity, cost, and dissipated heat. A Microsoft Kinect installed 90 cm above the homecage localizes the animal position and orientation with 1.6 cm accuracy. An in vivo experiment was conducted on a freely behaving rat by continuously delivering 24 mW to the mobile unit for > 7 hours inside a standard homecage.

Insect-computer hybrid legged robot with user-adjustable speed, step length and walking gait

We have constructed an insect-computer hybrid legged robot using a living beetle (Mecynorrhina torquata; Coleoptera). The protraction/retraction and levation/depression motions in both forelegs of the beetle were elicited by electrically stimulating eight corresponding leg muscles via eight pairs of implanted electrodes. To perform a defined walking gait (e.g., gallop), different muscles were individually stimulated in a predefined sequence using a microcontroller. Different walking gaits were performed by reordering the applied stimulation signals (i.e., applying different sequences). By varying the duration of the stimulation sequences, we successfully controlled the step frequency and hence the beetle's walking speed. To the best of our knowledge, this paper presents the first demonstration of living insect locomotion control with a user-adjustable walking gait, step length and walking speed.

A Smart Wirelessly Powered Homecage for Long-Term High-Throughput Behavioral Experiments

A wirelessly powered homecage system, called the EnerCage-HC, that is equipped with multicoil wireless power transfer, closed-loop power control, optical behavioral tracking, and a graphic user interface is presented for longitudinal electrophysiology and behavioral neuroscience experiments. The EnerCage-HC system can wirelessly power a mobile unit attached to a small animal subject and also track its behavior in real-time as it is housed inside a standard homecage. The EnerCage-HC system is equipped with one central and four overlapping slanted wire-wound coils with optimal geometries to form three- and four-coil power transmission links while operating at 13.56 MHz. Utilizing multicoil links increases the power transfer efficiency (PTE) compared with conventional two-coil links and also reduces the number of power amplifiers to only one, which significantly reduces the system complexity, cost, and heat dissipation. A Microsoft Kinect installed 90 cm above the homecage localizes the animal position and orientation with 1.6-cm accuracy. Moreover, a power management ASIC, including a high efficiency active rectifier and automatic coil resonance tuning, was fabricated in a 0.35-μm 4M2P standard CMOS process for the mobile unit. The EnerCage-HC achieves a max/min PTE of 36.3%/16.1% at the nominal height of 7 cm. In vivo experiments were conducted on freely behaving rats by continuously delivering 24 mW to the mobile unit for >7 h inside a standard homecage.

Effective Stimulus Parameters for Directed Locomotion in Madagascar Hissing Cockroach Biobot

Swarms of insects instrumented with wireless electronic backpacks have previously been proposed for potential use in search and rescue operations. Before deploying such biobot swarms, an effective long-term neural-electric stimulus interface must be established, and the locomotion response to various stimuli quantified. To this end, we studied a variety of pulse types (mono- vs. bipolar; voltage- vs. current-controlled) and shapes (amplitude, frequency, duration) to parameters that are most effective for evoking locomotion along a desired path in the Madagascar hissing cockroach (G. portentosa) in response to antennal and cercal stimulation. We identified bipolar, 2 V, 50 Hz, 0.5 s voltage controlled pulses as being optimal for evoking forward motion and turns in the expected contraversive direction without habituation in ≈50% of test subjects, a substantial increase over ≈10% success rates previously reported. Larger amplitudes for voltage (1–4 V) and current (50–150 μA) pulses generally evoked larger forward walking (15.6–25.6 cm; 3.9–5.6 cm/s) but smaller concomitant turning responses (149 to 80.0 deg; 62.8 to 41.2 deg/s). Thus, the radius of curvature of the initial turn-then-run locomotor response (≈10–25 cm) could be controlled in a graded manner by varying the stimulus amplitude. These findings could be used to help optimize stimulus protocols for swarms of cockroach biobots navigating unknown terrain.

Towards fenceless boundaries for solar powered insect biobots

Demonstration of remote navigation with instrumented insects, such as the Madagascar Hissing Cockroach, Gromphadorhina portentosa, has enabled the concept of biobotic agents for search and rescue missions and environmental monitoring applications. The biobots can form the nodes of a mobile sensor network to be established, for example, in unknown and dynamic environments after natural disasters to pinpoint surviving victims. We demonstrate here, for the first time, the concept of an invisible fence for insect biobots with an ultimate goal of keeping insect biobots within a certain distance of each other or a base station to ensure a reliable wireless network. For extended mission durations, this fenceless boundary would also be used to guide insects towards light sources for autonomous solar charging of their on-board batteries.