Body modeling and model-based tracking for neuroethology

Automated Reconstruction of Three-Dimensional Fish Motion, Forces, and Torques

Fish can move freely through the water column and make complex three-dimensional motions to explore their environment, escape or feed. Nevertheless, the majority of swimming studies is currently limited to two-dimensional analyses. Accurate experimental quantification of changes in body shape, position and orientation (swimming kinematics) in three dimensions is therefore essential to advance biomechanical research of fish swimming. Here, we present a validated method that automatically tracks a swimming fish in three dimensions from multi-camera high-speed video. We use an optimisation procedure to fit a parameterised, morphology-based fish model to each set of video images. This results in a time sequence of position, orientation and body curvature. We post-process this data to derive additional kinematic parameters (e.g. velocities, accelerations) and propose an inverse-dynamics method to compute the resultant forces and torques during swimming. The presented method for quantifying 3D fish motion paves the way for future analyses of swimming biomechanics.

Long-term behavioral tracking of freely swimming weakly electric fish

Long-term behavioral tracking can capture and quantify natural animal behaviors, including those occurring infrequently. Behaviors such as exploration and social interactions can be best studied by observing unrestrained, freely behaving animals. Weakly electric fish (WEF) display readily observable exploratory and social behaviors by emitting electric organ discharge (EOD). Here, we describe three effective techniques to synchronously measure the EOD, body position, and posture of a free-swimming WEF for an extended period of time. First, we describe the construction of an experimental tank inside of an isolation chamber designed to block external sources of sensory stimuli such as light, sound, and vibration. The aquarium was partitioned to accommodate four test specimens, and automated gates remotely control the animals' access to the central arena. Second, we describe a precise and reliable real-time EOD timing measurement method from freely swimming WEF. Signal distortions caused by the animal's body movements are corrected by spatial averaging and temporal processing stages. Third, we describe an underwater near-infrared imaging setup to observe unperturbed nocturnal animal behaviors. Infrared light pulses were used to synchronize the timing between the video and the physiological signal over a long recording duration. Our automated tracking software measures the animal's body position and posture reliably in an aquatic scene. In combination, these techniques enable long term observation of spontaneous behavior of freely swimming weakly electric fish in a reliable and precise manner. We believe our method can be similarly applied to the study of other aquatic animals by relating their physiological signals with exploratory or social behaviors.

Real-Time Localization of Moving Dipole Sources for Tracking Multiple Free-Swimming Weakly Electric Fish

In order to survive, animals must quickly and accurately locate prey, predators, and conspecifics using the signals they generate. The signal source location can be estimated using multiple detectors and the inverse relationship between the received signal intensity (RSI) and the distance, but difficulty of the source localization increases if there is an additional dependence on the orientation of a signal source. In such cases, the signal source could be approximated as an ideal dipole for simplification. Based on a theoretical model, the RSI can be directly predicted from a known dipole location; but estimating a dipole location from RSIs has no direct analytical solution. Here, we propose an efficient solution to the dipole localization problem by using a lookup table (LUT) to store RSIs predicted by our theoretically derived dipole model at many possible dipole positions and orientations. For a given set of RSIs measured at multiple detectors, our algorithm found a dipole location having the closest matching normalized RSIs from the LUT, and further refined the location at higher resolution. Studying the natural behavior of weakly electric fish (WEF) requires efficiently computing their location and the temporal pattern of their electric signals over extended periods. Our dipole localization method was successfully applied to track single or multiple freely swimming WEF in shallow water in real-time, as each fish could be closely approximated by an ideal current dipole in two dimensions. Our optimized search algorithm found the animal’s positions, orientations, and tail-bending angles quickly and accurately under various conditions, without the need for calibrating individual-specific parameters. Our dipole localization method is directly applicable to studying the role of active sensing during spatial navigation, or social interactions between multiple WEF. Furthermore, our method could be extended to other application areas involving dipole source localization.

Undulating fins produce off-axis thrust and flow structures

While wake structures of many forms of swimming and flying are well characterized, the wake generated by a freely-swimming undulating fin has not yet been analyzed. These elongated fins allow fish to achieve enhanced agility exemplified by the forward, backward, and vertical swimming capabilities of knifefish and also have potential applications in the design of more maneuverable underwater vehicles. We present the flow structure of an undulating robotic fin model using particle image velocimetry to measure fluid velocity fields in the wake. We supplement the experimental robotic work with high-fidelity computational fluid dynamics, simulating the hydrodynamics of both a virtual fish whose fin kinematics and fin plus body morphology are measured from a freely-swimming knifefish as well as a virtual rendering of our robot. Our results indicate a series of linked vortex tubes is shed off the long edge of the fin as the undulatory wave travels lengthwise along the fin. A jet at an oblique angle to the fin is associated with the successive vortex tubes, propelling the fish forward. The vortex structure bears similarity to the linked vortex ring structure trailing the oscillating caudal fin of a carangiform swimmer, though the vortex rings are distorted due to the undulatory kinematics of the elongated fin.

Three-dimensional neurophenotyping of adult zebrafish behavior

The use of adult zebrafish (Danio rerio) in neurobehavioral research is rapidly expanding. The present large-scale study applied the newest video-tracking and data-mining technologies to further examine zebrafish anxiety-like phenotypes. Here, we generated temporal and spatial three-dimensional (3D) reconstructions of zebrafish locomotion, globally assessed behavioral profiles evoked by several anxiogenic and anxiolytic manipulations, mapped individual endpoints to 3D reconstructions, and performed cluster analysis to reconfirm behavioral correlates of high- and low-anxiety states. The application of 3D swim path reconstructions consolidates behavioral data (while increasing data density) and provides a novel way to examine and represent zebrafish behavior. It also enables rapid optimization of video tracking settings to improve quantification of automated parameters, and suggests that spatiotemporal organization of zebrafish swimming activity can be affected by various experimental manipulations in a manner predicted by their anxiolytic or anxiogenic nature. Our approach markedly enhances the power of zebrafish behavioral analyses, providing innovative framework for high-throughput 3D phenotyping of adult zebrafish behavior.

Aquatic manoeuvering with counter-propagating waves: a novel locomotive strategy

Many aquatic organisms swim by means of an undulating fin. These undulations often form a single wave travelling from one end of the fin to the other. However, when these aquatic animals are holding station or hovering, there is often a travelling wave from the head to the tail, and another moving from the tail to the head, meeting in the middle of the fin. Our study uses a biomimetic fish robot and computational fluid dynamics on a model of a real fish to uncover the mechanics of these inward counter-propagating waves. In addition, we compare the flow structure and upward force generated by inward counter-propagating waves to standing waves, unidirectional waves, and outward counter-propagating waves (i.e. one wave travelling from the middle of the fin to the head, and another wave travelling from the middle of the fin to the tail). Using digital particle image velocimetry to capture the flow structure around the fish robot, and computational fluid dynamics, we show that inward counter-propagating waves generate a clear mushroom-cloud-like flow structure with an inverted jet. The two streams of fluid set up by the two travelling waves 'collide' together (forming the mushroom cap) and collect into a narrow jet away from the cap (the mushroom stem). The reaction force from this jet acts to push the body in the opposite direction to the jet, perpendicular to the direction of movement provided by a single travelling wave. This downward jet provides a substantial increase in the perpendicular force when compared with the other types of fin actuation. Animals can thereby move upward if the fin is along the bottom midline of the body (or downward if on top); or left-right if the fins are along the lateral margins. In addition to illuminating how a large number of undulatory swimmers can use elongated fins to move in unexpected directions, the phenomenon of counter-propagating waves provides novel motion capabilities for systems using robotic undulators, an emerging technology for propelling underwater vehicles.

Energy-information trade-offs between movement and sensing

While there is accumulating evidence for the importance of the metabolic cost of information in sensory systems, how these costs are traded-off with movement when sensing is closely linked to movement is poorly understood. For example, if an animal needs to search a given amount of space beyond the range of its vision system, is it better to evolve a higher acuity visual system, or evolve a body movement system that can more rapidly move the body over that space? How is this trade-off dependent upon the three-dimensional shape of the field of sensory sensitivity (hereafter, sensorium)? How is it dependent upon sensorium mobility, either through rotation of the sensorium via muscles at the base of the sense organ (e.g., eye or pinna muscles) or neck rotation, or by whole body movement through space? Here we show that in an aquatic model system, the electric fish, a choice to swim in a more inefficient manner during prey search results in a higher prey encounter rate due to better sensory performance. The increase in prey encounter rate more than counterbalances the additional energy expended in swimming inefficiently. The reduction of swimming efficiency for improved sensing arises because positioning the sensory receptor surface to scan more space per unit time results in an increase in the area of the body pushing through the fluid, increasing wasteful body drag forces. We show that the improvement in sensory performance that occurs with the costly repositioning of the body depends upon having an elongated sensorium shape. Finally, we show that if the fish was able to reorient their sensorium independent of body movement, as fish with movable eyes can, there would be significant energy savings. This provides insight into the ubiquity of sensory organ mobility in animal design. This study exposes important links between the morphology of the sensorium, sensorium mobility, and behavioral strategy for maximally extracting energy from the environment. An “infomechanical” approach to complex behavior helps to elucidate how animals distribute functions across sensory systems and movement systems with their diverse energy loads.

Animals thrive by sensing their environment and using the information they've gathered to guide their movement. But collecting better information can result in less efficient movement: Bicycling while standing up on the pedals may help you see over obstacles ahead of you, but it causes more air drag, forcing your legs to work harder. Nocturnal weakly electric fish search for prey with their body tilted. This tilting more than doubles the resistance to movement from the water, but because the fish's ability to sense prey improves when tilted, it is better to swim this way. Beyond a certain amount of tilt, the costs of movement become too great. Interestingly, the benefit of tilting is dependent on the shape of the volume around the fish where it detects prey. We also found that if the fish was able to swivel its region of prey sensitivity, like a vision-based animal can shift its gaze, it would save energy. This conclusion helps us understand why animals like us can move our eyes. A Polish folk saying succinctly captures the gist: “He who doesn't have it in the head has it in the legs” (Ten kto nie ma w głowie ma w nogach).

The hydrodynamics of ribbon-fin propulsion during impulsive motion

Weakly electric fish are extraordinarily maneuverable swimmers, able to swim as easily forward as backward and rapidly switch swim direction, among other maneuvers. The primary propulsor of gymnotid electric fish is an elongated ribbon-like anal fin. To understand the mechanical basis of their maneuverability, we examine the hydrodynamics of a non-translating ribbon fin in stationary water using computational fluid dynamics and digital particle image velocimetry (DPIV) of the flow fields around a robotic ribbon fin. Computed forces are compared with drag measurements from towing a cast of the fish and with thrust estimates for measured swim-direction reversals. We idealize the movement of the fin as a traveling sinusoidal wave, and derive scaling relationships for how thrust varies with the wavelength, frequency,amplitude of the traveling wave and fin height. We compare these scaling relationships with prior theoretical work. The primary mechanism of thrust production is the generation of a streamwise central jet and the associated attached vortex rings. Under certain traveling wave regimes, the ribbon fin also generates a heave force, which pushes the body up in the body-fixed frame. In one such regime, we show that as the number of waves along the fin decreases to approximately two-thirds, the heave force surpasses the surge force. This switch from undulatory parallel thrust to oscillatory normal thrust may be important in understanding how the orientation of median fins may vary with fin length and number of waves along them. Our results will be useful for understanding the neural basis of control in the weakly electric knifefish as well as for engineering bio-inspired vehicles with undulatory thrusters.

Optimal movement in the prey strikes of weakly electric fish: a case study of the interplay of body plan and movement capability

Animal behaviour arises through a complex mixture of biomechanical, neuronal, sensory and control constraints. By focusing on a simple, stereotyped movement, the prey capture strike of a weakly electric fish, we show that the trajectory of a strike is one which minimizes effort. Specifically, we model the fish as a rigid ellipsoid moving through a fluid with no viscosity, governed by Kirchhoff's equations. This formulation allows us to exploit methods of discrete mechanics and optimal control to compute idealized fish trajectories that minimize a cost function. We compare these with the measured prey capture strikes of weakly electric fish from a previous study. The fish has certain movement limitations that are not incorporated in the mathematical model, such as not being able to move sideways. Nonetheless, we show quantitatively that the computed least-cost trajectories are remarkably similar to the measured trajectories. Since, in this simplified model, the basic geometry of the idealized fish determines the favourable modes of movement, this suggests a high degree of influence between body shape and movement capability. Simplified minimal models and optimization methods can give significant insight into how body morphology and movement capability are closely attuned in fish locomotion.

Video tracking in the extreme: video analysis for nocturnal underwater animal movement

Computer analysis of video footage is one option for recording locomotor behavior for a range of neurophysiological and behavioral studies. This technique is reasonably well established and accepted, but its use for some behavioral analyses remains a challenge. For example, filming through water can lead to reflection, and filming nocturnal activity can reduce resolution and clarity of filmed images. The aim of this study was to develop a noninvasive method for recording nocturnal activity in aquatic decapods and test the accuracy of analysis by video tracking software. We selected crayfish, Cherax destructor, because they are often active at night, they live underwater, and data on their locomotion is important for answering biological and physiological questions such as how they explore and navigate. We constructed recording arenas and filmed animals in infrared light. Wethen compared human observer data and software-acquired values. In this article, we outline important apparatus and software issues to obtain reliable computer tracking.

Omnidirectional sensory and motor volumes in electric fish

Active sensing organisms, such as bats, dolphins, and weakly electric fish, generate a 3-D space for active sensation by emitting self-generated energy into the environment. For a weakly electric fish, we demonstrate that the electrosensory space for prey detection has an unusual, omnidirectional shape. We compare this sensory volume with the animal's motor volume--the volume swept out by the body over selected time intervals and over the time it takes to come to a stop from typical hunting velocities. We find that the motor volume has a similar omnidirectional shape, which can be attributed to the fish's backward-swimming capabilities and body dynamics. We assessed the electrosensory space for prey detection by analyzing simulated changes in spiking activity of primary electrosensory afferents during empirically measured and synthetic prey capture trials. The animal's motor volume was reconstructed from video recordings of body motion during prey capture behavior. Our results suggest that in weakly electric fish, there is a close connection between the shape of the sensory and motor volumes. We consider three general spatial relationships between 3-D sensory and motor volumes in active and passive-sensing animals, and we examine hypotheses about these relationships in the context of the volumes we quantify for weakly electric fish. We propose that the ratio of the sensory volume to the motor volume provides insight into behavioral control strategies across all animals.

Analyzing Octopus Movements Using Three-Dimensional Reconstruction

Octopus arms, as well as other muscular hydrostats, are characterized by a very large number of degrees of freedom and a rich motion repertoire. Over the years, several attempts have been made to elucidate the interplay between the biomechanics of these organs and their control systems. Recent developments in electrophysiological recordings from both the arms and brains of behaving octopuses mark significant progress in this direction. The next stage is relating these recordings to the octopus arm movements, which requires an accurate and reliable method of movement description and analysis. Here we describe a semiautomatic computerized system for 3D reconstruction of an octopus arm during motion. It consists of two digital video cameras and a PC computer running custom-made software. The system overcomes the difficulty of extracting the motion of smooth, nonrigid objects in poor viewing conditions. Some of the trouble is explained by the problem of light refraction in recording underwater motion. Here we use both experiments and simulations to analyze the refraction problem and show that accurate reconstruction is possible. We have used this system successfully to reconstruct different types of octopus arm movements, such as reaching and bend initiation movements. Our system is noninvasive and does not require attaching any artificial markers to the octopus arm. It may therefore be of more general use in reconstructing other nonrigid, elongated objects in motion.

The decoding of electrosensory systems

Progress in the study of electrosensory systems has been facilitated by the systematic use of behavior as a tool to probe the nervous system. Indeed, a specific behavior that is found in a subset of weakly electric fishes, the jamming avoidance response, was used to identify and characterize an entire suite of brain circuits, from sensory receptors to motor units, that are involved in control of this behavior. Recent progress has focused on a re-analysis of this circuit in relation to newly described electrosensory behaviors, including prey capture, social signaling and the tracking of electrosensory objects. This re-analysis has led to a re-evaluation of the broader functional relevance of specific neural solutions to computational problems that are related to the control of the jamming avoidance response. Some of the recent insights that have emerged from this work include descriptions of mechanisms underlying dynamic receptive field properties, descriptions of the neural activity related to simultaneously occurring sensory stimuli, and a greater understanding of the role of short-term synaptic plasticity in temporal processing.

Quantitative Characterization and Classification of Leech Behavior

This paper describes an automatic system for the analysis and classification of leech behavior. Three colored beads were attached to the dorsal side of a free moving or pinned leech, and color CCD camera images were taken of the animal. The leech was restrained to moving in a small tank or petri dish, where the water level can be varied. An automatic system based on color processing tracked the colored beads over time, allowing real-time monitoring of the leech motion for several hours. At the end of each experimental session, six time series (2 for each bead) describing the leech body motion were obtained. A statistical analysis based on the speed and frequency content of bead motion indicated the existence of several stereotypical patterns of motion, corresponding to different leech behaviors. The identified patterns corresponded to swimming, pseudo-swimming, crawling, exploratory behavior, stationary states, abrupt movements, and combinations of these behaviors. The automatic characterization of leech behavior demonstrated here represents an important step toward understanding leech behavior and its properties. This method can be used to characterize the behavior of other invertebrates and also for some small vertebrates.

Modeling signal and background components of electrosensory scenes

Weakly electric fish are able to detect and localize prey based on microvolt-level perturbations in the fish's self-generated electric field. In natural environments, weak prey-related signals are embedded in much stronger electrosensory background noise. To better characterize the signal and background components associated with natural electrolocation tasks, we recorded transdermal voltage modulations in restrained Apteronotus albifrons in response to moving spheres, tail bends, and large nonconducting boundaries. Spherical objects give rise to ipsilateral images with center-surround structure and contralateral images that are weak and diffuse. Tail bends and laterally placed nonconducting boundaries induce relatively strong ipsilateral and contralateral modulations of opposite polarity. We present a computational model of electric field generation and electrosensory image formation that is able to reproduce the key features of these empirically measured signal and background components in a unified framework. The model comprises an array of point sources and sinks distributed along the midline of the fish, which can conform to arbitrary body bends. The model is computationally fast and can be used to estimate the spatiotemporal pattern of activation across the entire electroreceptor array of the fish during natural behaviors.

A posture optimization algorithm for model-based motion capture of movement sequences

We have developed and evaluated a new optical motion capture approach that is suitable for a wide range of studies in neuroethology and motor control. Based on the stochastic search algorithm of Simulated Annealing (SA), it utilizes a kinematic body model that includes joint angle constraints to reconstruct posture from an arbitrary number of views. Rather than tracking marker trajectories in time, the algorithm minimizes an error function that compares predicted model projections to the recorded views. Thus, each video-frame is analyzed independently from other frames, enabling the system to recover from incorrectly analyzed postures. The system works with standard computer and video equipment. Its accuracy is evaluated using videos of animated locust leg movements, recorded by two orthogonal views. The resulting joint angle RMS errors range between 0.7 degrees and 4.9 degrees, limited by the pixel resolution of the digital video. 3D-movement reconstruction is possible even from a single view. In a real experimental application, stick insect walking sequences are analyzed with leg joint angle deviations between 0.5 degrees and 3.0 degrees. This robust and accurate performance is reached in spite of marker fusions and occlusions, simply by exploiting the natural constraints imposed by a kinematic chain and a known experimental setup.

Prey-capture behavior in gymnotid electric fish: motion analysis and effects of water conductivity

Animals can actively influence the content and quality of sensory information they acquire from the environment through the positioning of peripheral sensory surfaces. This study investigated receptor surface positioning during prey-capture behavior in weakly electric gymnotiform fish of the genus Apteronotus. Infrared video techniques and three-dimensional model-based tracking methods were used to provide quantitative information on body position and conformation as black ghost (A. albifrons) and brown ghost (A. leptorhynchus) knifefish hunted for prey (Daphnia magna) in the dark. We found that detection distance depends on the electrical conductivity of the surrounding water. Best performance was observed at low water conductivity (2.8 cm mean detection distance and 2 % miss rate at 35 microS cm(−)(1), A. albifrons) and poorest performance at high conductivity (1.5 cm mean detection distance and 11 % miss rate at 600 microS cm(−)(1), A. albifrons). The observed conductivity-dependence implies that nonvisual prey detection in Apteronotus is likely to be dominated by the electrosense over the range of water conductivities experienced by the animal in its natural environment. This result provides the first evidence for the involvement of electrosensory cues in the prey-capture behavior of gymnotids, but it leaves open the possibility that both the high-frequency (tuberous) and low-frequency (ampullary) electroreceptors may contribute. We describe an electrosensory orienting response to prey, whereby the fish rolls its body following detection to bring the prey above the dorsum. This orienting response and the spatial distribution of prey at the time of detection highlight the importance of the dorsal surface of the trunk for electrosensory signal acquisition. Finally, quantitative analysis of fish motion demonstrates that Apteronotus can adapt its trajectory to account for post-detection motion of the prey, suggesting that it uses a closed-loop adaptive tracking strategy, rather than an open-loop ballistic strike strategy, to intercept the prey.