NeuroMechFly, a neuromechanical model of adult Drosophila melanogaster

NeuroMechFly v2: simulating embodied sensorimotor control in adult Drosophila

Discovering principles underlying the control of animal behavior requires a tight dialogue between experiments and neuromechanical models. Such models have primarily been used to investigate motor control with less emphasis on how the brain and motor systems work together during hierarchical sensorimotor control. NeuroMechFly v2 expands Drosophila neuromechanical modeling by enabling vision, olfaction, ascending motor feedback and complex terrains that can be navigated using leg adhesion. We illustrate its capabilities by constructing biologically inspired controllers that use ascending feedback to perform path integration and head stabilization. After adding vision and olfaction, we train a controller using reinforcement learning to perform a multimodal navigation task. Finally, we illustrate more bio-realistic modeling involving complex odor plume navigation, and fly-fly following using a connectome-constrained visual network. NeuroMechFly can be used to accelerate the discovery of explanatory models of the nervous system and to develop machine learning-based controllers for autonomous artificial agents and robots.

An integrative data-driven model simulating C. elegans brain, body and environment interactions

The behavior of an organism is influenced by the complex interplay between its brain, body and environment. Existing data-driven models focus on either the brain or the body-environment. Here we present BAAIWorm, an integrative data-driven model of Caenorhabditis elegans, which consists of two submodels: the brain model and the body-environment model. The brain model was built by multicompartment models with realistic morphology, connectome and neural population dynamics based on experimental data. Simultaneously, the body-environment model used a lifelike body and a three-dimensional physical environment. Through the closed-loop interaction between the two submodels, BAAIWorm reproduced the realistic zigzag movement toward attractors observed in C. elegans. Leveraging this model, we investigated the impact of neural system structure on both neural activities and behaviors. Consequently, BAAIWorm can enhance our understanding of how the brain controls the body to interact with its surrounding environment.

Integrative techniques for insect behavior analysis using micro-CT and Blender

In this paper, we provide an approach that can simulate the behavior of insects, and the aggressive behavior of fruit flies is shown as an example. The specific workflow is as follows. (1) We obtained high-speed camera video of the fly's aggressive behavior. (2) Based on the high-speed camera video, we generated the key action diagrams for each movement. (3, 4) We used micro-computed tomography imaging to segment the leg exoskeleton models using Amira 6.0. (5) With the Blender software, we optimized the OBJ model. (6) We gave motion properties to the 3-dimensional biomechanical model in Blender. (7) Based on high-speed camera videos and the key action diagrams, we generated a 4-dimensional precision adult Drosophila melanogaster biomechanical model. Our study provides a new approach to study rapid locomotion in insects. In addition, our study provides a new idea for establishment of a 4D database, the design and fabrication of bionic multipedal robots, and the linking of nerve signaling and muscle stretching processes.

Connectome-constrained networks predict neural activity across the fly visual system

We can now measure the connectivity of every neuron in a neural circuit1-9, but we cannot measure other biological details, including the dynamical characteristics of each neuron. The degree to which measurements of connectivity alone can inform the understanding of neural computation is an open question10. Here we show that with experimental measurements of only the connectivity of a biological neural network, we can predict the neural activity underlying a specified neural computation. We constructed a model neural network with the experimentally determined connectivity for 64 cell types in the motion pathways of the fruit fly optic lobe1-5 but with unknown parameters for the single-neuron and single-synapse properties. We then optimized the values of these unknown parameters using techniques from deep learning11, to allow the model network to detect visual motion12. Our mechanistic model makes detailed, experimentally testable predictions for each neuron in the connectome. We found that model predictions agreed with experimental measurements of neural activity across 26 studies. Our work demonstrates a strategy for generating detailed hypotheses about the mechanisms of neural circuit function from connectivity measurements. We show that this strategy is more likely to be successful when neurons are sparsely connected-a universally observed feature of biological neural networks across species and brain regions.

Digital twins and the digital logics of biodiversity

Biodiversity is a multidimensional concept that can be understood and measured in many different ways. However, the next generation of digital technologies for biodiversity monitoring currently being funded and developed fail to engage its multidimensional and relational aspects. Based on empirical data from interviews, a conference visit, online meetings, webinars, and project reports, this study articulates four digital logics that structure how biodiversity becomes monitored and understood within recent technological developments. The four digital logics illustrate how intensified practices of capturing, connecting, simulating, and computing produce particular techno-scientific formats for creating biodiversity knowledge. While ongoing projects advance technological development in areas of automation, prediction, and the creation of large-scale species databases, their developmental processes structurally limit the future of biodiversity technology. To better address the complex challenges of the global biodiversity crisis, it is crucial to develop digital technologies and practices that can engage with a wider range of perspectives and understandings of relational and multidimensional approaches to biodiversity.

Synaptic architecture of leg and wing premotor control networks in Drosophila

Animal movement is controlled by motor neurons (MNs), which project out of the central nervous system to activate muscles1. MN activity is coordinated by complex premotor networks that facilitate the contribution of individual muscles to many different behaviours2-6. Here we use connectomics7 to analyse the wiring logic of premotor circuits controlling the Drosophila leg and wing. We find that both premotor networks cluster into modules that link MNs innervating muscles with related functions. Within most leg motor modules, the synaptic weights of each premotor neuron are proportional to the size of their target MNs, establishing a circuit basis for hierarchical MN recruitment. By contrast, wing premotor networks lack proportional synaptic connectivity, which may enable more flexible recruitment of wing steering muscles. Through comparison of the architecture of distinct motor control systems within the same animal, we identify common principles of premotor network organization and specializations that reflect the unique biomechanical constraints and evolutionary origins of leg and wing motor control.

Connectomic reconstruction of a female Drosophila ventral nerve cord

A deep understanding of how the brain controls behaviour requires mapping neural circuits down to the muscles that they control. Here, we apply automated tools to segment neurons and identify synapses in an electron microscopy dataset of an adult female Drosophila melanogaster ventral nerve cord (VNC)1, which functions like the vertebrate spinal cord to sense and control the body. We find that the fly VNC contains roughly 45 million synapses and 14,600 neuronal cell bodies. To interpret the output of the connectome, we mapped the muscle targets of leg and wing motor neurons using genetic driver lines2 and X-ray holographic nanotomography3. With this motor neuron atlas, we identified neural circuits that coordinate leg and wing movements during take-off. We provide the reconstruction of VNC circuits, the motor neuron atlas and tools for programmatic and interactive access as resources to support experimental and theoretical studies of how the nervous system controls behaviour.

A leg model based on anatomical landmarks to study 3D joint kinematics of walking in Drosophila melanogaster

Walking is the most common form of how animals move on land. The model organism Drosophila melanogaster has become increasingly popular for studying how the nervous system controls behavior in general and walking in particular. Despite recent advances in tracking and modeling leg movements of walking Drosophila in 3D, there are still gaps in knowledge about the biomechanics of leg joints due to the tiny size of fruit flies. For instance, the natural alignment of joint rotational axes was largely neglected in previous kinematic analyses. In this study, we therefore present a detailed kinematic leg model in which not only the segment lengths but also the main rotational axes of the joints were derived from anatomical landmarks, namely, the joint condyles. Our model with natural oblique joint axes is able to adapt to the 3D leg postures of straight and forward walking fruit flies with high accuracy. When we compared our model to an orthogonalized version, we observed that our model showed a smaller error as well as differences in the used range of motion (ROM), highlighting the advantages of modeling natural rotational axes alignment for the study of joint kinematics. We further found that the kinematic profiles of front, middle, and hind legs differed in the number of required degrees of freedom as well as their contributions to stepping, time courses of joint angles, and ROM. Our findings provide deeper insights into the joint kinematics of walking in Drosophila, and, additionally, will help to develop dynamical, musculoskeletal, and neuromechanical simulations.

Synaptic architecture of leg and wing premotor control networks in Drosophila

Animal movement is controlled by motor neurons (MNs), which project out of the central nervous system to activate muscles. MN activity is coordinated by complex premotor networks that allow individual muscles to contribute to many different behaviors. Here, we use connectomics to analyze the wiring logic of premotor circuits controlling the Drosophila leg and wing. We find that both premotor networks cluster into modules that link MNs innervating muscles with related functions. Within most leg motor modules, the synaptic weights of each premotor neuron are proportional to the size of their target MNs, establishing a circuit basis for hierarchical MN recruitment. In contrast, wing premotor networks lack proportional synaptic connectivity, which may allow wing steering muscles to be recruited with different relative timing. By comparing the architecture of distinct limb motor control systems within the same animal, we identify common principles of premotor network organization and specializations that reflect the unique biomechanical constraints and evolutionary origins of leg and wing motor control.

Sensorimotor delays constrain robust locomotion in a 3D kinematic model of fly walking

Walking animals must maintain stability in the presence of external perturbations, despite significant temporal delays in neural signaling and muscle actuation. Here, we develop a 3D kinematic model with a layered control architecture to investigate how sensorimotor delays constrain robustness of walking behavior in the fruit fly, Drosophila. Motivated by the anatomical architecture of insect locomotor control circuits, our model consists of three component layers: a neural network that generates realistic 3D joint kinematics for each leg, an optimal controller that executes the joint kinematics while accounting for delays, and an inter-leg coordinator. The model generates realistic simulated walking that matches real fly walking kinematics and sustains walking even when subjected to unexpected perturbations, generalizing beyond its training data. However, we found that the model's robustness to perturbations deteriorates when sensorimotor delay parameters exceed the physiological range. These results suggest that fly sensorimotor control circuits operate close to the temporal limit at which they can detect and respond to external perturbations. More broadly, we show how a modular, layered model architecture can be used to investigate physiological constraints on animal behavior.

FARMS: Framework for Animal and Robot Modeling and Simulation

The study of animal locomotion and neuromechanical control offers valuable insights for advancing research in neuroscience, biomechanics, and robotics. We have developed FARMS (Framework for Animal and Robot Modeling and Simulation), an open-source, interdisciplinary framework, designed to facilitate access to neuromechanical simulations for modeling, simulation, and analysis of animal locomotion and bio-inspired robotic systems. By providing an accessible and user-friendly platform, FARMS aims to lower the barriers for researchers to explore the complex interactions between the nervous system, musculoskeletal structures, and their environment. Integrating the MuJoCo physics engine in a modular manner, FARMS enables realistic simulations and fosters collaboration among neuroscientists, biologists, and roboticists. FARMS has already been extensively used to study locomotion in animals such as mice, drosophila, fish, salamanders, and centipedes, serving as a platform to investigate the role of central pattern generators and sensory feedback. This article provides an overview of the FARMS framework, discusses its interdisciplinary approach, showcases its versatility through specific case studies, and highlights its effectiveness in advancing our understanding of locomotion. In particular, we show how we used FARMS to study amphibious locomotion by presenting experimental demonstrations across morphologies and environments based on neural controllers with central pattern generators and sensory feedback circuits models. Overall, the goal of FARMS is to contribute to a deeper understanding of animal locomotion, the development of innovative bio-inspired robotic systems, and promote accessibility in neuromechanical research.

Grounding neuroscience in behavioral changes using artificial neural networks

Connecting neural activity to function is a common aim in neuroscience. How to define and conceptualize function, however, can vary. Here I focus on grounding this goal in the specific question of how a given change in behavior is produced by a change in neural circuits or activity. Artificial neural network models offer a particularly fruitful format for tackling such questions because they use neural mechanisms to perform complex transformations and produce appropriate behavior. Therefore, they can be a means of causally testing the extent to which a neural change can be responsible for an experimentally observed behavioral change. Furthermore, because the field of interpretability in artificial intelligence has similar aims, neuroscientists can look to interpretability methods for new ways of identifying neural features that drive performance and behaviors.

Three-dimensional surface motion capture of multiple freely moving pigs using MAMMAL

Understandings of the three-dimensional social behaviors of freely moving large-size mammals are valuable for both agriculture and life science, yet challenging due to occlusions in close interactions. Although existing animal pose estimation methods captured keypoint trajectories, they ignored deformable surfaces which contained geometric information essential for social interaction prediction and for dealing with the occlusions. In this study, we develop a Multi-Animal Mesh Model Alignment (MAMMAL) system based on an articulated surface mesh model. Our self-designed MAMMAL algorithms automatically enable us to align multi-view images into our mesh model and to capture 3D surface motions of multiple animals, which display better performance upon severe occlusions compared to traditional triangulation and allow complex social analysis. By utilizing MAMMAL, we are able to quantitatively analyze the locomotion, postures, animal-scene interactions, social interactions, as well as detailed tail motions of pigs. Furthermore, experiments on mouse and Beagle dogs demonstrate the generalizability of MAMMAL across different environments and mammal species.

A hull reconstruction-reprojection method for pose estimation of free-flying fruit flies

Understanding the mechanisms of insect flight requires high-quality data of free-flight kinematics, e.g. for comparative studies or genetic screens. Although recent improvements in high-speed videography allow us to acquire large amounts of free-flight data, a significant bottleneck is automatically extracting accurate body and wing kinematics. Here, we present an experimental system and a hull reconstruction-reprojection algorithm for measuring the flight kinematics of fruit flies. The experimental system can automatically record hundreds of flight events per day. Our algorithm resolves a significant portion of the occlusions in this system by a reconstruction-reprojection scheme that integrates information from all cameras. Wing and body kinematics, including wing deformation, are then extracted from the hulls of the wing boundaries and body. This model-free method is fully automatic, accurate and open source, and can be readily adjusted for different camera configurations or insect species.

Multilevel visuomotor control of locomotion in Drosophila

Vision is critical for the control of locomotion, but the underlying neural mechanisms by which visuomotor circuits contribute to the movement of the body through space are yet not well understood. Locomotion engages multiple control systems, forming distinct interacting "control levels" driven by the activity of distributed and overlapping circuits. Therefore, a comprehensive understanding of the mechanisms underlying locomotion control requires the consideration of all control levels and their necessary coordination. Due to their small size and the wide availability of experimental tools, Drosophila has become an important model system to study this coordination. Traditionally, insect locomotion has been divided into studying either the biomechanics and local control of limbs, or navigation and course control. However, recent developments in tracking techniques, and physiological and genetic tools in Drosophila have prompted researchers to examine multilevel control coordination in flight and walking.

Mechanosensory Control of Locomotion in Animals and Robots: Moving Forward

While animals swim, crawl, walk, and fly with apparent ease, building robots capable of robust locomotion remains a significant challenge. In this review, we draw attention to mechanosensation-the sensing of mechanical forces generated within and outside the body-as a key sense that enables robust locomotion in animals. We discuss differences between mechanosensation in animals and current robots with respect to (1) the encoding properties and distribution of mechanosensors and (2) the integration and regulation of mechanosensory feedback. We argue that robotics would benefit greatly from a detailed understanding of these aspects in animals. To that end, we highlight promising experimental and engineering approaches to study mechanosensation, emphasizing the mutual benefits for biologists and engineers that emerge from moving forward together.

The neuromechanics of animal locomotion: From biology to robotics and back

Robotics and neuroscience are sister disciplines that both aim to understand how agile, efficient, and robust locomotion can be achieved in autonomous agents. Robotics has already benefitted from neuromechanical principles discovered by investigating animals. These include the use of high-level commands to control low-level central pattern generator-like controllers, which, in turn, are informed by sensory feedback. Reciprocally, neuroscience has benefited from tools and intuitions in robotics to reveal how embodiment, physical interactions with the environment, and sensory feedback help sculpt animal behavior. We illustrate and discuss exemplar studies of this dialog between robotics and neuroscience. We also reveal how the increasing biorealism of simulations and robots is driving these two disciplines together, forging an integrative science of autonomous behavioral control with many exciting future opportunities.

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.

Ascending neurons convey behavioral state to integrative sensory and action selection brain regions

Knowing one's own behavioral state has long been theorized as critical for contextualizing dynamic sensory cues and identifying appropriate future behaviors. Ascending neurons (ANs) in the motor system that project to the brain are well positioned to provide such behavioral state signals. However, what ANs encode and where they convey these signals remains largely unknown. Here, through large-scale functional imaging in behaving animals and morphological quantification, we report the behavioral encoding and brain targeting of hundreds of genetically identifiable ANs in the adult fly, Drosophila melanogaster. We reveal that ANs encode behavioral states, specifically conveying self-motion to the anterior ventrolateral protocerebrum, an integrative sensory hub, as well as discrete actions to the gnathal ganglia, a locus for action selection. Additionally, AN projection patterns within the motor system are predictive of their encoding. Thus, ascending populations are well poised to inform distinct brain hubs of self-motion and ongoing behaviors and may provide an important substrate for computations that are required for adaptive behavior.

How the conception of control influences our understanding of actions

Wilful movement requires neural control. Commonly, neural computations are thought to generate motor commands that bring the musculoskeletal system - that is, the plant - from its current physical state into a desired physical state. The current state can be estimated from past motor commands and from sensory information. Modelling movement on the basis of this concept of plant control strives to explain behaviour by identifying the computational principles for control signals that can reproduce the observed features of movements. From an alternative perspective, movements emerge in a dynamically coupled agent-environment system from the pursuit of subjective perceptual goals. Modelling movement on the basis of this concept of perceptual control aims to identify the controlled percepts and their coupling rules that can give rise to the observed characteristics of behaviour. In this Perspective, we discuss a broad spectrum of approaches to modelling human motor control and their notions of control signals, internal models, handling of sensory feedback delays and learning. We focus on the influence that the plant control and the perceptual control perspective may have on decisions when modelling empirical data, which may in turn shape our understanding of actions.

Emergent behaviour and neural dynamics in artificial agents tracking odour plumes

Tracking an odour plume to locate its source under variable wind and plume statistics is a complex task. Flying insects routinely accomplish such tracking, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behaviour and its underlying neural circuitry have been studied experimentally. Here we take a complementary in silico approach to develop an integrated understanding of their behaviour and neural computations. Specifically, we train artificial recurrent neural network agents using deep reinforcement learning to locate the source of simulated odour plumes that mimic features of plumes in a turbulent flow. Interestingly, the agents' emergent behaviours resemble those of flying insects, and the recurrent neural networks learn to compute task-relevant variables with distinct dynamic structures in population activity. Our analyses put forward a testable behavioural hypothesis for tracking plumes in changing wind direction, and we provide key intuitions for memory requirements and neural dynamics in odour plume tracking.

Descending neuron population dynamics during odor-evoked and spontaneous limb-dependent behaviors

Deciphering how the brain regulates motor circuits to control complex behaviors is an important, long-standing challenge in neuroscience. In the fly, Drosophila melanogaster, this is coordinated by a population of ~ 1100 descending neurons (DNs). Activating only a few DNs is known to be sufficient to drive complex behaviors like walking and grooming. However, what additional role the larger population of DNs plays during natural behaviors remains largely unknown. For example, they may modulate core behavioral commands or comprise parallel pathways that are engaged depending on sensory context. We evaluated these possibilities by recording populations of nearly 100 DNs in individual tethered flies while they generated limb-dependent behaviors, including walking and grooming. We found that the largest fraction of recorded DNs encode walking while fewer are active during head grooming and resting. A large fraction of walk-encoding DNs encode turning and far fewer weakly encode speed. Although odor context does not determine which behavior-encoding DNs are recruited, a few DNs encode odors rather than behaviors. Lastly, we illustrate how one can identify individual neurons from DN population recordings by using their spatial, functional, and morphological properties. These results set the stage for a comprehensive, population-level understanding of how the brain's descending signals regulate complex motor actions.

Economic Order Quantity Model-Based Optimized Fuzzy Nonlinear Dynamic Mathematical Schemes

Fuzzy mathematics-informed methods are beneficial in cases when observations display uncertainty and volatility since it is of vital importance to make predictions about the future considering the stages of interpreting, planning, and strategy building. It is possible to realize this aim through accurate, reliable, and realistic data and information analysis, emerging from past to present time. The principal expenditures are treated as fuzzy numbers in this article, which includes a blurry categorial prototype with pattern-diverse stipulation and collapse with salvation worth. Multiple parameters such as a shortage, ordering, and degrading cost are not fixed in nature due to uncertainty in the marketplace. Obtaining an accurate estimate of such expenditures is challenging. Accordingly, in this research, we develop an adaptive and integrative economic order quantity model with a fuzzy method and present an appropriate structure to manage such uncertain parameters, boosting the inventory system's exactness, and computing efficiency. The major goal of the study was to assess a set of changes to the company current inventory processes that allowed an achievement in its inventory costs optimization and system development in optimizing inventory costs for better control and monitoring. The approach of graded mean integration is used to determine the most efficient actual solution. The evidence-based model is illustrated with the help of appropriate numerical and sensitivity analysis through the related visual graphical depictions. The proposed method in our study aims at investigating the economic order quantity (EOQ), as the optimal order quantity, which is significant in inventory management to minimize the total costs related to ordering, receiving, and holding inventory in the dynamic domains with nonlinear features of the complex dynamic and nonlinear systems as well as structures.