DeepLabStream enables closed-loop behavioral experiments using deep learning-based markerless, real-time posture detection

A-SOiD, an active-learning platform for expert-guided, data-efficient discovery of behavior

To identify and extract naturalistic behavior, two methods have become popular: supervised and unsupervised. Each approach carries its own strengths and weaknesses (for example, user bias, training cost, complexity and action discovery), which the user must consider in their decision. Here, an active-learning platform, A-SOiD, blends these strengths, and in doing so, overcomes several of their inherent drawbacks. A-SOiD iteratively learns user-defined groups with a fraction of the usual training data, while attaining expansive classification through directed unsupervised classification. In socially interacting mice, A-SOiD outperformed standard methods despite requiring 85% less training data. Additionally, it isolated ethologically distinct mouse interactions via unsupervised classification. We observed similar performance and efficiency using nonhuman primate and human three-dimensional pose data. In both cases, the transparency in A-SOiD's cluster definitions revealed the defining features of the supervised classification through a game-theoretic approach. To facilitate use, A-SOiD comes as an intuitive, open-source interface for efficient segmentation of user-defined behaviors and discovered sub-actions.

An automated, low-latency environment for studying the neural basis of behavior in freely moving rats

BACKGROUND

Behavior consists of the interaction between an organism and its environment, and is controlled by the brain. Brain activity varies at sub-second time scales, but behavioral measures are usually coarse (often consisting of only binary trial outcomes).

RESULTS

To overcome this mismatch, we developed the Rat Interactive Foraging Facility (RIFF): a programmable interactive arena for freely moving rats with multiple feeding areas, multiple sound sources, high-resolution behavioral tracking, and simultaneous electrophysiological recordings. The paper provides detailed information about the construction of the RIFF and the software used to control it. To illustrate the flexibility of the RIFF, we describe two complex tasks implemented in the RIFF, a foraging task and a sound localization task. Rats quickly learned to obtain rewards in both tasks. Neurons in the auditory cortex as well as neurons in the auditory field in the posterior insula had sound-driven activity during behavior. Remarkably, neurons in both structures also showed sensitivity to non-auditory parameters such as location in the arena and head-to-body angle.

CONCLUSIONS

The RIFF provides insights into the cognitive capabilities and learning mechanisms of rats and opens the way to a better understanding of how brains control behavior. The ability to do so depends crucially on the combination of wireless electrophysiology and detailed behavioral documentation available in the RIFF.

Open-MAC: A low-cost open-source motorized commutator for electro- and opto-physiological recordings in freely moving rodents

In vivo electro- and optophysiology experiments in rodents reveal the neural mechanisms underlying behavior and brain disorders but mostly involve a cable connection between an implant in the animal and an external recording device. Standard tethers with thin cables or non-motorized commutators require constant monitoring and often manual interference to untwist the cable. Motorized commutators offer a solution, but those few that are commercially available are expensive and often not adapted to widely used connector standards of the open-source community like 12-channel SPI. Here we introduce an open-source motorized all-in-one commutator (Open-MAC): a low-cost (240-390 EUR), low-torque motorized commutator that can operate with minimal audible noise in a torque-based mode relying on dual magnetic Hall sensors. It further includes electronics to operate in a torque-free, online pose-estimation-based mode, with future developments. Operation is controlled by an onboard microcontroller (XIAO SAMD21) powered by a USB-C cable or DC power supply. The body and movable parts are 3D-printed. Different Open-MAC versions can support electrophysiology with up to 64 recording channels using the Open-Ephys / Intan^TM recording systems as well as miniature endoscope (miniscope) recordings using the UCLA Miniscope v3/4, and can host a fibre for optogenetic modulation.

Open-source tools for behavioral video analysis: Setup, methods, and best practices

Recently developed methods for video analysis, especially models for pose estimation and behavior classification, are transforming behavioral quantification to be more precise, scalable, and reproducible in fields such as neuroscience and ethology. These tools overcome long-standing limitations of manual scoring of video frames and traditional 'center of mass' tracking algorithms to enable video analysis at scale. The expansion of open-source tools for video acquisition and analysis has led to new experimental approaches to understand behavior. Here, we review currently available open-source tools for video analysis and discuss how to set up these methods for labs new to video recording. We also discuss best practices for developing and using video analysis methods, including community-wide standards and critical needs for the open sharing of datasets and code, more widespread comparisons of video analysis methods, and better documentation for these methods especially for new users. We encourage broader adoption and continued development of these tools, which have tremendous potential for accelerating scientific progress in understanding the brain and behavior.

Automatic extraction of upper-limb kinematic activity using deep learning-based markerless tracking during deep brain stimulation implantation for Parkinson's disease: A proof of concept study

Optimal placement of deep brain stimulation (DBS) therapy for treating movement disorders routinely relies on intraoperative motor testing for target determination. However, in current practice, motor testing relies on subjective interpretation and correlation of motor and neural information. Recent advances in computer vision could improve assessment accuracy. We describe our application of deep learning-based computer vision to conduct markerless tracking for measuring motor behaviors of patients undergoing DBS surgery for the treatment of Parkinson's disease. Video recordings were acquired during intraoperative kinematic testing (N = 5 patients), as part of standard of care for accurate implantation of the DBS electrode. Kinematic data were extracted from videos post-hoc using the Python-based computer vision suite DeepLabCut. Both manual and automated (80.00% accuracy) approaches were used to extract kinematic episodes from threshold derived kinematic fluctuations. Active motor epochs were compressed by modeling upper limb deflections with a parabolic fit. A semi-supervised classification model, support vector machine (SVM), trained on the parameters defined by the parabolic fit reliably predicted movement type. Across all cases, tracking was well calibrated (i.e., reprojection pixel errors 0.016-0.041; accuracies >95%). SVM predicted classification demonstrated high accuracy (85.70%) including for two common upper limb movements, arm chain pulls (92.30%) and hand clenches (76.20%), with accuracy validated using a leave-one-out process for each patient. These results demonstrate successful capture and categorization of motor behaviors critical for assessing the optimal brain target for DBS surgery. Conventional motor testing procedures have proven informative and contributory to targeting but have largely remained subjective and inaccessible to non-Western and rural DBS centers with limited resources. This approach could automate the process and improve accuracy for neuro-motor mapping, to improve surgical targeting, optimize DBS therapy, provide accessible avenues for neuro-motor mapping and DBS implantation, and advance our understanding of the function of different brain areas.

Neuroscience Cloud Analysis As a Service: An open-source platform for scalable, reproducible data analysis

A key aspect of neuroscience research is the development of powerful, general-purpose data analyses that process large datasets. Unfortunately, modern data analyses have a hidden dependence upon complex computing infrastructure (e.g., software and hardware), which acts as an unaddressed deterrent to analysis users. Although existing analyses are increasingly shared as open-source software, the infrastructure and knowledge needed to deploy these analyses efficiently still pose significant barriers to use. In this work, we develop Neuroscience Cloud Analysis As a Service (NeuroCAAS): a fully automated open-source analysis platform offering automatic infrastructure reproducibility for any data analysis. We show how NeuroCAAS supports the design of simpler, more powerful data analyses and that many popular data analysis tools offered through NeuroCAAS outperform counterparts on typical infrastructure. Pairing rigorous infrastructure management with cloud resources, NeuroCAAS dramatically accelerates the dissemination and use of new data analyses for neuroscientific discovery.

Pi USB Cam: A Simple and Affordable DIY Solution That Enables High-Quality, High-Throughput Video Capture for Behavioral Neuroscience Research

Video recording is essential for behavioral neuroscience research, but the majority of available systems suffer from poor cost-to-functionality ratio. Commercial options frequently come at high financial cost that prohibits scalability and throughput, whereas DIY solutions often require significant expertise and time investment unaffordable to many researchers. To address this, we combined a low-cost Raspberry Pi microcomputer, DIY electronics peripherals, freely available open-source firmware, and custom 3D-printed casings to create Pi USB Cam, a simple yet powerful and highly versatile video recording solution. Pi USB Cam is constructed using affordable and widely available components and requires no expertise to build and implement. The result is a system that functions as a plug-and-play USB camera that can be easily installed in various animal testing and housing sites and is readily compatible with popular behavioral and neural recording software. Here, we provide a comprehensive parts list and step-by-step instructions for users to build and implement their own Pi USB Cam system. In a series of benchmark comparisons, Pi USB Cam was able to capture ultra-wide fields of view of behaving rats given limited object distance and produced high image quality while maintaining consistent frame rates even under low-light and no-light conditions relative to a standard, commercially available USB camera. Video recordings were easily scaled using free, open-source software. Altogether, Pi USB Cam presents an elegant yet simple solution for behavioral neuroscientists seeking an affordable and highly flexible system to enable quality video recordings.

Technologies for large-scale mapping of functional neural circuits active during a user-defined time window

The mapping of neural circuits activated during behavior down to individual neurons is crucial for decoding how the brain processes information. Technologies allowing activity-dependent labeling of neurons during user-defined restricted time windows are rapidly developing. Precise marking of the time window with light, in addition to chemicals, is now possible. In these technologies, genetically encoded molecules integrate molecular events resulting from neuronal activity with light/drug-dependent events. The outputs are either changes in fluorescence or activation of gene expression. Molecular reporters allow labeling of activated neurons for visualization and cell-type identification. The transcriptional readout also allows further control of activated neuronal populations using optogenetic tools as reporters. Here we review the design of these technologies and discuss their demonstrated applications to reveal previously unknown connections in the mammalian brain. We also consider the strengths and weaknesses of the current approaches and provide a perspective for the future.

Toward the explainability, transparency, and universality of machine learning for behavioral classification in neuroscience

The use of rigorous ethological observation via machine learning techniques to understand brain function (computational neuroethology) is a rapidly growing approach that is poised to significantly change how behavioral neuroscience is commonly performed. With the development of open-source platforms for automated tracking and behavioral recognition, these approaches are now accessible to a wide array of neuroscientists despite variations in budget and computational experience. Importantly, this adoption has moved the field toward a common understanding of behavior and brain function through the removal of manual bias and the identification of previously unknown behavioral repertoires. Although less apparent, another consequence of this movement is the introduction of analytical tools that increase the explainabilty, transparency, and universality of the machine-based behavioral classifications both within and between research groups. Here, we focus on three main applications of such machine model explainabilty tools and metrics in the drive toward behavioral (i) standardization, (ii) specialization, and (iii) explainability. We provide a perspective on the use of explainability tools in computational neuroethology, and detail why this is a necessary next step in the expansion of the field. Specifically, as a possible solution in behavioral neuroscience, we propose the use of Shapley values via Shapley Additive Explanations (SHAP) as a diagnostic resource toward explainability of human annotation, as well as supervised and unsupervised behavioral machine learning analysis.

AI-enabled, implantable, multichannel wireless telemetry for photodynamic therapy

Photodynamic therapy (PDT) offers several advantages for treating cancers, but its efficacy is highly dependent on light delivery to activate a photosensitizer. Advances in wireless technologies enable remote delivery of light to tumors, but suffer from key limitations, including low levels of tissue penetration and photosensitizer activation. Here, we introduce DeepLabCut (DLC)-informed low-power wireless telemetry with an integrated thermal/light simulation platform that overcomes the above constraints. The simulator produces an optimized combination of wavelengths and light sources, and DLC-assisted wireless telemetry uses the parameters from the simulator to enable adequate illumination of tumors through high-throughput (<20 mice) and multi-wavelength operation. Together, they establish a range of guidelines for effective PDT regimen design. In vivo Hypericin and Foscan mediated PDT, using cancer xenograft models, demonstrates substantial suppression of tumor growth, warranting further investigation in research and/or clinical settings.

SLEAP: A deep learning system for multi-animal pose tracking

The desire to understand how the brain generates and patterns behavior has driven rapid methodological innovation in tools to quantify natural animal behavior. While advances in deep learning and computer vision have enabled markerless pose estimation in individual animals, extending these to multiple animals presents unique challenges for studies of social behaviors or animals in their natural environments. Here we present Social LEAP Estimates Animal Poses (SLEAP), a machine learning system for multi-animal pose tracking. This system enables versatile workflows for data labeling, model training and inference on previously unseen data. SLEAP features an accessible graphical user interface, a standardized data model, a reproducible configuration system, over 30 model architectures, two approaches to part grouping and two approaches to identity tracking. We applied SLEAP to seven datasets across flies, bees, mice and gerbils to systematically evaluate each approach and architecture, and we compare it with other existing approaches. SLEAP achieves greater accuracy and speeds of more than 800 frames per second, with latencies of less than 3.5 ms at full 1,024 × 1,024 image resolution. This makes SLEAP usable for real-time applications, which we demonstrate by controlling the behavior of one animal on the basis of the tracking and detection of social interactions with another animal.

Modelling behaviors relevant to brain disorders in the nonhuman primate: Are we there yet?

Recent years have seen a profound resurgence of activity with nonhuman primates (NHPs) to model human brain disorders. From marmosets to macaques, the study of NHP species offers a unique window into the function of primate-specific neural circuits that are impossible to examine in other models. Examining how these circuits manifest into the complex behaviors of primates, such as advanced cognitive and social functions, has provided enormous insights to date into the mechanisms underlying symptoms of numerous neurological and neuropsychiatric illnesses. With the recent optimization of modern techniques to manipulate and measure neural activity in vivo, such as optogenetics and calcium imaging, NHP research is more well-equipped than ever to probe the neural mechanisms underlying pathological behavior. However, methods for behavioral experimentation and analysis in NHPs have noticeably failed to keep pace with these advances. As behavior ultimately lies at the junction between preclinical findings and its translation to clinical outcomes for brain disorders, approaches to improve the integrity, reproducibility, and translatability of behavioral experiments in NHPs requires critical evaluation. In this review, we provide a unifying account of existing brain disorder models using NHPs, and provide insights into the present and emerging contributions of behavioral studies to the field.

B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors

Studying naturalistic animal behavior remains a difficult objective. Recent machine learning advances have enabled limb localization; however, extracting behaviors requires ascertaining the spatiotemporal patterns of these positions. To provide a link from poses to actions and their kinematics, we developed B-SOiD - an open-source, unsupervised algorithm that identifies behavior without user bias. By training a machine classifier on pose pattern statistics clustered using new methods, our approach achieves greatly improved processing speed and the ability to generalize across subjects or labs. Using a frameshift alignment paradigm, B-SOiD overcomes previous temporal resolution barriers. Using only a single, off-the-shelf camera, B-SOiD provides categories of sub-action for trained behaviors and kinematic measures of individual limb trajectories in any animal model. These behavioral and kinematic measures are difficult but critical to obtain, particularly in the study of rodent and other models of pain, OCD, and movement disorders.

Measuring and modeling the motor system with machine learning

The utility of machine learning in understanding the motor system is promising a revolution in how to collect, measure, and analyze data. The field of movement science already elegantly incorporates theory and engineering principles to guide experimental work, and in this review we discuss the growing use of machine learning: from pose estimation, kinematic analyses, dimensionality reduction, and closed-loop feedback, to its use in understanding neural correlates and untangling sensorimotor systems. We also give our perspective on new avenues, where markerless motion capture combined with biomechanical modeling and neural networks could be a new platform for hypothesis-driven research.

Systems Neuroscience of Natural Behaviors in Rodents

Animals evolved in complex environments, producing a wide range of behaviors, including navigation, foraging, prey capture, and conspecific interactions, which vary over timescales ranging from milliseconds to days. Historically, these behaviors have been the focus of study for ecology and ethology, while systems neuroscience has largely focused on short timescale behaviors that can be repeated thousands of times and occur in highly artificial environments. Thanks to recent advances in machine learning, miniaturization, and computation, it is newly possible to study freely moving animals in more natural conditions while applying systems techniques: performing temporally specific perturbations, modeling behavioral strategies, and recording from large numbers of neurons while animals are freely moving. The authors of this review are a group of scientists with deep appreciation for the common aims of systems neuroscience, ecology, and ethology. We believe it is an extremely exciting time to be a neuroscientist, as we have an opportunity to grow as a field, to embrace interdisciplinary, open, collaborative research to provide new insights and allow researchers to link knowledge across disciplines, species, and scales. Here we discuss the origins of ethology, ecology, and systems neuroscience in the context of our own work and highlight how combining approaches across these fields has provided fresh insights into our research. We hope this review facilitates some of these interactions and alliances and helps us all do even better science, together.