Wireless recording from unrestrained monkeys reveals motor goal encoding beyond immediate reach in frontoparietal cortex

Emerging perspectives for the study of the neural basis of motor behaviour

The 33rd Annual Meeting of the Society for the Neural Control of Movement (NCM) brought together over 500 experts to discuss recent advancements in motor control. This article highlights key topics from the conference, including the foundational mechanisms of motor control, the ongoing debate over the context-dependency of feedforward and feedback processes, and the interplay between motor and cognitive functions in learning, memory, and decision-making. It also presents innovative methods for studying movement in complex, real-world environments.

Peripersonal space: why so last-second?

A vast range of neurophysiological, neuropsychological and behavioural results in monkeys and humans have shown that the immediate surroundings of the body, also known as peripersonal space (PPS), are processed in a unique way. Three roles have been ascribed to PPS mechanisms: to react to threats, to avoid obstacles and to act on objects. However, in many circumstances, one does not wait for objects or agents to enter PPS to plan these behaviours. Typically, one has more chances to survive if one starts running away from the lion when one sees it in the distance than if it is a few steps away. PPS makes sense in shortsighted creatures but we are not such creatures. The crucial question is thus twofold: (i) why are these adaptive processes triggered only at the last second or even milliseconds? And (ii) what is their exact contribution, especially for defensive and navigational behaviours? Here, we propose that PPS mechanisms correspond to a plan B, useful in unpredictable situations or when other anticipatory mechanisms have failed. Furthermore, we argue that there are energetic, cognitive and behavioural costs to PPS mechanisms, which explain why this plan B is triggered only at the last second. This article is part of the theme issue 'Minds in movement: embodied cognition in the age of artificial intelligence'.

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Objective

Creating an intracortical brain-computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.

Approach

We devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.

Main results

We found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network (RNN) decoder with 10-12 clusters.

Significance

This study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

Exploring the Consistent Roles of Motor Areas Across Voluntary Movement and Locomotion

Multiple cortical motor areas are critically involved in the voluntary control of discrete movement (e.g., reaching) and gait. Here, we outline experimental findings in nonhuman primates with clinical reports and research in humans that explain characteristic movement control mechanisms in the primary, supplementary, and presupplementary motor areas, as well as in the dorsal premotor area. We then focus on single-neuron activity recorded while monkeys performed motor sequences consisting of multiple discrete movements, and we consider how area-specific control mechanisms may contribute to the performance of complex movements. Following this, we explore the motor areas in cats that we have considered as analogs of those in primates based on similarities in their cortical surface topology, anatomic connections, microstimulation effects, and activity patterns. Emphasizing that discrete movement and gait modification entail similar control mechanisms, we argue that single-neuron activity in each area of the cat during gait modification is compatible with the function ascribed to the activity in the corresponding area in primates, recorded during the performance of discrete movements. The findings that demonstrate the premotor areas' contribution to locomotion, currently unique to the cat model, should offer highly valuable insights into the control mechanisms of locomotion in primates, including humans.

Deciphering social traits and pathophysiological conditions from natural behaviors in common marmosets

Nonhuman primates (NHPs) are indispensable animal models by virtue of the continuity of behavioral repertoires across primates, including humans. However, behavioral assessment at the laboratory level has so far been limited. Employing the application of three-dimensional (3D) pose estimation and the optimal integration of subsequent analytic methodologies, we demonstrate that our artificial intelligence (AI)-based approach has successfully deciphered the ethological, cognitive, and pathological traits of common marmosets from their natural behaviors. By applying multiple deep neural networks trained with large-scale datasets, we established an evaluation system that could reconstruct and estimate the 3D poses of the marmosets, a small NHP that is suitable for analyzing complex natural behaviors in laboratory setups. We further developed downstream analytic methodologies to quantify a variety of behavioral parameters beyond motion kinematics. We revealed the distinct parental roles of male and female marmosets through automated detections of food-sharing behaviors using a spatial-temporal filter on 3D poses. Employing a recurrent neural network to analyze 3D pose time series data during social interactions, we additionally discovered that marmosets adjusted their behaviors based on others' internal state, which is not directly observable but can be inferred from the sequence of others' actions. Moreover, a fully unsupervised approach enabled us to detect progressively appearing symptomatic behaviors over a year in a Parkinson's disease model. The high-throughput and versatile nature of an AI-driven approach to analyze natural behaviors will open a new avenue for neuroscience research dealing with big-data analyses of social and pathophysiological behaviors in NHPs.

Toward a neuroscience of natural behavior

One of the most exciting new developments in systems neuroscience is the progress being made toward neurophysiological experiments that move beyond simplified laboratory settings and address the richness of natural behavior. This is enabled by technological advances such as wireless recording in freely moving animals, automated quantification of behavior, and new methods for analyzing large data sets. Beyond new empirical methods and data, however, there is also a need for new theories and concepts to interpret that data. Such theories need to address the particular challenges of natural behavior, which often differ significantly from the scenarios studied in traditional laboratory settings. Here, we discuss some strategies for developing such novel theories and concepts and some example hypotheses being proposed.

A brain machine interface framework for exploring proactive control of smart environments

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

Population coding of strategic variables during foraging in freely moving macaques

Until now, it has been difficult to examine the neural bases of foraging in naturalistic environments because previous approaches have relied on restrained animals performing trial-based foraging tasks. Here we allowed unrestrained monkeys to freely interact with concurrent reward options while we wirelessly recorded population activity in the dorsolateral prefrontal cortex. The animals decided when and where to forage based on whether their prediction of reward was fulfilled or violated. This prediction was not solely based on a history of reward delivery, but also on the understanding that waiting longer improves the chance of reward. The task variables were continuously represented in a subspace of the high-dimensional population activity, and this compressed representation predicted the animal's subsequent choices better than the true task variables and as well as the raw neural activity. Our results indicate that monkeys' foraging strategies are based on a cortical model of reward dynamics as animals freely explore their environment.

Development of an assessment method for freely moving nonhuman primates' eating behavior using manual and deep learning analysis

Purpose

Although eating is imperative for survival, few comprehensive methods have been developed to assess freely moving nonhuman primates' eating behavior. In the current study, we distinguished eating behavior into appetitive and consummatory phases and developed nine indices to study them using manual and deep learning-based (DeepLabCut) techniques.

Method

The indices were utilized to three rhesus macaques by different palatability and hunger levels to validate their utility. To execute the experiment, we designed the eating behavior cage and manufactured the artificial food. The total number of trials was 3, with 1 trial conducted using natural food and 2 trials using artificial food.

Result

As a result, the indices of highest utility for hunger effect were approach frequency and consummatory duration. Appetitive composite score and consummatory duration showed the highest utility for palatability effect. To elucidate the effects of hunger and palatability, we developed 2D visualization plots based on manual indices. These 2D visualization methods could intuitively depict the palatability perception and hunger internal state. Furthermore, the developed deep learning-based analysis proved accurate and comparable with manual analysis. When comparing the time required for analysis, deep learning-based analysis was 24-times faster than manual analysis. Moreover, temporal and spatial dynamics were visualized via manual and deep learning-based analysis. Based on temporal dynamics analysis, the patterns were classified into four categories: early decline, steady decline, mid-peak with early incline, and late decline. Heatmap of spatial dynamics and trajectory-related visualization could elucidate a consumption posture and a higher spatial occupancy of food zone in hunger and with palatable food.

Discussion

Collectively, this study describes a newly developed and validated multi-phase method for assessing freely moving nonhuman primate eating behavior using manual and deep learning-based analyses. These effective tools will prove valuable in food reward (palatability effect) and homeostasis (hunger effect) research.

Dynamical flexible inference of nonlinear latent factors and structures in neural population activity

Modelling the spatiotemporal dynamics in the activity of neural populations while also enabling their flexible inference is hindered by the complexity and noisiness of neural observations. Here we show that the lower-dimensional nonlinear latent factors and latent structures can be computationally modelled in a manner that allows for flexible inference causally, non-causally and in the presence of missing neural observations. To enable flexible inference, we developed a neural network that separates the model into jointly trained manifold and dynamic latent factors such that nonlinearity is captured through the manifold factors and the dynamics can be modelled in tractable linear form on this nonlinear manifold. We show that the model, which we named 'DFINE' (for 'dynamical flexible inference for nonlinear embeddings') achieves flexible inference in simulations of nonlinear dynamics and across neural datasets representing a diversity of brain regions and behaviours. Compared with earlier neural-network models, DFINE enables flexible inference, better predicts neural activity and behaviour, and better captures the latent neural manifold structure. DFINE may advance the development of neurotechnology and investigations in neuroscience.

Learning of object-in-context sequences in freely-moving macaques

Flexible learning is a hallmark of primate cognition, which arises through interactions with changing environments. Studies of the neural basis for this flexibility are typically limited by laboratory settings that use minimal environmental cues and restrict interactions with the environment, including active sensing and exploration. To address this, we constructed a 3-D enclosure containing touchscreens on its walls, for studying cognition in freely moving macaques. To test flexible learning, two monkeys completed trials consisting of a regular sequence of object selections across four touchscreens. On each screen, the monkeys had to select by touching the sole correct object item ('target') among a set of four items, irrespective of their positions on the screen. Each item was the target on exactly one screen of the sequence, making correct performance conditioned on the spatiotemporal sequence rule across screens. Both monkeys successfully learned multiple 4-item sets (N=14 and 22 sets), totaling over 50 and 80 unique, conditional item-context memoranda, with no indication of capacity limits. The enclosure allowed freedom of movements leading up to and following the touchscreen interactions. To determine whether movement economy changed with learning, we reconstructed 3D position and movement dynamics using markerless tracking software and gyroscopic inertial measurements. Whereas general body positions remained consistent across repeated sequences, fine head movements varied as monkeys learned, within and across sequence sets, demonstrating learning set or "learning to learn". These results demonstrate monkeys' rapid, capacious, and flexible learning within an integrated, multisensory 3-D space. Furthermore, this approach enables the measurement of continuous behavior while ensuring precise experimental control and behavioral repetition of sequences over time. Overall, this approach harmonizes the design features that are needed for electrophysiological studies with tasks that showcase fully situated, flexible cognition.

Deep learning-based activity recognition and fine motor identification using 2D skeletons of cynomolgus monkeys

Video-based action recognition is becoming a vital tool in clinical research and neuroscientific study for disorder detection and prediction. However, action recognition currently used in non-human primate (NHP) research relies heavily on intense manual labor and lacks standardized assessment. In this work, we established two standard benchmark datasets of NHPs in the laboratory: MonkeyinLab (MiL), which includes 13 categories of actions and postures, and MiL2D, which includes sequences of two-dimensional (2D) skeleton features. Furthermore, based on recent methodological advances in deep learning and skeleton visualization, we introduced the MonkeyMonitorKit (MonKit) toolbox for automatic action recognition, posture estimation, and identification of fine motor activity in monkeys. Using the datasets and MonKit, we evaluated the daily behaviors of wild-type cynomolgus monkeys within their home cages and experimental environments and compared these observations with the behaviors exhibited by cynomolgus monkeys possessing mutations in the MECP2 gene as a disease model of Rett syndrome (RTT). MonKit was used to assess motor function, stereotyped behaviors, and depressive phenotypes, with the outcomes compared with human manual detection. MonKit established consistent criteria for identifying behavior in NHPs with high accuracy and efficiency, thus providing a novel and comprehensive tool for assessing phenotypic behavior in monkeys.

Transcranial magnetic stimulation in non-human primates: A systematic review

Transcranial magnetic stimulation (TMS) is widely employed as a tool to investigate and treat brain diseases. However, little is known about the direct effects of TMS on the brain. Non-human primates (NHPs) are a valuable translational model to investigate how TMS affects brain circuits given their neurophysiological similarity with humans and their capacity to perform complex tasks that approach human behavior. This systematic review aimed to identify studies using TMS in NHPs as well as to assess their methodological quality through a modified reference checklist. The results show high heterogeneity and superficiality in the studies regarding the report of the TMS parameters, which have not improved over the years. This checklist can be used for future TMS studies with NHPs to ensure transparency and critical appraisal. The use of the checklist would improve methodological soundness and interpretation of the studies, facilitating the translation of the findings to humans. The review also discusses how advancements in the field can elucidate the effects of TMS in the brain.

Non-human primate models and systems for gait and neurophysiological analysis

Brain-computer interfaces (BCIs) have garnered extensive interest and become a groundbreaking technology to restore movement, tactile sense, and communication in patients. Prior to their use in human subjects, clinical BCIs require rigorous validation and verification (V&V). Non-human primates (NHPs) are often considered the ultimate and widely used animal model for neuroscience studies, including BCIs V&V, due to their proximity to humans. This literature review summarizes 94 NHP gait analysis studies until 1 June, 2022, including seven BCI-oriented studies. Due to technological limitations, most of these studies used wired neural recordings to access electrophysiological data. However, wireless neural recording systems for NHPs enabled neuroscience research in humans, and many on NHP locomotion, while posing numerous technical challenges, such as signal quality, data throughout, working distance, size, and power constraint, that have yet to be overcome. Besides neurological data, motion capture (MoCap) systems are usually required in BCI and gait studies to capture locomotion kinematics. However, current studies have exclusively relied on image processing-based MoCap systems, which have insufficient accuracy (error: ≥4° and 9 mm). While the role of the motor cortex during locomotion is still unclear and worth further exploration, future BCI and gait studies require simultaneous, high-speed, accurate neurophysiological, and movement measures. Therefore, the infrared MoCap system which has high accuracy and speed, together with a high spatiotemporal resolution neural recording system, may expand the scope and improve the quality of the motor and neurophysiological analysis in NHPs.

Home-Enclosure-Based Behavioral and Wireless Neural Recording Setup for Unrestrained Rhesus Macaques

Electrophysiological studies with behaving nonhuman primates often require the separation of animals from their social group as well as partial movement restraint to perform well-controlled experiments. When the research goal per se does not mandate constraining the animals' movements, there are often still experimental needs imposed by tethered data acquisition. Recent technological advances meanwhile allow wireless neurophysiological recordings at high band-width in limited-size enclosures. Here, we demonstrate wireless neural recordings at single unit resolution from unrestrained rhesus macaques while they performed self-paced, structured visuomotor tasks on our custom-built, stand-alone touchscreen system [eXperimental Behavioral Instrument (XBI)] in their home environment. We were able to successfully characterize neural tuning to task parameters, such as visuo-spatial selectivity during movement planning and execution, as expected from existing findings obtained via setup-based neurophysiology recordings. We conclude that when movement restraint and/or a highly controlled, insulated environment are not necessary for scientific reasons, cage-based wireless neural recordings are a viable option. We propose an approach that allows the animals to engage in a self-paced manner with our XBI device, both for fully automatized training and cognitive testing, as well as neural data acquisition in their familiar environment, maintaining auditory and sometimes visual contact with their conspecifics.

Emergence of Distinct Neural Subspaces in Motor Cortical Dynamics during Volitional Adjustments of Ongoing Locomotion

The ability to modulate ongoing walking gait with precise, voluntary adjustments is what allows animals to navigate complex terrains. However, how the nervous system generates the signals to precisely control the limbs while simultaneously maintaining locomotion is poorly understood. One potential strategy is to distribute the neural activity related to these two functions into distinct cortical activity coactivation subspaces so that both may be conducted simultaneously without disruptive interference. To investigate this hypothesis, we recorded the activity of primary motor cortex in male nonhuman primates during obstacle avoidance on a treadmill. We found that the same neural population was active during both basic unobstructed locomotion and volitional obstacle avoidance movements. We identified the neural modes spanning the subspace of the low-dimensional dynamics in primary motor cortex and found a subspace that consistently maintains the same cyclic activity throughout obstacle stepping, despite large changes in the movement itself. All of the variance corresponding to this large change in movement during the obstacle avoidance was confined to its own distinct subspace. Furthermore, neural decoders built for ongoing locomotion did not generalize to decoding obstacle avoidance during locomotion. Our findings suggest that separate underlying subspaces emerge during complex locomotion that coordinates ongoing locomotor-related neural dynamics with volitional gait adjustments. These findings may have important implications for the development of brain-machine interfaces.SIGNIFICANCE STATEMENT Locomotion and precise, goal-directed movements are two distinct movement modalities with known differing requirements of motor cortical input. Previous studies have characterized the cortical activity during obstacle avoidance while walking in rodents and felines, but, to date, no such studies have been completed in primates. Additionally, in any animal model, it is unknown how these two movements are represented in primary motor cortex (M1) low-dimensional dynamics when both activities are performed at the same time, such as during obstacle avoidance. We developed a novel obstacle avoidance paradigm in freely moving nonhuman primates and discovered that the rhythmic locomotion-related dynamics and the voluntary, gait-adjustment movement separate into distinct subspaces in M1 cortical activity. Our analysis of decoding generalization may also have important implications for the development of brain-machine interfaces.

A large-scale neural network training framework for generalized estimation of single-trial population dynamics

Achieving state-of-the-art performance with deep neural population dynamics models requires extensive hyperparameter tuning for each dataset. AutoLFADS is a model-tuning framework that automatically produces high-performing autoencoding models on data from a variety of brain areas and tasks, without behavioral or task information. We demonstrate its broad applicability on several rhesus macaque datasets: from motor cortex during free-paced reaching, somatosensory cortex during reaching with perturbations, and dorsomedial frontal cortex during a cognitive timing task.

Complementary Roles of Primate Dorsal Premotor and Pre-Supplementary Motor Areas to the Control of Motor Sequences

We are able to temporally organize multiple movements in a purposeful manner in everyday life. Both the dorsal premotor (PMd) area and pre-supplementary motor area (pre-SMA) are known to be involved in the performance of motor sequences. However, it is unclear how each area differentially contributes to controlling multiple motor sequences. To address this issue, we recorded single-unit activity in both areas while monkeys (one male, one female) performed sixteen motor sequences. Each sequence comprised either a series of two identical movements (repetition) or two different movements (nonrepetition). The sequence was initially instructed with visual signals but had to be remembered thereafter. Here, we showed that the activity of single neurons in both areas transitioned from reactive- to predictive encoding while motor sequences were memorized. In the memory-guided trials, in particular, the activity of PMd cells preferentially represented the second movement (2M) in the sequence leading to a reward generally regardless of the first movement (1M). Such activity frequently began even before the 1M in a prospective manner, and was enhanced in nonrepetition sequences. Behaviorally, a lack of the activity enhancement often resulted in premature execution of the 2M. In contrast, cells in pre-SMA instantiated particular sequences of actions by coordinating switching or nonswitching movements in sequence. Our findings suggest that PMd and pre-SMA play complementary roles within behavioral contexts: PMd preferentially controls the movement that leads to a reward rather than the sequence per se, whereas pre-SMA coordinates all elements in a sequence by integrating temporal orders of multiple movements.SIGNIFICANCE STATEMENT Although both dorsal premotor (PMd) area and pre-supplementary motor area (pre-SMA) are involved in the control of motor sequences, it is not clear how these two areas contribute to coordination of sequential movements differently. To address this issue, we directly compared neuronal activity in the two areas recorded while monkeys memorized and performed multiple motor sequences. Our findings suggest that PMd preferentially controls the final action that ultimately leads to a reward in a prospective manner, whereas the pre-SMA coordinates switching among multiple actions within the context of the sequence. Our findings are of significance to understand the distinct roles for motor-related areas in the planning and executing motor sequences and the pathophysiology of apraxia and/or Parkinson's diseases that disables skilled motor actions.

Reducing behavioral dimensions to study brain-environment interactions

Movement vigor provides a window on action valuation. But what is vigor, and how to measure it in the first place? Strikingly, many different co-varying vigor-related metrics can be found in the literature. I believe this is because vigor, just like the neural circuits that determine it, is an integrated, low-dimensional parameter. As such, it can only be roughly estimated.

Parietal maps of visual signals for bodily action planning

The posterior parietal cortex (PPC) has long been understood as a high-level integrative station for computing motor commands for the body based on sensory (i.e., mostly tactile and visual) input from the outside world. In the last decade, accumulating evidence has shown that the parietal areas not only extract the pragmatic features of manipulable objects, but also subserve sensorimotor processing of others’ actions. A paradigmatic case is that of the anterior intraparietal area (AIP), which encodes the identity of observed manipulative actions that afford potential motor actions the observer could perform in response to them. On these bases, we propose an AIP manipulative action-based template of the general planning functions of the PPC and review existing evidence supporting the extension of this model to other PPC regions and to a wider set of actions: defensive and locomotor actions. In our model, a hallmark of PPC functioning is the processing of information about the physical and social world to encode potential bodily actions appropriate for the current context. We further extend the model to actions performed with man-made objects (e.g., tools) and artifacts, because they become integral parts of the subject’s body schema and motor repertoire. Finally, we conclude that existing evidence supports a generally conserved neural circuitry that transforms integrated sensory signals into the variety of bodily actions that primates are capable of preparing and performing to interact with their physical and social world.

A markerless platform for ambulatory systems neuroscience

Motor systems neuroscience seeks to understand how the brain controls movement. To minimize confounding variables, large-animal studies typically constrain body movement from areas not under observation, ensuring consistent, repeatable behaviors. Such studies have fueled decades of research, but they may be artificially limiting the richness of neural data observed, preventing generalization to more natural movements and settings. Neuroscience studies of unconstrained movement would capture a greater range of behavior and a more complete view of neuronal activity, but instrumenting an experimental rig suitable for large animals presents substantial engineering challenges. Here, we present a markerless, full-body motion tracking and synchronized wireless neural electrophysiology platform for large, ambulatory animals. Composed of four depth (RGB-D) cameras that provide a 360° view of a 4.5-square-meters enclosed area, this system is designed to record a diverse range of neuroethologically relevant behaviors. This platform also allows for the simultaneous acquisition of hundreds of wireless neural recording channels in multiple brain regions. As behavioral and neuronal data are generated at rates below 200 megabytes per second, a single desktop can facilitate hours of continuous recording. This setup is designed for systems neuroscience and neuroengineering research, where synchronized kinematic behavior and neural data are the foundation for investigation. By enabling the study of previously unexplored movement tasks, this system can generate insights into the functioning of the mammalian motor system and provide a platform to develop brain-machine interfaces for unconstrained applications.

Multi-scale neural decoding and analysis

Objective.

Complex spatiotemporal neural activity encodes rich information related to behavior and cognition. Conventional research has focused on neural activity acquired using one of many different measurement modalities, each of which provides useful but incomplete assessment of the neural code. Multi-modal techniques can overcome tradeoffs in the spatial and temporal resolution of a single modality to reveal deeper and more comprehensive understanding of system-level neural mechanisms. Uncovering multi-scale dynamics is essential for a mechanistic understanding of brain function and for harnessing neuroscientific insights to develop more effective clinical treatment.

Approach.

We discuss conventional methodologies used for characterizing neural activity at different scales and review contemporary examples of how these approaches have been combined. Then we present our case for integrating activity across multiple scales to benefit from the combined strengths of each approach and elucidate a more holistic understanding of neural processes.

Main results.

We examine various combinations of neural activity at different scales and analytical techniques that can be used to integrate or illuminate information across scales, as well the technologies that enable such exciting studies. We conclude with challenges facing future multi-scale studies, and a discussion of the power and potential of these approaches.

Significance.

This roadmap will lead the readers toward a broad range of multi-scale neural decoding techniques and their benefits over single-modality analyses. This Review article highlights the importance of multi-scale analyses for systematically interrogating complex spatiotemporal mechanisms underlying cognition and behavior.

Levels of naturalism in social neuroscience research

In order to understand ecologically meaningful social behaviors and their neural substrates in humans and other animals, researchers have been using a variety of social stimuli in the laboratory with a goal of extracting specific processes in real-life scenarios. However, certain stimuli may not be sufficiently effective at evoking typical social behaviors and neural responses. Here, we review empirical research employing different types of social stimuli by classifying them into five levels of naturalism. We describe the advantages and limitations while providing selected example studies for each level. We emphasize the important trade-off between experimental control and ecological validity across the five levels of naturalism. Taking advantage of newly emerging tools, such as real-time videos, virtual avatars, and wireless neural sampling techniques, researchers are now more than ever able to adopt social stimuli at a higher level of naturalism to better capture the dynamics and contingency of real-life social interaction.

Supporting generalization in non-human primate behavior by tapping into structural knowledge: Examples from sensorimotor mappings, inference, and decision-making

The complex behaviors we ultimately wish to understand are far from those currently used in systems neuroscience laboratories. A salient difference are the closed loops between action and perception prominently present in natural but not laboratory behaviors. The framework of reinforcement learning and control naturally wades across action and perception, and thus is poised to inform the neurosciences of tomorrow, not only from a data analyses and modeling framework, but also in guiding experimental design. We argue that this theoretical framework emphasizes active sensing, dynamical planning, and the leveraging of structural regularities as key operations for intelligent behavior within uncertain, time-varying environments. Similarly, we argue that we may study natural task strategies and their neural circuits without over-training animals when the tasks we use tap into our animal's structural knowledge. As proof-of-principle, we teach animals to navigate through a virtual environment - i.e., explore a well-defined and repetitive structure governed by the laws of physics - using a joystick. Once these animals have learned to 'drive', without further training they naturally (i) show zero- or one-shot learning of novel sensorimotor contingencies, (ii) infer the evolving path of dynamically changing latent variables, and (iii) make decisions consistent with maximizing reward rate. Such task designs allow for the study of flexible and generalizable, yet controlled, behaviors. In turn, they allow for the exploitation of pillars of intelligence - flexibility, prediction, and generalization -, properties whose neural underpinning have remained elusive.

From the field to the lab and back: neuroethology of primate social behavior

Social mammals with more numerous and stronger social relationships live longer, healthier lives. Despite the established importance of social relationships, our understanding of the neurobiological mechanisms by which they are pursued, formed, and maintained in primates remains largely confined to highly controlled laboratory settings which do not allow natural, dynamic social interactions to unfold. In this review, we argue that the neurobiological study of primate social behavior would benefit from adopting a neuroethological approach, that is, a perspective grounded in natural, species-typical behavior, with careful selection of animal models according to the scientific question at hand. We highlight macaques and marmosets as key animal models for human social behavior and summarize recent findings in the social domain for both species. We then review pioneering studies of dynamic social behaviors in small animals, which can inspire studies in larger primates where the technological landscape is now ripe for an ethological overhaul.

The role of eye movements in manual interception: A mini-review

When we catch a moving object in mid-flight, our eyes and hands are directed toward the object. Yet, the functional role of eye movements in guiding interceptive hand movements is not yet well understood. This review synthesizes emergent views on the importance of eye movements during manual interception with an emphasis on laboratory studies published since 2015. We discuss the role of eye movements in forming visual predictions about a moving object, and for enhancing the accuracy of interceptive hand movements through feedforward (extraretinal) and feedback (retinal) signals. We conclude by proposing a framework that defines the role of human eye movements for manual interception accuracy as a function of visual certainty and object motion predictability.

Primate extrastriate cortical area MST: a gateway between sensation and cognition

Primate visual cortex consists of dozens of distinct brain areas, each providing a highly specialized component to the sophisticated task of encoding the incoming sensory information and creating a representation of our visual environment that underlies our perception and action. One such area is the medial superior temporal cortex (MST), a motion-sensitive, direction-selective part of the primate visual cortex. It receives most of its input from the middle temporal (MT) area, but MST cells have larger receptive fields and respond to more complex motion patterns. The finding that MST cells are tuned for optic flow patterns has led to the suggestion that the area plays an important role in the perception of self-motion. This hypothesis has received further support from studies showing that some MST cells also respond selectively to vestibular cues. Furthermore, the area is part of a network that controls the planning and execution of smooth pursuit eye movements and its activity is modulated by cognitive factors, such as attention and working memory. This review of more than 90 studies focuses on providing clarity of the heterogeneous findings on MST in the macaque cortex and its putative homolog in the human cortex. From this analysis of the unique anatomical and functional position in the hierarchy of areas and processing steps in primate visual cortex, MST emerges as a gateway between perception, cognition, and action planning. Given this pivotal role, this area represents an ideal model system for the transition from sensation to cognition.

Macaque monkeys learn and perform a non-match-to-goal task using an automated home cage training procedure

In neurophysiology, nonhuman primates represent an important model for studying the brain. Typically, monkeys are moved from their home cage to an experimental room daily, where they sit in a primate chair and interact with electronic devices. Refining this procedure would make the researchers' work easier and improve the animals' welfare. To address this issue, we used home-cage training to train two macaque monkeys in a non-match-to-goal task, where each trial required a switch from the choice made in the previous trial to obtain a reward. The monkeys were tested in two versions of the task, one in which they acted as the agent in every trial and one in which some trials were completed by a "ghost agent". We evaluated their involvement in terms of their performance and their interaction with the apparatus. Both monkeys were able to maintain a constant involvement in the task with good, stable performance within sessions in both versions of the task. Our study confirms the feasibility of home-cage training and demonstrates that even with challenging tasks, monkeys can complete a large number of trials at a high performance level, which is a prerequisite for electrophysiological studies of monkey behavior.

MacaquePose: A Novel "In the Wild" Macaque Monkey Pose Dataset for Markerless Motion Capture

Video-based markerless motion capture permits quantification of an animal's pose and motion, with a high spatiotemporal resolution in a naturalistic context, and is a powerful tool for analyzing the relationship between the animal's behaviors and its brain functions. Macaque monkeys are excellent non-human primate models, especially for studying neuroscience. Due to the lack of a dataset allowing training of a deep neural network for the macaque's markerless motion capture in the naturalistic context, it has been challenging to apply this technology for macaques-based studies. In this study, we created MacaquePose, a novel open dataset with manually labeled body part positions (keypoints) for macaques in naturalistic scenes, consisting of >13,000 images. We also validated the application of the dataset by training and evaluating an artificial neural network with the dataset. The results indicated that the keypoint estimation performance of the trained network was close to that of a human-level. The dataset will be instrumental to train/test the neural networks for markerless motion capture of the macaques and developments of the algorithms for the networks, contributing establishment of an innovative platform for behavior analysis for non-human primates for neuroscience and medicine, as well as other fields using macaques as a model organism.