BrainGazer--visual queries for neurobiology research

A Visual Interface for Exploring Hypotheses About Neural Circuits

One of the fundamental problems in neurobiological research is to understand how neural circuits generate behaviors in response to sensory stimuli. Elucidating such neural circuits requires anatomical and functional information about the neurons that are active during the processing of the sensory information and generation of the respective response, as well as an identification of the connections between these neurons. With modern imaging techniques, both morphological properties of individual neurons as well as functional information related to sensory processing, information integration and behavior can be obtained. Given the resulting information, neurobiologists are faced with the task of identifying the anatomical structures down to individual neurons that are linked to the studied behavior and the processing of the respective sensory stimuli. Here, we present a novel interactive tool that assists neurobiologists in the aforementioned tasks by allowing them to extract hypothetical neural circuits constrained by anatomical and functional data. Our approach is based on two types of structural data: brain regions that are anatomically or functionally defined, and morphologies of individual neurons. Both types of structural data are interlinked and augmented with additional information. The presented tool allows the expert user to identify neurons using Boolean queries. The interactive formulation of these queries is supported by linked views, using, among other things, two novel 2D abstractions of neural circuits. The approach was validated in two case studies investigating the neural basis of vision-based behavioral responses in zebrafish larvae. Despite this particular application, we believe that the presented tool will be of general interest for exploring hypotheses about neural circuits in other species, genera and taxa.

TimeTubesX: A Query-Driven Visual Exploration of Observable, Photometric, and Polarimetric Behaviors of Blazars

Blazars are celestial bodies of high interest to astronomers. In particular, through the analysis of photometric and polarimetric observations of blazars, astronomers aim to understand the physics of the blazar's relativistic jet. However, it is challenging to recognize correlations and time variations of the observed polarization, intensity, and color of the emitted light. In our prior study, we proposed TimeTubes to visualize a blazar dataset as a 3D volumetric tube. In this paper, we build primarily on the TimeTubes representation of blazar datasets to present a new visual analytics environment named TimeTubesX, into which we have integrated sophisticated feature and pattern detection techniques for effective location of observable and recurring time variation patterns in long-term, multi-dimensional datasets. Automatic feature extraction detects time intervals corresponding to well-known blazar behaviors. Dynamic visual querying allows users to search long-term observations for time intervals similar to a time interval of interest (query-by-example) or a sketch of temporal patterns (query-by-sketch). Users are also allowed to build up another visual query guided by the time interval of interest found in the previous process and refine the results. We demonstrate how TimeTubesX has been used successfully by domain experts for the detailed analysis of blazar datasets and report on the results.

SynCoPa: Visualizing Connectivity Paths and Synapses Over Detailed Morphologies

Brain complexity has traditionally fomented the division of neuroscience into somehow separated compartments; the coexistence of the anatomical, physiological, and connectomics points of view is just a paradigmatic example of this situation. However, there are times when it is important to combine some of these standpoints for getting a global picture, like for fully analyzing the morphological and topological features of a specific neuronal circuit. Within this framework, this article presents SynCoPa, a tool designed for bridging gaps among representations by providing techniques that allow combining detailed morphological neuron representations with the visualization of neuron interconnections at the synapse level. SynCoPa has been conceived for the interactive exploration and analysis of the connectivity elements and paths of simple to medium complexity neuronal circuits at the connectome level. This has been done by providing visual metaphors for synapses and interconnection paths, in combination with the representation of detailed neuron morphologies. SynCoPa could be helpful, for example, for establishing or confirming a hypothesis about the spatial distributions of synapses, or for answering questions about the way neurons establish connections or the relationships between connectivity and morphological features. Last, SynCoPa is easily extendable to include functional data provided, for example, by any of the morphologically-detailed simulators available nowadays, such as Neuron and Arbor, for providing a deep insight into the circuits features prior to simulating it, in particular any analysis where it is important to combine morphology, network topology, and physiology.

Ray Tracing Generalized Tube Primitives: Method and Applications

We present a general high-performance technique for ray tracing generalized tube primitives. Our technique efficiently supports tube primitives with fixed and varying radii, general acyclic graph structures with bifurcations, and correct transparency with interior surface removal. Such tube primitives are widely used in scientific visualization to represent diffusion tensor imaging tractographies, neuron morphologies, and scalar or vector fields of 3D flow. We implement our approach within the OSPRay ray tracing framework, and evaluate it on a range of interactive visualization use cases of fixed- and varying-radius streamlines, pathlines, complex neuron morphologies, and brain tractographies. Our proposed approach provides interactive, high-quality rendering, with low memory overhead.

Towards Advanced Interactive Visualization for Virtual Atlases

An atlas is generally defined as a bound collection of tables, charts or illustrations describing a phenomenon. In an anatomical atlas for example, a collection of representative illustrations and text describes anatomy for the purpose of communicating anatomical knowledge. The atlas serves as reference frame for comparing and integrating data from different sources by spatially or semantically relating collections of drawings, imaging data, and/or text. In the field of medical image processing, atlas information is often constructed from a collection of regions of interest, which are based on medical images that are annotated by domain experts. Such an atlas may be employed, for example, for automatic segmentation of medical imaging data. The combination of interactive visualization techniques with atlas information opens up new possibilities for content creation, curation, and navigation in virtual atlases. With interactive visualization of atlas information, students are able to inspect and explore anatomical atlases in ways that were not possible with the traditional method of presenting anatomical atlases in book format, such as viewing the illustrations from other viewpoints. With advanced interaction techniques, it becomes possible to query the data that forms the basis for the atlas, thus empowering researchers to access a wealth of information in new ways. So far, atlas-based visualization has been employed mainly for medical education, as well as biological research. In this survey, we provide an overview of current digital biomedical atlas tasks and applications and summarize relevant visualization techniques. We discuss recent approaches for providing next-generation visual interfaces to navigate atlas data that go beyond common text-based search and hierarchical lists. Finally, we reflect on open challenges and opportunities for the next steps in interactive atlas visualization.

Culling for Extreme-Scale Segmentation Volumes: A Hybrid Deterministic and Probabilistic Approach

With the rapid increase in raw volume data sizes, such as terabyte-sized microscopy volumes, the corresponding segmentation label volumes have become extremely large as well. We focus on integer label data, whose efficient representation in memory, as well as fast random data access, pose an even greater challenge than the raw image data. Often, it is crucial to be able to rapidly identify which segments are located where, whether for empty space skipping for fast rendering, or for spatial proximity queries. We refer to this process as culling. In order to enable efficient culling of millions of labeled segments, we present a novel hybrid approach that combines deterministic and probabilistic representations of label data in a data-adaptive hierarchical data structure that we call the label list tree. In each node, we adaptively encode label data using either a probabilistic constant-time access representation for fast conservative culling, or a deterministic logarithmic-time access representation for exact queries. We choose the best data structures for representing the labels of each spatial region while building the label list tree. At run time, we further employ a novel query-adaptive culling strategy. While filtering a query down the tree, we prune it successively, and in each node adaptively select the representation that is best suited for evaluating the pruned query, depending on its size. We show an analysis of the efficiency of our approach with several large data sets from connectomics, including a brain scan with more than 13 million labeled segments, and compare our method to conventional culling approaches. Our approach achieves significant reductions in storage size as well as faster query times.

A Statistically Representative Atlas for Mapping Neuronal Circuits in the Drosophila Adult Brain

Imaging the expression patterns of reporter constructs is a powerful tool to dissect the neuronal circuits of perception and behavior in the adult brain of Drosophila, one of the major models for studying brain functions. To date, several Drosophila brain templates and digital atlases have been built to automatically analyze and compare collections of expression pattern images. However, there has been no systematic comparison of performances between alternative atlasing strategies and registration algorithms. Here, we objectively evaluated the performance of different strategies for building adult Drosophila brain templates and atlases. In addition, we used state-of-the-art registration algorithms to generate a new group-wise inter-sex atlas. Our results highlight the benefit of statistical atlases over individual ones and show that the newly proposed inter-sex atlas outperformed existing solutions for automated registration and annotation of expression patterns. Over 3,000 images from the Janelia Farm FlyLight collection were registered using the proposed strategy. These registered expression patterns can be searched and compared with a new version of the BrainBaseWeb system and BrainGazer software. We illustrate the validity of our methodology and brain atlas with registration-based predictions of expression patterns in a subset of clock neurons. The described registration framework should benefit to brain studies in Drosophila and other insect species.

Abstractocyte: A Visual Tool for Exploring Nanoscale Astroglial Cells

This paper presents Abstractocyte, a system for the visual analysis of astrocytes and their relation to neurons, in nanoscale volumes of brain tissue. Astrocytes are glial cells, i.e., non-neuronal cells that support neurons and the nervous system. The study of astrocytes has immense potential for understanding brain function. However, their complex and widely-branching structure requires high-resolution electron microscopy imaging and makes visualization and analysis challenging. Furthermore, the structure and function of astrocytes is very different from neurons, and therefore requires the development of new visualization and analysis tools. With Abstractocyte, biologists can explore the morphology of astrocytes using various visual abstraction levels, while simultaneously analyzing neighboring neurons and their connectivity. We define a novel, conceptual 2D abstraction space for jointly visualizing astrocytes and neurons. Neuroscientists can choose a specific joint visualization as a point in this space. Interactively moving this point allows them to smoothly transition between different abstraction levels in an intuitive manner. In contrast to simply switching between different visualizations, this preserves the visual context and correlations throughout the transition. Users can smoothly navigate from concrete, highly-detailed 3D views to simplified and abstracted 2D views. In addition to investigating astrocytes, neurons, and their relationships, we enable the interactive analysis of the distribution of glycogen, which is of high importance to neuroscientists. We describe the design of Abstractocyte, and present three case studies in which neuroscientists have successfully used our system to assess astrocytic coverage of synapses, glycogen distribution in relation to synapses, and astrocytic-mitochondria coverage.

Learning from connectomics on the fly

Parallels between invertebrates and vertebrates in nervous system development, organisation and circuits are powerful reasons to use insects to study the mechanistic basis of behaviour. The last few years have seen the generation in Drosophila melanogaster of very large light microscopy data sets, genetic driver lines and tools to report or manipulate neural activity. These resources in conjunction with computational tools are enabling large scale characterisation of neuronal types and their functional properties. These are complemented by 3D electron microscopy, providing synaptic resolution data. A whole brain connectome of the fly larva is approaching completion based on manual reconstruction of electron-microscopy data. An adult whole brain dataset is already publicly available and focussed reconstruction is under way, but its 40× greater volume would require ∼500-5000 person-years of manual labour. Nevertheless rapid technical improvements in imaging and especially automated segmentation will likely deliver a complete adult connectome in the next 5 years. To enhance our understanding of the circuit basis of behaviour, light and electron microscopy outputs must be integrated with functional and physiological information into comprehensive databases. We review presently available data, tools and opportunities in Drosophila. We then consider the limits and potential of future progress and how this may impact neuroscience in rich model systems provided by larger insects and vertebrates.

Brain Modulyzer: Interactive Visual Analysis of Functional Brain Connectivity

We present Brain Modulyzer, an interactive visual exploration tool for functional magnetic resonance imaging (fMRI) brain scans, aimed at analyzing the correlation between different brain regions when resting or when performing mental tasks. Brain Modulyzer combines multiple coordinated views-such as heat maps, node link diagrams and anatomical views-using brushing and linking to provide an anatomical context for brain connectivity data. Integrating methods from graph theory and analysis, e.g., community detection and derived graph measures, makes it possible to explore the modular and hierarchical organization of functional brain networks. Providing immediate feedback by displaying analysis results instantaneously while changing parameters gives neuroscientists a powerful means to comprehend complex brain structure more effectively and efficiently and supports forming hypotheses that can then be validated via statistical analysis. To demonstrate the utility of our tool, we present two case studies-exploring progressive supranuclear palsy, as well as memory encoding and retrieval.

Automatic Segmentation of Drosophila Neural Compartments Using GAL4 Expression Data Reveals Novel Visual Pathways

Identifying distinct anatomical structures within the brain and developing genetic tools to target them are fundamental steps for understanding brain function. We hypothesize that enhancer expression patterns can be used to automatically identify functional units such as neuropils and fiber tracts. We used two recent, genome-scale Drosophila GAL4 libraries and associated confocal image datasets to segment large brain regions into smaller subvolumes. Our results (available at https://strawlab.org/braincode) support this hypothesis because regions with well-known anatomy, namely the antennal lobes and central complex, were automatically segmented into familiar compartments. The basis for the structural assignment is clustering of voxels based on patterns of enhancer expression. These initial clusters are agglomerated to make hierarchical predictions of structure. We applied the algorithm to central brain regions receiving input from the optic lobes. Based on the automated segmentation and manual validation, we can identify and provide promising driver lines for 11 previously identified and 14 novel types of visual projection neurons and their associated optic glomeruli. The same strategy can be used in other brain regions and likely other species, including vertebrates.

•

Genome-scale enhancer expression patterns can be used to predict brain structure

•

Automated clustering of images finds known structures such as olfactory glomeruli

•

Results identify GAL4 lines with strong expression in the predicted structures

•

We validate novel predictions to reveal previously undescribed optic glomeruli

PyramidalExplorer: A New Interactive Tool to Explore Morpho-Functional Relations of Human Pyramidal Neurons

This work presents PyramidalExplorer, a new tool to interactively explore and reveal the detailed organization of the microanatomy of pyramidal neurons with functionally related models. It consists of a set of functionalities that allow possible regional differences in the pyramidal cell architecture to be interactively discovered by combining quantitative morphological information about the structure of the cell with implemented functional models. The key contribution of this tool is the morpho-functional oriented design that allows the user to navigate within the 3D dataset, filter and perform Content-Based Retrieval operations. As a case study, we present a human pyramidal neuron with over 9000 dendritic spines in its apical and basal dendritic trees. Using PyramidalExplorer, we were able to find unexpected differential morphological attributes of dendritic spines in particular compartments of the neuron, revealing new aspects of the morpho-functional organization of the pyramidal neuron.

NeuroBlocks--Visual Tracking of Segmentation and Proofreading for Large Connectomics Projects

In the field of connectomics, neuroscientists acquire electron microscopy volumes at nanometer resolution in order to reconstruct a detailed wiring diagram of the neurons in the brain. The resulting image volumes, which often are hundreds of terabytes in size, need to be segmented to identify cell boundaries, synapses, and important cell organelles. However, the segmentation process of a single volume is very complex, time-intensive, and usually performed using a diverse set of tools and many users. To tackle the associated challenges, this paper presents NeuroBlocks, which is a novel visualization system for tracking the state, progress, and evolution of very large volumetric segmentation data in neuroscience. NeuroBlocks is a multi-user web-based application that seamlessly integrates the diverse set of tools that neuroscientists currently use for manual and semi-automatic segmentation, proofreading, visualization, and analysis. NeuroBlocks is the first system that integrates this heterogeneous tool set, providing crucial support for the management, provenance, accountability, and auditing of large-scale segmentations. We describe the design of NeuroBlocks, starting with an analysis of the domain-specific tasks, their inherent challenges, and our subsequent task abstraction and visual representation. We demonstrate the utility of our design based on two case studies that focus on different user roles and their respective requirements for performing and tracking the progress of segmentation and proofreading in a large real-world connectomics project.

Evolving spatio-temporal data machines based on the NeuCube neuromorphic framework: Design methodology and selected applications

The paper describes a new type of evolving connectionist systems (ECOS) called evolving spatio-temporal data machines based on neuromorphic, brain-like information processing principles (eSTDM). These are multi-modular computer systems designed to deal with large and fast spatio/spectro temporal data using spiking neural networks (SNN) as major processing modules. ECOS and eSTDM in particular can learn incrementally from data streams, can include 'on the fly' new input variables, new output class labels or regression outputs, can continuously adapt their structure and functionality, can be visualised and interpreted for new knowledge discovery and for a better understanding of the data and the processes that generated it. eSTDM can be used for early event prediction due to the ability of the SNN to spike early, before whole input vectors (they were trained on) are presented. A framework for building eSTDM called NeuCube along with a design methodology for building eSTDM using this is presented. The implementation of this framework in MATLAB, Java, and PyNN (Python) is presented. The latter facilitates the use of neuromorphic hardware platforms to run the eSTDM. Selected examples are given of eSTDM for pattern recognition and early event prediction on EEG data, fMRI data, multisensory seismic data, ecological data, climate data, audio-visual data. Future directions are discussed, including extension of the NeuCube framework for building neurogenetic eSTDM and also new applications of eSTDM.

Temperature representation in the Drosophila brain

In Drosophila, rapid temperature changes are detected at the periphery by dedicated receptors forming a simple sensory map for hot and cold in the brain. However, flies show a host of complex innate and learned responses to temperature, indicating that they are able to extract a range of information from this simple input. Here we define the anatomical and physiological repertoire for temperature representation in the Drosophila brain. First, we use a photolabelling strategy to trace the connections that relay peripheral thermosensory information to higher brain centres, and show that they largely converge onto three target regions: the mushroom body, the lateral horn (both of which are well known centres for sensory processing) and the posterior lateral protocerebrum, a region we now define as a major site of thermosensory representation. Next, using in vivo calcium imaging, we describe the thermosensory projection neurons selectively activated by hot or cold stimuli. Fast-adapting neurons display transient ON and OFF responses and track rapid temperature shifts remarkably well, while slow-adapting cell responses better reflect the magnitude of simple thermal changes. Unexpectedly, we also find a population of broadly tuned cells that respond to both heating and cooling, and show that they are required for normal behavioural avoidance of both hot and cold in a simple two-choice temperature preference assay. Taken together, our results uncover a coordinated ensemble of neural responses to temperature in the Drosophila brain, demonstrate that a broadly tuned thermal line contributes to rapid avoidance behaviour, and illustrate how stimulus quality, temporal structure, and intensity can be extracted from a simple glomerular map at a single synaptic station.

NeuroLines: A Subway Map Metaphor for Visualizing Nanoscale Neuronal Connectivity

We present NeuroLines, a novel visualization technique designed for scalable detailed analysis of neuronal connectivity at the nanoscale level. The topology of 3D brain tissue data is abstracted into a multi-scale, relative distance-preserving subway map visualization that allows domain scientists to conduct an interactive analysis of neurons and their connectivity. Nanoscale connectomics aims at reverse-engineering the wiring of the brain. Reconstructing and analyzing the detailed connectivity of neurons and neurites (axons, dendrites) will be crucial for understanding the brain and its development and diseases. However, the enormous scale and complexity of nanoscale neuronal connectivity pose big challenges to existing visualization techniques in terms of scalability. NeuroLines offers a scalable visualization framework that can interactively render thousands of neurites, and that supports the detailed analysis of neuronal structures and their connectivity. We describe and analyze the design of NeuroLines based on two real-world use-cases of our collaborators in developmental neuroscience, and investigate its scalability to large-scale neuronal connectivity data.

Interactive visual analysis of heterogeneous cohort-study data

Medical cohort studies enable the study of medical hypotheses with many samples. Often, these studies acquire a large amount of heterogeneous data from many subjects. Usually, researchers study a specific data subset to confirm or reject specific hypotheses. A new approach enables the interactive visual exploration and analysis of such data, helping to generate and validate hypotheses. A data-cube-based model handles partially overlapping data subsets during the interactive visualization. This model enables seamless integration of the heterogeneous data and the linking of spatial and nonspatial views of the data. Researchers implemented this model in a prototype application and used it to analyze data acquired in a cohort study on cognitive aging. Case studies employed the prototype to study aspects of brain connectivity, demonstrating the model's potential and flexibility.

ConnectomeExplorer: query-guided visual analysis of large volumetric neuroscience data

This paper presents ConnectomeExplorer, an application for the interactive exploration and query-guided visual analysis of large volumetric electron microscopy (EM) data sets in connectomics research. Our system incorporates a knowledge-based query algebra that supports the interactive specification of dynamically evaluated queries, which enable neuroscientists to pose and answer domain-specific questions in an intuitive manner. Queries are built step by step in a visual query builder, building more complex queries from combinations of simpler queries. Our application is based on a scalable volume visualization framework that scales to multiple volumes of several teravoxels each, enabling the concurrent visualization and querying of the original EM volume, additional segmentation volumes, neuronal connectivity, and additional meta data comprising a variety of neuronal data attributes. We evaluate our application on a data set of roughly one terabyte of EM data and 750 GB of segmentation data, containing over 4,000 segmented structures and 1,000 synapses. We demonstrate typical use-case scenarios of our collaborators in neuroscience, where our system has enabled them to answer specific scientific questions using interactive querying and analysis on the full-size data for the first time.

A systematic review on the practice of evaluating visualization

We present an assessment of the state and historic development of evaluation practices as reported in papers published at the IEEE Visualization conference. Our goal is to reflect on a meta-level about evaluation in our community through a systematic understanding of the characteristics and goals of presented evaluations. For this purpose we conducted a systematic review of ten years of evaluations in the published papers using and extending a coding scheme previously established by Lam et al. [2012]. The results of our review include an overview of the most common evaluation goals in the community, how they evolved over time, and how they contrast or align to those of the IEEE Information Visualization conference. In particular, we found that evaluations specific to assessing resulting images and algorithm performance are the most prevalent (with consistently 80-90% of all papers since 1997). However, especially over the last six years there is a steady increase in evaluation methods that include participants, either by evaluating their performances and subjective feedback or by evaluating their work practices and their improved analysis and reasoning capabilities using visual tools. Up to 2010, this trend in the IEEE Visualization conference was much more pronounced than in the IEEE Information Visualization conference which only showed an increasing percentage of evaluation through user performance and experience testing. Since 2011, however, also papers in IEEE Information Visualization show such an increase of evaluations of work practices and analysis as well as reasoning using visual tools. Further, we found that generally the studies reporting requirements analyses and domain-specific work practices are too informally reported which hinders cross-comparison and lowers external validity.

MObjects--a novel method for the visualization and interactive exploration of defects in industrial XCT data

This paper describes an advanced visualization method for the analysis of defects in industrial 3D X-Ray Computed Tomography (XCT) data. We present a novel way to explore a high number of individual objects in a dataset, e.g., pores, inclusions, particles, fibers, and cracks demonstrated on the special application area of pore extraction in carbon fiber reinforced polymers (CFRP). After calculating the individual object properties volume, dimensions and shape factors, all objects are clustered into a mean object (MObject). The resulting MObject parameter space can be explored interactively. To do so, we introduce the visualization of mean object sets (MObject Sets) in a radial and a parallel arrangement. Each MObject may be split up into sub-classes by selecting a specific property, e.g., volume or shape factor, and the desired number of classes. Applying this interactive selection iteratively leads to the intended classifications and visualizations of MObjects along the selected analysis path. Hereby the given different scaling factors of the MObjects down the analysis path are visualized through a visual linking approach. Furthermore the representative MObjects are exported as volumetric datasets to serve as input for successive calculations and simulations. In the field of porosity determination in CFRP non-destructive testing practitioners use representative MObjects to improve ultrasonic calibration curves. Representative pores also serve as input for heat conduction simulations in active thermography. For a fast overview of the pore properties in a dataset we propose a local MObjects visualization in combination with a color-coded homogeneity visualization of cells. The advantages of our novel approach are demonstrated using real world CFRP specimens. The results were evaluated through a questionnaire in order to determine the practicality of the MObjects visualization as a supportive tool for domain specialists.