Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

A searchable image resource of Drosophila GAL4 driver expression patterns with single neuron resolution

Precise, repeatable genetic access to specific neurons via GAL4/UAS and related methods is a key advantage of Drosophila neuroscience. Neuronal targeting is typically documented using light microscopy of full GAL4 expression patterns, which generally lack the single-cell resolution required for reliable cell type identification. Here, we use stochastic GAL4 labeling with the MultiColor FlpOut approach to generate cellular resolution confocal images at large scale. We are releasing aligned images of 74,000 such adult central nervous systems. An anticipated use of this resource is to bridge the gap between neurons identified by electron or light microscopy. Identifying individual neurons that make up each GAL4 expression pattern improves the prediction of split-GAL4 combinations targeting particular neurons. To this end, we have made the images searchable on the NeuronBridge website. We demonstrate the potential of NeuronBridge to rapidly and effectively identify neuron matches based on morphology across imaging modalities and datasets.

Automated reconstruction of whole-embryo cell lineages by learning from sparse annotations

We present a method to automatically identify and track nuclei in time-lapse microscopy recordings of entire developing embryos. The method combines deep learning and global optimization. On a mouse dataset, it reconstructs 75.8% of cell lineages spanning 1 h, as compared to 31.8% for the competing method. Our approach improves understanding of where and when cell fate decisions are made in developing embryos, tissues, and organs.

A connectome and analysis of the adult Drosophila central brain

The neural circuits responsible for animal behavior remain largely unknown. We summarize new methods and present the circuitry of a large fraction of the brain of the fruit fly Drosophila melanogaster. Improved methods include new procedures to prepare, image, align, segment, find synapses in, and proofread such large data sets. We define cell types, refine computational compartments, and provide an exhaustive atlas of cell examples and types, many of them novel. We provide detailed circuits consisting of neurons and their chemical synapses for most of the central brain. We make the data public and simplify access, reducing the effort needed to answer circuit questions, and provide procedures linking the neurons defined by our analysis with genetic reagents. Biologically, we examine distributions of connection strengths, neural motifs on different scales, electrical consequences of compartmentalization, and evidence that maximizing packing density is an important criterion in the evolution of the fly's brain.

The cell adhesion protein CAR is a negative regulator of synaptic transmission

The Coxsackievirus and adenovirus receptor (CAR) is essential for normal electrical conductance in the heart, but its role in the postnatal brain is largely unknown. Using brain specific CAR knockout mice (KO), we discovered an unexpected role of CAR in neuronal communication. This includes increased basic synaptic transmission at hippocampal Schaffer collaterals, resistance to fatigue, and enhanced long-term potentiation. Spontaneous neurotransmitter release and speed of endocytosis are increased in KOs, accompanied by increased expression of the exocytosis associated calcium sensor synaptotagmin 2. Using proximity proteomics and binding studies, we link CAR to the exocytosis machinery as it associates with syntenin and synaptobrevin/VAMP2 at the synapse. Increased synaptic function does not cause adverse effects in KO mice, as behavior and learning are unaffected. Thus, unlike the connexin-dependent suppression of atrioventricular conduction in the cardiac knockout, communication in the CAR deficient brain is improved, suggesting a role for CAR in presynaptic processes.

Automated detection and quantification of single RNAs at cellular resolution in zebrafish embryos

Analysis of differential gene expression is crucial for the study of cell fate and behavior during embryonic development. However, automated methods for the sensitive detection and quantification of RNAs at cellular resolution in embryos are lacking. With the advent of single-molecule fluorescence in situ hybridization (smFISH), gene expression can be analyzed at single-molecule resolution. However, the limited availability of protocols for smFISH in embryos and the lack of efficient image analysis pipelines have hampered quantification at the (sub)cellular level in complex samples such as tissues and embryos. Here, we present a protocol for smFISH on zebrafish embryo sections in combination with an image analysis pipeline for automated transcript detection and cell segmentation. We use this strategy to quantify gene expression differences between different cell types and identify differences in subcellular transcript localization between genes. The combination of our smFISH protocol and custom-made, freely available, analysis pipeline will enable researchers to fully exploit the benefits of quantitative transcript analysis at cellular and subcellular resolution in tissues and embryos.

Improving deformable surface meshes through omni-directional displacements and MRFs

Deformable surface models are often represented as triangular meshes in image segmentation applications. For a fast and easily regularized deformation onto the target object boundary, the vertices of the mesh are commonly moved along line segments (typically surface normals). However, in case of high mesh curvature, these lines may intersect with the target boundary at "non-corresponding" positions, or even not at all. Consequently, certain deformations cannot be achieved. We propose an approach that allows each vertex to move not only along a line segment, but within a surrounding sphere. We achieve globally regularized deformations via Markov Random Field optimization. We demonstrate the potential of our approach with experiments on synthetic data, as well as an evaluation on 2 x 106 coronoid processes of the mandible in Cone-Beam CTs, and 56 coccyxes (tailbones) in low-resolution CTs.

Automatic extraction of mandibular nerve and bone from cone-beam CT data

The exact localization of the mandibular nerve with respect to the bone is important for applications in dental implantology and maxillofacial surgery. Cone beam computed tomography (CBCT), often also called digital volume tomography (DVT), is increasingly utilized in maxillofacial or dental imaging. Compared to conventional CT, however, soft tissue discrimination is worse due to a reduced dose. Thus, small structures like the alveolar nerves are even harder recognizable within the image data. We show that it is nonetheless possible to accurately reconstruct the 3D bone surface and the course of the nerve in a fully automatic fashion, with a method that is based on a combined statistical shape model of the nerve and the bone and a Dijkstra-based optimization procedure. Our method has been validated on 106 clinical datasets: the average reconstruction error for the bone is 0.5 +/- 0.1 mm, and the nerve can be detected with an average error of 1.0 +/- 0.6 mm.

An articulated statistical shape model for accurate hip joint segmentation

In this paper we propose a framework for fully automatic, robust and accurate segmentation of the human pelvis and proximal femur in CT data. We propose a composite statistical shape model of femur and pelvis with a flexible hip joint, for which we extend the common definition of statistical shape models as well as the common strategy for their adaptation. We do not analyze the joint flexibility statistically, but model it explicitly by rotational parameters describing the bent in a ball-and-socket joint. A leave-one-out evaluation on 50 CT volumes shows that image driven adaptation of our composite shape model robustly produces accurate segmentations of both proximal femur and pelvis. As a second contribution, we evaluate a fine grain multi-object segmentation method based on graph optimization. It relies on accurate initializations of femur and pelvis, which our composite shape model can generate. Simultaneous optimization of both femur and pelvis yields more accurate results than separate optimizations of each structure. Shape model adaptation and graph based optimization are embedded in a fully automatic framework.