Improving deconvolution methods in biology through open innovation competitions: an application to the connectivity map

MOTIVATION

Do machine learning methods improve standard deconvolution techniques for gene expression data? This article uses a unique new dataset combined with an open innovation competition to evaluate a wide range of approaches developed by 294 competitors from 20 countries. The competition's objective was to address a deconvolution problem critical to analyzing genetic perturbations from the Connectivity Map. The issue consists of separating gene expression of individual genes from raw measurements obtained from gene pairs. We evaluated the outcomes using ground-truth data (direct measurements for single genes) obtained from the same samples.

RESULTS

We find that the top-ranked algorithm, based on random forest regression, beat the other methods in accuracy and reproducibility; more traditional gaussian-mixture methods performed well and tended to be faster, and the best deep learning approach yielded outcomes slightly inferior to the above methods. We anticipate researchers in the field will find the dataset and algorithms developed in this study to be a powerful research tool for benchmarking their deconvolution methods and a resource useful for multiple applications.

AVAILABILITY AND IMPLEMENTATION

The data is freely available at clue.io/data (section Contests) and the software is on GitHub at https://github.com/cmap/gene_deconvolution_challenge.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Development of a Deep Learning Algorithm for Periapical Disease Detection in Dental Radiographs

Periapical radiolucencies, which can be detected on panoramic radiographs, are one of the most common radiographic findings in dentistry and have a differential diagnosis including infections, granuloma, cysts and tumors. In this study, we seek to investigate the ability with which 24 oral and maxillofacial (OMF) surgeons assess the presence of periapical lucencies on panoramic radiographs, and we compare these findings to the performance of a predictive deep learning algorithm that we have developed using a curated data set of 2902 de-identified panoramic radiographs. The mean diagnostic positive predictive value (PPV) of OMF surgeons based on their assessment of panoramic radiographic images was 0.69(± 0.13), indicating that dentists on average falsely diagnose 31% of cases as radiolucencies. However, the mean diagnostic true positive rate (TPR) was 0.51(± 0.14), indicating that on average 49% of all radiolucencies were missed. We demonstrate that the deep learning algorithm achieves a better performance than 14 of 24 OMF surgeons within the cohort, exhibiting an average precision of 0.60(± 0.04), and an F1 score of 0.58(± 0.04) corresponding to a PPV of 0.67(± 0.05) and TPR of 0.51(± 0.05). The algorithm, trained on limited data and evaluated on clinically validated ground truth, has potential to assist OMF surgeons in detecting periapical lucencies on panoramic radiographs.

Advancing computational biology and bioinformatics research through open innovation competitions

Open data science and algorithm development competitions offer a unique avenue for rapid discovery of better computational strategies. We highlight three examples in computational biology and bioinformatics research in which the use of competitions has yielded significant performance gains over established algorithms. These include algorithms for antibody clustering, imputing gene expression data, and querying the Connectivity Map (CMap). Performance gains are evaluated quantitatively using realistic, albeit sanitized, data sets. The solutions produced through these competitions are then examined with respect to their utility and the prospects for implementation in the field. We present the decision process and competition design considerations that lead to these successful outcomes as a model for researchers who want to use competitions and non-domain crowds as collaborators to further their research.

Use of Crowd Innovation to Develop an Artificial Intelligence-Based Solution for Radiation Therapy Targeting

IMPORTANCE

Radiation therapy (RT) is a critical cancer treatment, but the existing radiation oncologist work force does not meet growing global demand. One key physician task in RT planning involves tumor segmentation for targeting, which requires substantial training and is subject to significant interobserver variation.

OBJECTIVE

To determine whether crowd innovation could be used to rapidly produce artificial intelligence (AI) solutions that replicate the accuracy of an expert radiation oncologist in segmenting lung tumors for RT targeting.

DESIGN, SETTING, AND PARTICIPANTS

We conducted a 10-week, prize-based, online, 3-phase challenge (prizes totaled $55 000). A well-curated data set, including computed tomographic (CT) scans and lung tumor segmentations generated by an expert for clinical care, was used for the contest (CT scans from 461 patients; median 157 images per scan; 77 942 images in total; 8144 images with tumor present). Contestants were provided a training set of 229 CT scans with accompanying expert contours to develop their algorithms and given feedback on their performance throughout the contest, including from the expert clinician.

MAIN OUTCOMES AND MEASURES

The AI algorithms generated by contestants were automatically scored on an independent data set that was withheld from contestants, and performance ranked using quantitative metrics that evaluated overlap of each algorithm's automated segmentations with the expert's segmentations. Performance was further benchmarked against human expert interobserver and intraobserver variation.

RESULTS

A total of 564 contestants from 62 countries registered for this challenge, and 34 (6%) submitted algorithms. The automated segmentations produced by the top 5 AI algorithms, when combined using an ensemble model, had an accuracy (Dice coefficient = 0.79) that was within the benchmark of mean interobserver variation measured between 6 human experts. For phase 1, the top 7 algorithms had average custom segmentation scores (S scores) on the holdout data set ranging from 0.15 to 0.38, and suboptimal performance using relative measures of error. The average S scores for phase 2 increased to 0.53 to 0.57, with a similar improvement in other performance metrics. In phase 3, performance of the top algorithm increased by an additional 9%. Combining the top 5 algorithms from phase 2 and phase 3 using an ensemble model, yielded an additional 9% to 12% improvement in performance with a final S score reaching 0.68.

CONCLUSIONS AND RELEVANCE

A combined crowd innovation and AI approach rapidly produced automated algorithms that replicated the skills of a highly trained physician for a critical task in radiation therapy. These AI algorithms could improve cancer care globally by transferring the skills of expert clinicians to under-resourced health care settings.

Massive Photons: An Infrared Regularization Scheme for Lattice QCD+QED

Standard methods for including electromagnetic interactions in lattice quantum chromodynamics calculations result in power-law finite-volume corrections to physical quantities. Removing these by extrapolation requires costly computations at multiple volumes. We introduce a photon mass to alternatively regulate the infrared, and rely on effective field theory to remove its unphysical effects. Electromagnetic modifications to the hadron spectrum are reliably estimated with a precision and cost comparable to conventional approaches that utilize multiple larger volumes. A significant overall cost advantage emerges when accounting for ensemble generation. The proposed method may benefit lattice calculations involving multiple charged hadrons, as well as quantum many-body computations with long-range Coulomb interactions.

Transdimensional equivalence of universal constants for Fermi gases at unitarity

I present lattice Monte Carlo calculations for a universal four-component Fermi gas confined to a finite box and to a harmonic trap in one spatial dimension. I obtain the values ξ(1D) = 0.370(4) and ξ(1D) = 0.372(1), respectively, for the Bertsch parameter, a nonperturbative universal constant defined as the (square of the) energy of the untrapped (trapped) system measured in units of the free gas energy. The Bertsch parameter obtained for the one-dimensional system is consistent to within ~1% uncertainties with the most recent numerical and experimental estimates of the analogous Bertsch parameter for a three-dimensional spin-1/2 Fermi gas at unitarity. The finding suggests the intriguing possibility that there exists a universality between two conformal theories in different dimensions. To lend support to this study, I also compute ground state energies for four and five fermions confined to a harmonic trap and demonstrate the restoration of a virial theorem in the continuum limit. The continuum few-body energies obtained are consistent with exact analytical calculations to within ~1.0% and ~0.3% statistical uncertainties, respectively.

Noise, sign problems, and statistics

We show how sign problems in simulations of many-body systems can manifest themselves in the form of heavy-tailed correlator distributions, similar to what is seen in electron propagation through disordered media. We propose an alternative statistical approach for extracting ground state energies in such systems, illustrating the method with a toy model and with lattice data for unitary fermions.

Interplay of drug metabolism and transport: a real phenomenon or an artifact of the site of measurement?

The interdependence of both transport and metabolism on the disposition of drugs has recently gained heightened attention in the literature, and has been termed the "interplay of transport and metabolism". Such "interplay" is observed when inhibition of biliary clearance of a drug results in an "apparent" increase in the metabolic clearance of the drug or vice versa. In this manuscript, we derived and explored through simulations a physiological-based pharmacokinetic model that integrates both transport and metabolism and explains the "apparent" dependence of hepatic clearance on both these processes. In addition, we show that the phenomenon of hepatic "transport-metabolism interplay" is a result of using the plasma concentration as a point of reference when calculating metabolic or biliary clearance, and this interplay is maximal when the drug is actively transported into the hepatocytes (i.e., hepatocyte sinusoidal influx clearance is greater than the sinusoidal efflux clearance). When the hepatic drug concentration is used as a reference point to calculate metabolic or biliary clearance, this interplay ceases to exist. A mechanistic understanding of this interplay phenomenon can be used to explain the somewhat paradoxical results that may be observed in drug-drug interaction studies when a drug is cleared by both metabolism and biliary excretion. That is, when one of these two pathways is inhibited, the other pathway appears to be induced or activated. This interplay results in an increase in hepatic drug concentrations and therefore has implications for the hepatic efficacy and toxicity of a drug.