Using public clinical trial reports to probe non-experimental causal inference methods

BACKGROUND

Non-experimental studies (also known as observational studies) are valuable for estimating the effects of various medical interventions, but are notoriously difficult to evaluate because the methods used in non-experimental studies require untestable assumptions. This lack of intrinsic verifiability makes it difficult both to compare different non-experimental study methods and to trust the results of any particular non-experimental study.

METHODS

We introduce TrialProbe, a data resource and statistical framework for the evaluation of non-experimental methods. We first collect a dataset of pseudo "ground truths" about the relative effects of drugs by using empirical Bayesian techniques to analyze adverse events recorded in public clinical trial reports. We then develop a framework for evaluating non-experimental methods against that ground truth by measuring concordance between the non-experimental effect estimates and the estimates derived from clinical trials. As a demonstration of our approach, we also perform an example methods evaluation between propensity score matching, inverse propensity score weighting, and an unadjusted approach on a large national insurance claims dataset.

RESULTS

From the 33,701 clinical trial records in our version of the ClinicalTrials.gov dataset, we are able to extract 12,967 unique drug/drug adverse event comparisons to form a ground truth set. During our corresponding methods evaluation, we are able to use that reference set to demonstrate that both propensity score matching and inverse propensity score weighting can produce estimates that have high concordance with clinical trial results and substantially outperform an unadjusted baseline.

CONCLUSIONS

We find that TrialProbe is an effective approach for probing non-experimental study methods, being able to generate large ground truth sets that are able to distinguish how well non-experimental methods perform in real world observational data.

A compound decision approach to covariance matrix estimation

Covariance matrix estimation is a fundamental statistical task in many applications, but the sample covariance matrix is sub-optimal when the sample size is comparable to or less than the number of features. Such high-dimensional settings are common in modern genomics, where covariance matrix estimation is frequently employed as a method for inferring gene networks. To achieve estimation accuracy in these settings, existing methods typically either assume that the population covariance matrix has some particular structure, for example sparsity, or apply shrinkage to better estimate the population eigenvalues. In this paper, we study a new approach to estimating high-dimensional covariance matrices. We first frame covariance matrix estimation as a compound decision problem. This motivates defining a class of decision rules and using a nonparametric empirical Bayes g-modeling approach to estimate the optimal rule in the class. Simulation results and gene network inference in an RNA-seq experiment in mouse show that our approach is comparable to or can outperform a number of state-of-the-art proposals. This article is protected by copyright. All rights reserved.

Holistic Process Models: A Bayesian Predictive Ensemble Method for Single and Coupled Unit Operation Models

The coupling of individual models in terms of end-to-end calculations for unit operations in manufacturing processes is a challenging task. We present a probability distribution-based approach for the combined outcomes of parametric and non-parametric models. With this so-called Bayesian predictive ensemble, the statistical moments such as mean value and standard deviation can be accurately computed without any further approximation. It is shown that the ensemble of different model predictions leads to an uninformed prior distribution, which can be transformed into a predictive posterior distribution using Bayesian inference and numerical Markov Chain Monte Carlo calculations. We demonstrate the advantages of our method using several numerical examples. Our approach is not restricted to certain unit operations, and can also be used for the more robust interpretation and assessment of model predictions in general.

Collective Variational Inference for Personalized and Generative Physiological Modeling: A Case Study on Hemorrhage Resuscitation

OBJECTIVE

Individual physiological experiments typically provide useful but incomplete information about a studied physiological process. As a result, inferring the unknown parameters of a physiological model from experimental data is often challenging. The objective of this paper is to propose and illustrate the efficacy of a collective variational inference (C-VI) method, intended to reconcile low-information and heterogeneous data from a collection of experiments to produce robust personalized and generative physiological models.

METHODS

To derive the C-VI method, we utilize a probabilistic graphical model to impose structure on the available physiological data, and algorithmically characterize the graphical model using variational Bayesian inference techniques. To illustrate the efficacy of the C-VI method, we apply it to a case study on the mathematical modeling of hemorrhage resuscitation.

RESULTS

In the context of hemorrhage resuscitation modeling, the C-VI method could reconcile heterogeneous combinations of hematocrit, cardiac output, and blood pressure data across multiple experiments to obtain (i) robust personalized models along with associated measures of uncertainty and signal quality, and (ii) a generative model capable of reproducing the physiological behavior of the population.

CONCLUSION

The C-VI method facilitates the personalized and generative modeling of physiological processes in the presence of low-information and heterogeneous data.

SIGNIFICANCE

The resulting models provide a solid basis for the development and testing of interpretable physiological monitoring, decision-support, and closed-loop control algorithms.