Statistical Decision Properties of Imprecise Trials Assessing Coronavirus Disease 2019 (COVID-19) Drugs

Toward Patient-Centered Drug Approval for Treatment of Rare Diseases

OBJECTIVES

This commentary seeks to improve the design and analysis of trials undertaken to obtain approval of drugs for treatment of rare diseases.

METHODS

Methodological analysis reveals that use of hypothesis testing in the Food and Drug Administration drug approval process is harmful. Conventional asymmetric error probabilities bias the approval process against approval of new drugs. Hypothesis testing is inattentive to the relative magnitudes of losses to patient welfare when types 1 and 2 errors occur. Requiring the sample size to be large enough to guarantee the specified statistical power particularly inhibits the development of new drugs for treating rare diseases. Rarity of a disease makes it difficult to enroll the number of trial subjects needed to meet the statistical power standards for drug approval.

RESULTS

Use of statistical decision theory in drug approval would overcome these serious deficiencies of hypothesis testing. Sample size would remain relevant to drug approval, but the criterion used to evaluate sample size would change. Rather than judging sample size by statistical power, the Food and Drug Administration could require a sample to be large enough to provide a specified nearness to optimality of the approval decision.

CONCLUSIONS

Using nearness to optimality to set sample size and making approval decisions to minimize distance from optimality would particularly benefit the evaluation of drugs for treatment of rare diseases. It would enable a dramatic reduction in sample size relative to current norms, without compromising the clinical informativeness of trials.

Partial Identification of Personalized Treatment Response with Trial-reported Analyses of Binary Subgroups

Medical journals have adhered to a reporting practice that seriously limits the usefulness of published trial findings. Medical decision makers commonly observe many patient covariates and seek to use this information to personalize treatment choices. Yet standard summaries of trial findings only partition subjects into broad subgroups, typically binary categories. Given this reporting practice, we study the problem of inference on long mean treatment outcomes E[y(t)|x], where t is a treatment, y(t) is a treatment outcome, and the covariate vector x has length K, each component being a binary variable. The available data are estimates of {E[y(t)|x k = 0], E[y(t)|x k = 1], P(x k )}, k = 1,..., K reported in journal articles. We show that reported trial findings partially identify {E[y(t)|x], P(x)}. Illustrative computations demonstrate that the summaries of trial findings in journal articles may imply only wide bounds on long mean outcomes. One can realistically tighten inferences if one can combine reported trial findings with credible assumptions having identifying power, such as bounded-variation assumptions.

True COVID-19 mortality rates from administrative data

In this paper, I use administrative data to estimate the number of deaths, the number of infections, and mortality rates from COVID-19 in Lombardia, the hot spot of the disease in Italy and Europe. The information will assist policy makers in reaching correct decisions and the public in adopting appropriate behaviors. As the available data suffer from sample selection bias, I use partial identification to derive the above quantities. Partial identification combines assumptions with the data to deliver a set of admissible values or bounds. Stronger assumptions yield stronger conclusions but decrease the credibility of the inference. Therefore, I start with assumptions that are always satisfied, then I impose increasingly more restrictive assumptions. Using my preferred bounds, during March 2020 in Lombardia, there were between 10,000 and 18,500 more deaths than in previous years. The narrowest bounds of mortality rates from COVID-19 are between 0.1 and 7.5%, much smaller than the 17.5% discussed in earlier reports. This finding suggests that the case of Lombardia may not be as special as some argue.