Causal inference methods for small non-randomized studies: Methods and an application in COVID-19

On the role of benchmarking data sets and simulations in method comparison studies

Method comparisons are essential to provide recommendations and guidance for applied researchers, who often have to choose from a plethora of available approaches. While many comparisons exist in the literature, these are often not neutral but favor a novel method. Apart from the choice of design and a proper reporting of the findings, there are different approaches concerning the underlying data for such method comparison studies. Most manuscripts on statistical methodology rely on simulation studies and provide a single real-world data set as an example to motivate and illustrate the methodology investigated. In the context of supervised learning, in contrast, methods are often evaluated using so-called benchmarking data sets, that is, real-world data that serve as gold standard in the community. Simulation studies, on the other hand, are much less common in this context. The aim of this paper is to investigate differences and similarities between these approaches, to discuss their advantages and disadvantages, and ultimately to develop new approaches to the evaluation of methods picking the best of both worlds. To this aim, we borrow ideas from different contexts such as mixed methods research and Clinical Scenario Evaluation.

Phase I study of a novel glioblastoma radiation therapy schedule exploiting cell-state plasticity

BACKGROUND

Glioblastomas comprise heterogeneous cell populations with dynamic, bidirectional plasticity between treatment-resistant stem-like and treatment-sensitive differentiated states, with treatment influencing this process. However, current treatment protocols do not account for this plasticity. Previously, we generated a mathematical model based on preclinical experiments to describe this process and optimize a radiation therapy fractionation schedule that substantially increased survival relative to standard fractionation in a murine glioblastoma model.

METHODS

We developed statistical models to predict the survival benefit of interventions to glioblastoma patients based on the corresponding survival benefit in the mouse model used in our preclinical study. We applied our mathematical model of glioblastoma radiation response to optimize a radiation therapy fractionation schedule for patients undergoing re-irradiation for glioblastoma and developed a first-in-human trial (NCT03557372) to assess the feasibility and safety of administering our schedule.

RESULTS

Our statistical modeling predicted that the hazard ratio when comparing our novel radiation schedule with a standard schedule would be 0.74. Our mathematical modeling suggested that a practical, near-optimal schedule for re-irradiation of recurrent glioblastoma patients was 3.96 Gy × 7 (1 fraction/day) followed by 1.0 Gy × 9 (3 fractions/day). Our optimized schedule was successfully administered to 14/14 (100%) patients.

CONCLUSIONS

A novel radiation therapy schedule based on mathematical modeling of cell-state plasticity is feasible and safe to administer to glioblastoma patients.

On the role of data, statistics and decisions in a pandemic

A pandemic poses particular challenges to decision-making because of the need to continuously adapt decisions to rapidly changing evidence and available data. For example, which countermeasures are appropriate at a particular stage of the pandemic? How can the severity of the pandemic be measured? What is the effect of vaccination in the population and which groups should be vaccinated first? The process of decision-making starts with data collection and modeling and continues to the dissemination of results and the subsequent decisions taken. The goal of this paper is to give an overview of this process and to provide recommendations for the different steps from a statistical perspective. In particular, we discuss a range of modeling techniques including mathematical, statistical and decision-analytic models along with their applications in the COVID-19 context. With this overview, we aim to foster the understanding of the goals of these modeling approaches and the specific data requirements that are essential for the interpretation of results and for successful interdisciplinary collaborations. A special focus is on the role played by data in these different models, and we incorporate into the discussion the importance of statistical literacy and of effective dissemination and communication of findings.

State-Level Masking Mandates and COVID-19 Outcomes in the United States: A Demonstration of the Causal Roadmap

BACKGROUND

We sought to investigate the effect of public masking mandates in US states on COVID-19 at the national level in Fall 2020. Specifically, we aimed to evaluate how the relative growth of COVID-19 cases and deaths would have differed if all states had issued a mandate to mask in public by 1 September 2020 versus if all states had delayed issuing such a mandate.

METHODS

We applied the Causal Roadmap, a formal framework for causal and statistical inference. We defined the outcome as the state-specific relative increase in cumulative cases and in cumulative deaths 21, 30, 45, and 60 days after 1 September. Despite the natural experiment occurring at the state-level, the causal effect of masking policies on COVID-19 outcomes was not identifiable. Nonetheless, we specified the target statistical parameter as the adjusted rate ratio (aRR): the expected outcome with early implementation divided by the expected outcome with delayed implementation, after adjusting for state-level confounders. To minimize strong estimation assumptions, primary analyses used targeted maximum likelihood estimation with Super Learner.

RESULTS

After 60 days and at a national level, early implementation was associated with a 9% reduction in new COVID-19 cases (aRR = 0.91 [95% CI = 0.88, 0.95]) and a 16% reduction in new COVID-19 deaths (aRR = 0.84 [95% CI = 0.76, 0.93]).

CONCLUSIONS

Although lack of identifiability prohibited causal interpretations, application of the Causal Roadmap facilitated estimation and inference of statistical associations, providing timely answers to pressing questions in the COVID-19 response.

Variation in Use of Repurposed Medications Among Patients With Coronavirus Disease 2019. From The Society of Critical Care Medicine Discovery Viral Infection and Respiratory Illness Universal Study: Coronavirus Disease 2019 Registry Investigator Group

IMPORTANCE

At the start of the coronavirus disease 2019 pandemic, medications repurposed for management of coronavirus disease 2019 were used in the absence of clinical trial evidence.

OBJECTIVES

To describe the variation and evolution in use of repurposed medications for coronavirus disease 2019.

DESIGN SETTING AND PARTICIPANTS

Observational cohort study of adults hospitalized with coronavirus disease 2019 between February 15, 2020, and April 12, 2021, across 76 United States and international hospitals within the Society of Critical Care Medicine's Discovery Viral Infection and Respiratory Illness Universal Study coronavirus disease 2019 registry.

MAIN OUTCOMES AND MEASURES

Hospital variation was quantified using multivariable adjusted random effects logistic regression models and unsupervised clustering. Repurposed medications included antivirals, corticosteroids, hydroxychloroquine, immunomodulators, and therapeutic dose anticoagulants.

RESULTS

Among 7,069 adults hospitalized with coronavirus disease 2019, 1,979 (28%) received antivirals, 2,876 (41%) received corticosteroids, 1,779 (25%) received hydroxychloroquine, 620 (9%) received immunomodulators, and 2,154 (31%) received therapeutic dose anticoagulants. Contribution of hospital site to risk-adjusted variation was 46% for antivirals, 30% for corticosteroids, 48% for hydroxychloroquine, 46% for immunomodulators, and 52% for therapeutic dose anticoagulants. Compared with the early pandemic, the later pandemic practice phenotypes converged with increased use of antivirals (odds ratio, 3.14; 95% CI, 2.40-4.10) and corticosteroids (odds ratio, 5.43; 95% CI, 4.23-6.97), with decreased use of hydroxychloroquine (odds ratio, 0.02; 95% CI, 0.01-0.04) and immunomodulators (odds ratio, 0.49; 95% CI, 0.34-0.70). There was no clinically significant change in the use of therapeutic dose anticoagulants (odds ratio, 1.01; 95% CI, 1.01-1.02). There were no differences in risk-adjusted mortality between hospitals with high rates of repurposed medication use compared with hospitals with low rates of use.

CONCLUSIONS AND RELEVANCE

Hospital variation in the use of repurposed medications varied widely across hospitals early in the pandemic and later converged with the emergence of randomized clinical trials. Platforms developed for rapid activation and enrollment in clinical trials of repurposed medications are needed prior to the next pandemic to expedite effective, evidence-based practice.