The Current Landscape in Biostatistics of Real-World Data and Evidence: Clinical Study Design and Analysis

Real Effect or Bias? Good Practices for Evaluating the Robustness of Evidence From Comparative Observational Studies Through Quantitative Sensitivity Analysis for Unmeasured Confounding

The assumption of "no unmeasured confounders" is a critical but unverifiable assumption required for causal inference yet quantitative sensitivity analyses to assess robustness of real-world evidence remains under-utilized. The lack of use is likely in part due to complexity of implementation and often specific and restrictive data requirements for application of each method. With the advent of methods that are broadly applicable in that they do not require identification of a specific unmeasured confounder-along with publicly available code for implementation-roadblocks toward broader use of sensitivity analyses are decreasing. To spur greater application, here we offer a good practice guidance to address the potential for unmeasured confounding at both the design and analysis stages, including framing questions and an analytic toolbox for researchers. The questions at the design stage guide the researcher through steps evaluating the potential robustness of the design while encouraging gathering of additional data to reduce uncertainty due to potential confounding. At the analysis stage, the questions guide quantifying the robustness of the observed result and providing researchers with a clearer indication of the strength of their conclusions. We demonstrate the application of this guidance using simulated data based on an observational fibromyalgia study, applying multiple methods from our analytic toolbox for illustration purposes.

Considerations for master protocols using external controls

There has been an increasing use of master protocols in oncology clinical trials because of its efficiency to accelerate cancer drug development and flexibility to accommodate multiple substudies. Depending on the study objective and design, a master protocol trial can be a basket trial, an umbrella trial, a platform trial, or any other form of trials in which multiple investigational products and/or subpopulations are studied under a single protocol. Master protocols can use external data and evidence (e.g. external controls) for treatment effect estimation, which can further improve efficiency of master protocol trials. This paper provides an overview of different types of external controls and their unique features when used in master protocols. Some key considerations in master protocols with external controls are discussed including construction of estimands, assessment of fit-for-use real-world data, and considerations for different types of master protocols. Similarities and differences between regular randomized controlled trials and master protocols when using external controls are discussed. A targeted learning-based causal roadmap is presented which constitutes three key steps: (1) define a target statistical estimand that aligns with the causal estimand for the study objective, (2) use an efficient estimator to estimate the target statistical estimand and its uncertainty, and (3) evaluate the impact of causal assumptions on the study conclusion by performing sensitivity analyses. Two illustrative examples for master protocols using external controls are discussed for their merits and possible improvement in causal effect estimation.

FAIR enough: Building an academic data ecosystem to make real-world data available for translational research

The Stanford Population Health Sciences Data Ecosystem was created to facilitate the use of large datasets containing health records from hundreds of millions of individuals. This necessitated technical solutions optimized for an academic medical center to manage and share high-risk data at scale. Through collaboration with internal and external partners, we have built a Data Ecosystem to host, curate, and share data with hundreds of users in a secure and compliant manner. This platform has enabled us to host unique data assets and serve the needs of researchers across Stanford University, and the technology and approach were designed to be replicable and portable to other institutions. We have found, however, that though these technological advances are necessary, they are not sufficient. Challenges around making data Findable, Accessible, Interoperable, and Reusable remain. Our experience has demonstrated that there is a high demand for access to real-world data, and that if the appropriate tools and structures are in place, translational research can be advanced considerably. Together, technological solutions, management structures, and education to support researcher, data science, and community collaborations offer more impactful processes over the long-term for supporting translational research with real-world data.

Comparative effectiveness of tocilizumab vs standard care in patients with severe COVID-19-related pneumonia: a retrospective cohort study utilizing registry data as a synthetic control

BACKGROUND

The severity of coronavirus disease 2019 (COVID-19) infections has led to the development of several therapeutic agents, with tocilizumab becoming increasingly used to treat patients with COVID-19-related pneumonia. This study compared the use of tocilizumab treatment with the standard of care (SOC) to determine its efficacy against severe COVID-19-related pneumonia in Japan.

METHODS

This retrospective cohort study was designed to evaluate the efficacy of tocilizumab in two different databases: the JA42434 single-arm study and COVID-19 Registry Japan (COVIREGI-JP), with a synthetic control group from the COVIREGI-JP cohort as a benchmark for the tocilizumab group. The study's primary objective was to evaluate the efficacy of tocilizumab in treating severe COVID-19-related pneumonia compared to the SOC among patients included in the above two databases. The SOC group was extracted as the synthetic control group using exact matching and a propensity score matching in sequence per subject. As a secondary objective, the efficacy of tocilizumab compared to the SOC was evaluated exclusively among patients included in the COVIREGI-JP database. In each objective, the primary endpoint was defined as the time to discharge or the status of awaiting discharge.

RESULTS

For the primary endpoint, the hazard ratio (HR) of the tocilizumab group against the SOC group was 1.070 (95% confidence interval [CI]: 0.565-2.028). The median time from Study Day 1 to discharge or the state of awaiting discharge was 15 days in the tocilizumab group and 16 days in the SOC group. The HRs for the secondary endpoints, namely, time to improvement in the clinical state, time to clinical failure, and time to recovery, were 1.112 (95% CI: 0.596-2.075), 0.628 (95% CI: 0.202-1.953), and 1.019 (95% CI: 0.555-1.871), respectively. Similarly, the HR of the primary endpoint for the secondary objective was 0.846 (95% CI: 0.582-1.230).

CONCLUSIONS

Tocilizumab did not demonstrate a positive effect on time to discharge or the state of awaiting discharge. Furthermore, no statistically significant differences in other clinical outcomes, such as time to improvement in the clinical state, time to clinical failure, and time to recovery, were observed among the groups.

Considerations for pooling real-world data as a comparator cohort to a single arm trial: a simulation study on assessment of heterogeneity

BACKGROUND

Novel precision medicine therapeutics target increasingly granular, genomically-defined populations. Rare sub-groups make it challenging to study within a clinical trial or single real-world data (RWD) source; therefore, pooling from disparate sources of RWD may be required for feasibility. Heterogeneity assessment for pooled data is particularly complex when contrasting a pooled real-world comparator cohort (rwCC) with a single-arm clinical trial (SAT), because the individual comparisons are not independent as all compare a rwCC to the same SAT. Our objective was to develop a methodological framework for pooling RWD focused on the rwCC use case, and simulate novel approaches of heterogeneity assessment, especially for small datasets.

METHODS

We present a framework with the following steps: pre-specification, assessment of dataset eligibility, and outcome analyses (including assessment of outcome heterogeneity). We then simulated heterogeneity assessments for a binary response outcome in a SAT compared to two rwCCs, using standard methods for meta-analysis, and an Adjusted Cochran's Q test, and directly comparing the individual participant data (IPD) from the rwCCs.

RESULTS

We found identical power to detect a true difference for the adjusted Cochran's Q test and the IPD method, with both approaches superior to a standard Cochran's Q test. When assessing the impact of heterogeneity in the null scenario of no difference between the SAT and rwCCs, a lack of statistical power led to Type 1 error inflation. Similarly, in the alternative scenario of a true difference between SAT and rwCCs, we found substantial Type 2 error, with underpowered heterogeneity testing leading to underestimation of the treatment effect.

CONCLUSIONS

We developed a methodological framework for pooling RWD sources in the context of designing a rwCC for a SAT. When testing for heterogeneity during this process, the adjusted Cochran's Q test matches the statistical power of IPD heterogeneity testing. Limitations of quantitative heterogeneity testing in protecting against Type 1 or Type 2 error indicate these tests are best used descriptively, and after careful selection of datasets based on clinical/data considerations. We hope these findings will facilitate the rigorous pooling of RWD to unlock insights to benefit oncology patients.

Increasing patient participation in oncology clinical trials

AIM

Timely recruitment of eligible participants is essential for the success of clinical trials, with insufficient accrual being the leading cause for premature termination of both oncology and non-oncology trials.

METHODS

In this paper we further elaborate on the challenges for patient participation in oncology trials from physician, patient, healthcare system, and some trial-related perspectives.

RESULTS

We present strategies such as use of digital healthcare technologies, real-world data and real-world evidence, decentralized clinical trials, pragmatic trial designs, and supportive services to increase patient participation.

CONCLUSIONS

Multifaceted measures are necessary to increase patient participation, especially for those who are under-represented in cancer trials.

Propensity-score-based meta-analytic predictive prior for incorporating real-world and historical data

As the availability of real-world data sources (eg, EHRs, claims data, registries) and historical data has rapidly surged in recent years, there is an increasing interest and need from investigators and health authorities to leverage all available information to reduce patient burden and accelerate both drug development and regulatory decision making. Bayesian meta-analytic approaches are a popular historical borrowing method that has been developed to leverage such data using robust hierarchical models. The model structure accounts for various degrees of between-trial heterogeneity, resulting in adaptively discounting the external information in the case of data conflict. In this article, we propose to integrate the propensity score method and Bayesian meta-analytic-predictive (MAP) prior to leverage external real-world and historical data. The propensity score methodology is applied to select a subset of patients from external data that are similar to those in the current study with regards to key baseline covariates and to stratify the selected patients together with those in the current study into more homogeneous strata. The MAP prior approach is used to obtain stratum-specific MAP prior and derive the overall propensity score integrated meta-analytic predictive (PS-MAP) prior. Additionally, we allow for tuning the prior effective sample size for the proposed PS-MAP prior, which quantifies the amount of information borrowed from external data. We evaluate the performance of the proposed PS-MAP prior by comparing it to the existing propensity score-integrated power prior approach in a simulation study and illustrate its implementation with an example of a single-arm phase II trial.