Impervious to Randomness: Confounding and Selection Biases in Randomized Clinical Trials

Interpreting Randomized Controlled Trials

This article describes rationales and limitations for making inferences based on data from randomized controlled trials (RCTs). We argue that obtaining a representative random sample from a patient population is impossible for a clinical trial because patients are accrued sequentially over time and thus comprise a convenience sample, subject only to protocol entry criteria. Consequently, the trial's sample is unlikely to represent a definable patient population. We use causal diagrams to illustrate the difference between random allocation of interventions within a clinical trial sample and true simple or stratified random sampling, as executed in surveys. We argue that group-specific statistics, such as a median survival time estimate for a treatment arm in an RCT, have limited meaning as estimates of larger patient population parameters. In contrast, random allocation between interventions facilitates comparative causal inferences about between-treatment effects, such as hazard ratios or differences between probabilities of response. Comparative inferences also require the assumption of transportability from a clinical trial's convenience sample to a targeted patient population. We focus on the consequences and limitations of randomization procedures in order to clarify the distinctions between pairs of complementary concepts of fundamental importance to data science and RCT interpretation. These include internal and external validity, generalizability and transportability, uncertainty and variability, representativeness and inclusiveness, blocking and stratification, relevance and robustness, forward and reverse causal inference, intention to treat and per protocol analyses, and potential outcomes and counterfactuals.

Obesity and renal cell carcinoma: Biological mechanisms and perspectives

Obesity, defined by body mass index (BMI), is an established risk factor for specific renal cell carcinoma (RCC) subtypes such as clear cell RCC, the most common RCC histology. Many studies have identified an association between obesity and improved survival after diagnosis of RCC, a potential "obesity paradox." Clinically, there is uncertainty whether improved outcomes observed after diagnosis are driven by stage, type of treatment received, or artifacts of longitudinal changes in weight and body composition. The biological mechanisms underlying obesity's influence on RCC are not fully established, but multiomic and mechanistic studies suggest an impact on tumor metabolism, particularly fatty acid metabolism, angiogenesis, and peritumoral inflammation, which are known to be key biological hallmarks of clear cell RCC. Conversely, high-intensity exercise associated with increased muscle mass may be a risk factor for renal medullary carcinoma, a rare RCC subtype that predominantly occurs in individuals with sickle hemoglobinopathies. Herein, we highlight methodologic challenges associated with studying the influence of obesity on RCC and review the clinical evidence and potential underlying mechanisms associating RCC with BMI and body composition.

Less is More? First Impressions From COSMIC-313

The COSMIC-313 phase 3 randomized controlled trial tested the triplet combination of cabozantinib with nivolumab and ipilimumab in comparison with nivolumab plus ipilimumab control as fist-line systemic therapy in metastatic clear cell renal cell carcinoma. The first results presented at the 2022 European Society of Medical Oncology Congress are a milestone for the renal cell carcinoma field because they signal the advent of triplet combinations as potential treatment options for our patients. The present commentary highlights some considerations and potential next steps based on these first impressions.

A Causal Framework for Making Individualized Treatment Decisions in Oncology

Simple Summary

Physicians routinely make individualized treatment decisions by accounting for the joint effects of patient prognostic covariates and treatments on clinical outcomes. Ideally, this is performed using historical randomized clinical trial (RCT) data. Randomization ensures that unbiased estimates of causal treatment effect parameters can be obtained from the historical RCT data and used to predict each new patient’s outcome based on the joint effect of their baseline covariates and each treatment being considered. However, this process becomes problematic if a patient seen in the clinic is very different from the patients who were enrolled in the RCT. That is, if a new patient does not satisfy the entry criteria of the RCT, then the patient does not belong to the population represented by the patients who were studied in the RCT. In such settings, it still may be possible to utilize the RCT data to help choose a new patient’s treatment. This may be achieved by combining the RCT data with data from other clinical trials, or possibly preclinical experiments, and using the combined dataset to predict the patient’s expected outcome for each treatment being considered. In such settings, combining data from multiple sources in a way that is statistically reliable is not entirely straightforward, and correctly identifying and estimating the effects of treatments and patient covariates on clinical outcomes can be complex. Causal diagrams provide a rational basis to guide this process. The first step is to construct a causal diagram that reflects the plausible relationships between treatment variables, patient covariates, and clinical outcomes. If the diagram is correct, it can be used to determine what additional data may be needed, how to combine data from multiple sources, how to formulate a statistical model for clinical outcomes as a function of treatment and covariates, and how to compute an unbiased treatment effect estimate for each new patient. We use adjuvant therapy of renal cell carcinoma to illustrate how causal diagrams may be used to guide these steps.

Abstract

We discuss how causal diagrams can be used by clinicians to make better individualized treatment decisions. Causal diagrams can distinguish between settings where clinical decisions can rely on a conventional additive regression model fit to data from a historical randomized clinical trial (RCT) to estimate treatment effects and settings where a different approach is needed. This may be because a new patient does not meet the RCT’s entry criteria, or a treatment’s effect is modified by biomarkers or other variables that act as mediators between treatment and outcome. In some settings, the problem can be addressed simply by including treatment–covariate interaction terms in the statistical regression model used to analyze the RCT dataset. However, if the RCT entry criteria exclude a new patient seen in the clinic, it may be necessary to combine the RCT data with external data from other RCTs, single-arm trials, or preclinical experiments evaluating biological treatment effects. For example, external data may show that treatment effects differ between histological subgroups not recorded in an RCT. A causal diagram may be used to decide whether external observational or experimental data should be obtained and combined with RCT data to compute statistical estimates for making individualized treatment decisions. We use adjuvant treatment of renal cell carcinoma as our motivating example to illustrate how to construct causal diagrams and apply them to guide clinical decisions.

Is short-course radiotherapy and total neoadjuvant therapy the new standard of care in locally advanced rectal cancer? A sensitivity analysis of the RAPIDO clinical trial

BACKGROUND

The results of the RAPIDO trial have been accepted as evidence in favour of short-course radiotherapy (SC-RT) followed by chemotherapy before total mesorectal excision in high-risk locally advanced rectal cancer. A noteworthy concern is that the RAPIDO trial did not ensure that all patients in the control arm received adjuvant chemotherapy. This may bias statistical estimates in favour of the experimental arm if adjuvant chemotherapy is active in rectal cancer. Moreover, the 5-year update revealed an increase in the risk of local relapse in the experimental arm.

MATERIALS AND METHODS

We carried out sensitivity analyses to determine how plausible effects of adjuvant chemotherapy, adjusted by the proportion of patients in the standard arm receiving adjuvant treatment, would have influenced the observed treatment effect estimate of the RAPIDO trial. The most plausible values for the benefit of adjuvant chemotherapy were determined by Bayesian re-analysis of a prior meta-analysis.

RESULTS

The meta-analysis suggested that oxaliplatin/fluorouracil-based adjuvant chemotherapy may improve disease-free survival (DFS) in rectal cancer although the signal is weak [hazard ratio (HR) 0.84, 95% credible interval, 0.57-1.15]; probability of benefit (HR <1) was 91.2%. In the sensitivity analysis, the HR for disease-related treatment failure would remain <1, thus favouring total neoadjuvant therapy (TNT), on most occasions, but the null hypothesis would not have been rejected in various credible settings. For the RAPIDO data to be consistent with the null effect, a moderate benefit of adjuvant chemotherapy (HR for DFS between 0.75 and 0.80) and 70%-80% of exposed participants would suffice.

CONCLUSION

The decision to make adjuvant chemotherapy optional in the standard arm may have biased the results in favour of the experimental arm, in a scenario in which TNT does not offset the increase in local recurrences after SC-RT.

The Big Data Paradox in Clinical Practice

The big data paradox is a real-world phenomenon whereby as the number of patients enrolled in a study increases, the probability that the confidence intervals from that study will include the truth decreases. This occurs in both observational and experimental studies, including randomized clinical trials, and should always be considered when clinicians are interpreting research data. Furthermore, as data quantity continues to increase in today's era of big data, the paradox is becoming more pernicious. Herein, I consider three mechanisms that underlie this paradox, as well as three potential strategies to mitigate it: (1) improving data quality; (2) anticipating and modeling patient heterogeneity; (3) including the systematic error, not just the variance, in the estimation of error intervals.

Missing the trees for the forest: most subgroup analyses using forest plots at the ASCO annual meeting are inconclusive

Background:

Oncologists often refer to forest plots to determine which patient subgroups may be more likely to benefit from a therapy tested in a randomized clinical trial (RCT). We sought to empirically determine the information content of subgroup comparisons from forest plots of RCTs.

Methods:

We assessed all forest plots from RCTs of therapeutic interventions presented orally at the American Society of Clinical Oncology Annual Meetings in 2020 and 2021. Subgroups were considered as showing evidence of treatment effect heterogeneity in forest plots when their confidence intervals (CIs) did not overlap with the vertical line corresponding to the main effect observed in the overall RCT cohort. Subgroups were considered as showing evidence of treatment effect homogeneity in forest plots when their CIs did not meaningfully differ, within 80–125% equivalence range, with the values compatible with the main effect. All other subgroups were considered as inconclusive.

Results:

A total of 99 forest plots were presented, and only 24.2% contained one or more subgroups suggestive of treatment effect heterogeneity. A total of 81 forest plots provided enough information to evaluate treatment effect heterogeneity and homogeneity. These 81 forest plots represented a total of 1344 individual subgroups, of which 57.2% were inconclusive, 41.1% showed evidence of treatment effect homogeneity, and 1.6% yielded evidence suggestive of treatment effect heterogeneity.

Conclusion:

The majority of subgroup comparisons were inconclusive in this empirical analysis of forest plots used in oncology RCTs. Different strategies should be considered to improve the estimation and representation of subgroup-specific effects.

Fooled by Randomness. The Misleading Effect of Treatment Crossover in Randomized Trials of Therapies with Marginal Treatment Benefit

Crossover can bias clinical outcomes of randomized clinical trials by increasing the risk of both type I (false positive) and type II (false negative) errors. To show how crossover can increase type I error, we provide computer simulation and review herein illustrative examples (iniparib, olaratumab) of recently reported RCTs that demonstrated false-positive treatment efficacy signals due to crossover. The ethical issues associated with crossover are also discussed.

Causal Considerations Can Inform the Interpretation of Surprising Associations in Medical Registries

An exploratory analysis of registry data from 2437 patients with advanced gastric cancer revealed a surprising association between astrological birth signs and overall survival (OS) with p = 0.01. After dichotomizing or changing the reference sign, p-values <0.05 were observed for several birth signs following adjustments for multiple comparisons. Bayesian models with moderately skeptical priors still pointed to these associations. A more plausible causal model, justified by contextual knowledge, revealed that these associations arose from the astrological sign association with seasonality. This case study illustrates how causal considerations can guide analyses through what would otherwise be a hopeless maze of statistical possibilities.