Reliably basing conclusions on subgroups of randomized clinical trials

Subgroup analyses in venous thromboembolism trials reporting pharmacological interventions: A systematic review

INTRODUCTION

Randomized controlled trials (RCTs) that conduct subgroup analyses have the potential to provide information on treatment decisions in specific groups of patients from heterogeneous populations. Although we understand several factors can modify the incidence of venous thromboembolism (VTE) and the benefit/risk ratio of anticoagulation treatments, further evidence is warranted to show the heterogeneity of treatment effects in different subgroups of patients.

AIMS

The primary purpose was to evaluate the appropriateness and interpretation of subgroup analysis performed on VTE RCTs reporting pharmacological interventions.

MATERIALS AND METHODS

A systematic review of RCTs published between January 2017 and January 2022 was conducted. Claims of subgroup effects were evaluated with predefined criteria. High-quality claims of subgroup effect were further analyzed and discussed.

RESULTS

Overall, 28 RCTs with a generally low bias risk were included. The purposes of the treatments included pharmacologic thromboprophylaxis (17), therapeutic dose anticoagulation (9), and catheter-directed pharmacologic thrombolysis (2). The evaluated subgroup analyses generally presented: a high number of subgroup analyses reported, a lack of prespecification, and a lack of usage of statistical tests for interaction. The authors reported 13 claims of subgroup effect; only two were considered potentially reliable to represent heterogeneity in the direction or magnitude of treatment effect.

CONCLUSIONS

Subgroup analyses of VTE RCTs reporting pharmacologic interventions are generally methodologically poor. Most claims of subgroup effect did not meet critical criteria and lacked credibility. Clinicians in this field may proceed with scepticism when assessing claims of subgroup effects due to methodological concerns and misleading interpretations.

Stratified split sampling of electronic health records

Although superficially similar to data from clinical research, data extracted from electronic health records may require fundamentally different approaches for model building and analysis. Because electronic health record data is designed for clinical, rather than scientific use, researchers must first provide clear definitions of outcome and predictor variables. Yet an iterative process of defining outcomes and predictors, assessing association, and then repeating the process may increase Type I error rates, and thus decrease the chance of replicability, defined by the National Academy of Sciences as the chance of "obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data."[1] In addition, failure to account for subgroups may mask heterogeneous associations between predictor and outcome by subgroups, and decrease the generalizability of the findings. To increase chances of replicability and generalizability, we recommend using a stratified split sample approach for studies using electronic health records. A split sample approach divides the data randomly into an exploratory set for iterative variable definition, iterative analyses of association, and consideration of subgroups. The confirmatory set is used only to replicate results found in the first set. The addition of the word 'stratified' indicates that rare subgroups are oversampled randomly by including them in the exploratory sample at higher rates than appear in the population. The stratified sampling provides a sufficient sample size for assessing heterogeneity of association by testing for effect modification by group membership. An electronic health record study of the associations between socio-demographic factors and uptake of hepatic cancer screening, and potential heterogeneity of association in subgroups defined by gender, self-identified race and ethnicity, census-tract level poverty and insurance type illustrates the recommended approach.

Subgroup Analysis in Pulmonary Hypertension-Specific Therapy Clinical Trials: A Systematic Review

Pulmonary hypertension (PH) treatment decisions are driven by the results of randomized controlled trials (RCTs). Subgroup analyses are often performed to assess whether the intervention effect will change due to the patient's characteristics, thus allowing for individualized decisions. This review aimed to evaluate the appropriateness and interpretation of subgroup analyses performed in PH-specific therapy RCTs published between 2000 and 2020. Claims of subgroup effects were evaluated with prespecified criteria. Overall, 30 RCTs were included. Subgroup analyses presented: a high number of subgroup analyses reported, lack of prespecification, and lack of interaction tests. The trial protocol was not available for most RCTs; significant differences were found in those articles that published the protocol. Authors reported 13 claims of subgroup effect, with 12 claims meeting four or fewer of Sun's criteria. Even when most RCTs were generally at low risk of bias and were published in high-impact journals, the credibility and general quality of subgroup analyses and subgroup claims were low due to methodological flaws. Clinicians should be skeptical of claims of subgroup effects and interpret subgroup analyses with caution, as due to their poor quality, these analyses may not serve as guidance for personalized care.

Subgroup analysis in haematologic malignancies phase III clinical trials: A systematic review

AIMS

To assess the appropriateness of the use and interpretation of subgroup analysis in haematology randomized clinical trials (RCTs).

METHOD

A systematic review of Medline, including haematology phase III RCTs published between January 2013 and October 2019, was carried out to identify reported subgroup analysis. Information related to trial characteristics, subgroup analysis and claims of subgroup difference were collected.

RESULTS

The initial search identified 1622 studies. A total of 98 studies reporting subgroup analyses were identified. Of those, 24 RCT reported 46 claims of subgroup difference. Among them, 44 were claims for the primary outcome, of which 25 were considered strong claims and 17 were considered suggestions of a possible effect. Authors included subgroup variables for the primary outcome measured at baseline for 38 claims (n = 86.36%), used a subgroup variable as a stratification factor at randomization for 15 (34.09%), clearly prespecified their hypothesis for 11 (25%), the subgroup effect was one of a small number of hypothesised effects tested (≤ 5) for 17 (38.64%), carried out a test of interaction that provide statistically significant for 18 (40.91%), documented replication of a subgroup effect with previously related studies for 11 (25%), identified the consistency of a subgroup effect across related outcome for 10 (22.72%) and provided a biological rationale for the effect for 8 (18.18%). Of the 44 claims for the primary outcome, 34 (77.27%) met four or fewer of the 10 credibility criteria.

CONCLUSION

The subgroup claims reported in haematology RCTs lack credibility, even when the claims are strong. Information about subgroup difference should be interpreted cautiously.

A systematic survey identified 36 criteria for assessing effect modification claims in randomized trials or meta-analyses

OBJECTIVE

The objective of the study was to systematically survey the methodological literature and collect suggested criteria for assessing the credibility of effect modification and associated rationales.

STUDY DESIGN AND SETTING

We searched MEDLINE, Embase, and WorldCat up to March 2018 for publications providing guidance for assessing the credibility of effect modification identified in randomized trials or meta-analyses. Teams of two investigators independently identified eligible publications and extracted credibility criteria and authors' rationale, reaching consensus through discussion. We created a taxonomy of criteria that we iteratively refined during data abstraction.

RESULTS

We identified 150 eligible publications that provided 36 criteria and associated rationales. Frequent criteria included significant test for interaction (n = 54), a priori hypothesis (n = 49), providing a causal explanation (n = 47), accounting for multiplicity (n = 45), testing a small number of effect modifiers (n = 38), and prespecification of analytic details (n = 39). For some criteria, we found more than one rationale; some criteria were connected through a common rationale. For some criteria, experts disagreed regarding their suitability (e.g., added value of stratified randomization; trustworthiness of biologic rationales).

CONCLUSION

Methodologists have expended substantial intellectual energy providing criteria for critical appraisal of apparent effect modification. Our survey highlights popular criteria, expert agreement and disagreement, and where more work is needed, including testing criteria in practice.

Level of evidence for promising subgroup findings: The case of trends and multiple subgroups

Subgroup analyses are an essential part of fully understanding the complete results from confirmatory clinical trials. However, they come with substantial methodological challenges. In case no statistically significant overall treatment effect is found in a clinical trial, this does not necessarily indicate that no patients will benefit from treatment. Subgroup analyses could be conducted to investigate whether a treatment might still be beneficial for particular subgroups of patients. Assessment of the level of evidence associated with such subgroup findings is primordial as it may form the basis for performing a new clinical trial or even drawing the conclusion that a specific patient group could benefit from a new therapy. Previous research addressed the overall type I error and the power associated with a single subgroup finding for continuous outcomes and suitable replication strategies. The current study aims at investigating two scenarios as part of a nonconfirmatory strategy in a trial with dichotomous outcomes: (a) when a covariate of interest is represented by ordered subgroups, eg, in case of biomarkers, and thus, a trend can be studied that may reflect an underlying mechanism, and (b) when multiple covariates, and thus multiple subgroups, are investigated at the same time. Based on simulation studies, this paper assesses the credibility of subgroup findings in overall nonsignificant trials and provides practical recommendations for evaluating the strength of evidence of subgroup findings in these settings.

Inability of positive phase II clinical trials of investigational treatments to subsequently predict positive phase III clinical trials in glioblastoma

Background

Glioblastoma is the most common primary malignant brain tumor in adults, but effective therapies are lacking. With the scarcity of positive phase III trials, which are increasing in cost, we examined the ability of positive phase II trials to predict statistically significant improvement in clinical outcomes of phase III trials.

Methods

A PubMed search was conducted to identify phase III clinical trials performed in the past 25 years for patients with newly diagnosed or recurrent glioblastoma. Trials were excluded if they did not examine an investigational chemotherapy or agent, if they were stopped early owing to toxicity, if they lacked prior phase II studies, or if a prior phase II study was negative.

Results

Seven phase III clinical trials in newly diagnosed glioblastoma and 4 phase III clinical trials in recurrent glioblastoma met the inclusion criteria. Only 1 (9%) phase III study documented an improvement in overall survival and changed the standard of care.

Conclusion

The high failure rate of phase III trials demonstrates the urgent need to increase the reliability of phase II trials of treatments for glioblastoma. Strategies such as the use of adaptive trial designs, Bayesian statistics, biomarkers, volumetric imaging, and mathematical modeling warrant testing. Additionally, it is critical to increase our expectations of phase II trials so that positive findings increase the probability that a phase III trial will be successful.

[Biostatistical support for decision making in drug licensing and reimbursement exemplified by implications of heterogeneous findings in subgroups of the study population]

In the context of both drug licensing and reimbursement, the target population is sometimes restricted to a specific subgroup. In the setting of drug licensing the discussion concerns a negative benefit/risk assessment in a relevant subgroup. For reimbursement the debate involves the detection of an additional benefit compared with standard treatment, which can in some situations not be accepted for the overall study population. In their Methods Paper, the Institute for Quality and Efficiency in Health Care (IQWiG) refers to published articles that name criteria for the evaluation of credibility to claim a therapeutic effect on the basis of results in the subgroups of a study population (BMJ 340:850-854, 2010). A number of these criteria have found their way into the regulatory debate, which was recently published in a draft guideline of the European Medicines Agency (EMA). However, the significance of the interaction/heterogeneity test has been mentioned as one criterion for the credibility of a finding in a subgroup of the study population. This aspect is critically challenged in our paper. In our estimation, the application of this criterion hinders the critical discussion of whether a global treatment effect is applicable to relevant subgroups of a study population and the potential implications of this. We feel that biostatistical support for decision-making strategies should be the same in both worlds, even though in some instances the outcomes in a specific situation may be different, depending on the objective to be demonstrated.

Tutorial on statistical considerations on subgroup analysis in confirmatory clinical trials

Clinical trials target patients who are expected to benefit from a new treatment under investigation. However, the magnitude of the treatment benefit, if it exists, often depends on the patient baseline characteristics. It is therefore important to investigate the consistency of the treatment effect across subgroups to ensure a proper interpretation of positive study findings in the overall population. Such assessments can provide guidance on how the treatment should be used. However, great care has to be taken when interpreting consistency results. An observed heterogeneity in treatment effect across subgroups can arise because of chance alone, whereas true heterogeneity may be difficult to detect by standard statistical tests because of their low power. This tutorial considers issues related to subgroup analyses and their impact on the interpretation of findings of completed trials that met their main objectives. In addition, we provide guidance on the design and analysis of clinical trials that account for the expected heterogeneity of treatment effects across subgroups by establishing treatment benefit in a pre-defined targeted subgroup and/or the overall population. Copyright © 2016 John Wiley & Sons, Ltd.

General guidance on exploratory and confirmatory subgroup analysis in late-stage clinical trials

This article focuses on a broad class of statistical and clinical considerations related to the assessment of treatment effects across patient subgroups in late-stage clinical trials. This article begins with a comprehensive review of clinical trial literature and regulatory guidelines to help define scientifically sound approaches to evaluating subgroup effects in clinical trials. All commonly used types of subgroup analysis are considered in the article, including different variations of prospectively defined and post-hoc subgroup investigations. In the context of confirmatory subgroup analysis, key design and analysis options are presented, which includes conventional and innovative trial designs that support multi-population tailoring approaches. A detailed summary of exploratory subgroup analysis (with the purpose of either consistency assessment or subgroup identification) is also provided. The article promotes a more disciplined approach to post-hoc subgroup identification and formulates key principles that support reliable evaluation of subgroup effects in this setting.

Subgroup analyses in confirmatory clinical trials: time to be specific about their purposes

Background

It is well recognized that treatment effects may not be homogeneous across the study population. Subgroup analyses constitute a fundamental step in the assessment of evidence from confirmatory (Phase III) clinical trials, where conclusions for the overall study population might not hold. Subgroup analyses can have different and distinct purposes, requiring specific design and analysis solutions. It is relevant to evaluate methodological developments in subgroup analyses against these purposes to guide health care professionals and regulators as well as to identify gaps in current methodology.

Methods

We defined four purposes for subgroup analyses: (1) Investigate the consistency of treatment effects across subgroups of clinical importance, (2) Explore the treatment effect across different subgroups within an overall non-significant trial, (3) Evaluate safety profiles limited to one or a few subgroup(s), (4) Establish efficacy in the targeted subgroup when included in a confirmatory testing strategy of a single trial. We reviewed the methodology in line with this “purpose-based” framework. The review covered papers published between January 2005 and April 2015 and aimed to classify them in none, one or more of the aforementioned purposes.

Results

In total 1857 potentially eligible papers were identified. Forty-eight papers were selected and 20 additional relevant papers were identified from their references, leading to 68 papers in total. Nineteen were dedicated to purpose 1, 16 to purpose 4, one to purpose 2 and none to purpose 3. Seven papers were dedicated to more than one purpose, the 25 remaining could not be classified unambiguously. Purposes of the methods were often not specifically indicated, methods for subgroup analysis for safety purposes were almost absent and a multitude of diverse methods were developed for purpose (1).

Conclusions

It is important that researchers developing methodology for subgroup analysis explicitly clarify the objectives of their methods in terms that can be understood from a patient’s, health care provider’s and/or regulator’s perspective. A clear operational definition for consistency of treatment effects across subgroups is lacking, but is needed to improve the usability of subgroup analyses in this setting. Finally, methods to particularly explore benefit-risk systematically across subgroups need more research.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0122-6) contains supplementary material, which is available to authorized users.

Translational research: precision medicine, personalized medicine, targeted therapies: marketing or science?

Personalized medicine is based on: 1) improved clinical or non-clinical methods (including biomarkers) for a more discriminating and precise diagnosis of diseases; 2) targeted therapies of the choice or the best drug for each patient among those available; 3) dose adjustment methods to optimize the benefit-risk ratio of the drugs chosen; 4) biomarkers of efficacy, toxicity, treatment discontinuation, relapse, etc. Unfortunately, it is still too often a theoretical concept because of the lack of convenient diagnostic methods or treatments, particularly of drugs corresponding to each subtype of pathology, hence to each patient. Stratified medicine is a component of personalized medicine employing biomarkers and companion diagnostics to target the patients likely to present the best benefit-risk balance for a given active compound. The concept of targeted therapy, mostly used in cancer treatment, relies on the existence of a defined molecular target, involved or not in the pathological process, and/or on the existence of a biomarker able to identify the target population, which should logically be small as compared to the population presenting the disease considered. Targeted therapies and biomarkers represent important stakes for the pharmaceutical industry, in terms of market access, of return on investment and of image among the prescribers. At the same time, they probably represent only the first generation of products resulting from the combination of clinical, pathophysiological and molecular research, i.e. of translational research.