Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading?

Description of subgroup reporting in clinical trials of chronic diseases: a meta-epidemiological study

INTRODUCTION

In trials, subgroup analyses are used to examine whether treatment effects differ by important patient characteristics. However, which subgroups are most commonly reported has not been comprehensively described.

DESIGN AND SETTINGS

Using a set of trials identified from the US clinical trials register (ClinicalTrials.gov), we describe every reported subgroup for a range of conditions and drug classes.

METHODS

We obtained trial characteristics from ClinicalTrials.gov via the Aggregate Analysis of ClinicalTrials.gov database. We subsequently obtained all corresponding PubMed-indexed papers and screened these for subgroup reporting. Tables and text for reported subgroups were extracted and standardised using Medical Subject Headings and WHO Anatomical Therapeutic Chemical codes. Via logistic and Poisson regression models we identified independent predictors of result reporting (any vs none) and subgroup reporting (any vs none and counts). We then summarised subgroup reporting by index condition and presented all subgroups for all trials via a web-based interactive heatmap (https://ihwph-hehta.shinyapps.io/subgroup_reporting_app/).

RESULTS

Among 2235 eligible trials, 23% (524 trials) reported subgroups. Follow-up time (OR, 95%CI: 1.13, 1.04-1.24), enrolment (per 10-fold increment, 3.48, 2.25-5.47), trial starting year (1.07, 1.03-1.11) and specific index conditions (eg, hypercholesterolaemia, hypertension, taking asthma as the reference, OR ranged from 0.15 to 10.44), predicted reporting, sponsoring source and number of arms did not. Results were similar on modelling any result reporting (except number of arms, 1.42, 1.15-1.74) and the total number of subgroups. Age (51%), gender (45%), racial group (28%) were the most frequently reported subgroups. Characteristics related to the index condition (severity/duration/types etc) were frequently reported (eg, 69% of myocardial infarction trials reported on its severity/duration/types). However, reporting on comorbidity/frailty (five trials) and mental health (four trials) was rare.

CONCLUSION

Other than age, sex, race ethnicity or geographic location and characteristics related to the index condition, information on variation in treatment effects is sparse.

PROSPERO REGISTRATION NUMBER

CRD42018048202.

Systematic literature review of traits and outcomes reported in randomised controlled trials of asthma with regular dosing of inhaled corticosteroids with short-acting β2-agonist reliever, as-needed ICS/formoterol, or ICS/formoterol maintenance and reliever therapy

INTRODUCTION

Asthma treatments based solely on diagnostic label do not benefit patients equally. To identify patient traits that may be associated with improved treatment response to regular inhaled corticosteroid (ICSs) dosing with short-acting β2-agonist reliever or ICS/formoterol-containing therapy, a systematic literature review (SLR) was conducted.

METHODS

Searches of databases including MEDLINE and Embase identified randomised controlled trials (RCTs) of patients with asthma, aged ≥12 years, published 1998-2022, containing ≥1 regular ICS dosing or ICS/formoterol-containing treatment arm, and reporting patient traits and outcomes of interest. Relevant data was extracted and underwent a feasibility assessment to determine suitability for meta-analysis.

RESULTS

The SLR identified 39 RCTs of 72,740 patients and 90 treatment arms, reporting 11 traits and 11 outcomes. Five patient traits (age, body mass index, FEV1, smoking history, asthma control) and five outcomes (exacerbation rate, lung function, asthma control, adherence, time to first exacerbation) were deemed feasible for inclusion in meta-analyses due to sufficient comparable reporting. Subgroups of clinical outcomes stratified by levels of patient traits were reported in 16 RCTs.

CONCLUSION

A systematic review of studies of regular ICS dosing with SABA or ICS/formoterol-containing treatment strategies in asthma identified consistent reporting of five traits and outcomes, allowing exploration of associations with treatment response. Conversely, many other traits and outcomes, although being potentially relevant, were inconsistently reported and limited subgroup reporting meant analyses of treatment response for subgroups of traits was not possible. We recommend more consistent measurement and reporting of clinically relevant patient traits and outcomes in respiratory RCTs.

Overcoming combination fatigue: Addressing high-dimensional effect measure modification and interaction in clinical, biomedical, and epidemiologic research using multilevel analysis of individual heterogeneity and discriminatory accuracy (MAIHDA)

Growing interest in precision medicine, gene-environment interactions, health equity, expanding diversity in research, and the generalizability results, requires researchers to evaluate how the effects of treatments or exposures differ across numerous subgroups. Evaluating combination complexity, in the form of effect measure modification and interaction, is therefore a common study aim in the biomedical, clinical, and epidemiologic sciences. There is also substantial interest in expanding the combinations of factors analyzed to include complex treatment protocols (e.g., multiple study arms or factorial randomization), comorbid medical conditions or risk factors, and sociodemographic and other subgroup identifiers. However, expanding the number of subgroup category combinations creates combination fatigue problems, including concerns over small sample size, reduced power, multiple testing, spurious results, and design and analytic complexity. Creative new approaches for managing combination fatigue and evaluating high-dimensional effect measure modification and interaction are needed. Intersectional MAIHDA (multilevel analysis of individual heterogeneity and discriminatory accuracy) has already attracted substantial interest in social epidemiology, and has been hailed as the new gold standard for investigating health inequities across complex intersections of social identity. Leveraging the inherent advantages of multilevel models, a more general multicategorical MAIHDA can be used to study statistical interactions and predict effects across high-dimensional combinations of conditions, with important advantages over alternative approaches. Though it has primarily been used thus far as an analytic approach, MAIHDA should also be used as a framework for study design. In this article, I introduce MAIHDA to the broader health sciences research community, discuss its advantages over conventional approaches, and provide an overview of potential applications in clinical, biomedical, and epidemiologic research.

Treatment effect modification due to comorbidity: Individual participant data meta-analyses of 120 randomised controlled trials

BACKGROUND

People with comorbidities are underrepresented in clinical trials. Empirical estimates of treatment effect modification by comorbidity are lacking, leading to uncertainty in treatment recommendations. We aimed to produce estimates of treatment effect modification by comorbidity using individual participant data (IPD).

METHODS AND FINDINGS

We obtained IPD for 120 industry-sponsored phase 3/4 trials across 22 index conditions (n = 128,331). Trials had to be registered between 1990 and 2017 and have recruited ≥300 people. Included trials were multicentre and international. For each index condition, we analysed the outcome most frequently reported in the included trials. We performed a two-stage IPD meta-analysis to estimate modification of treatment effect by comorbidity. First, for each trial, we modelled the interaction between comorbidity and treatment arm adjusted for age and sex. Second, for each treatment within each index condition, we meta-analysed the comorbidity-treatment interaction terms from each trial. We estimated the effect of comorbidity measured in 3 ways: (i) the number of comorbidities (in addition to the index condition); (ii) presence or absence of the 6 commonest comorbid diseases for each index condition; and (iii) using continuous markers of underlying conditions (e.g., estimated glomerular filtration rate (eGFR)). Treatment effects were modelled on the usual scale for the type of outcome (absolute scale for numerical outcomes, relative scale for binary outcomes). Mean age in the trials ranged from 37.1 (allergic rhinitis trials) to 73.0 (dementia trials) and percentage of male participants range from 4.4% (osteoporosis trials) to 100% (benign prostatic hypertrophy trials). The percentage of participants with 3 or more comorbidities ranged from 2.3% (allergic rhinitis trials) to 57% (systemic lupus erythematosus trials). We found no evidence of modification of treatment efficacy by comorbidity, for any of the 3 measures of comorbidity. This was the case for 20 conditions for which the outcome variable was continuous (e.g., change in glycosylated haemoglobin in diabetes) and for 3 conditions in which the outcomes were discrete events (e.g., number of headaches in migraine). Although all were null, estimates of treatment effect modification were more precise in some cases (e.g., sodium-glucose co-transporter-2 (SGLT2) inhibitors for type 2 diabetes-interaction term for comorbidity count 0.004, 95% CI -0.01 to 0.02) while for others credible intervals were wide (e.g., corticosteroids for asthma-interaction term -0.22, 95% CI -1.07 to 0.54). The main limitation is that these trials were not designed or powered to assess variation in treatment effect by comorbidity, and relatively few trial participants had >3 comorbidities.

CONCLUSIONS

Assessments of treatment effect modification rarely consider comorbidity. Our findings demonstrate that for trials included in this analysis, there was no empirical evidence of treatment effect modification by comorbidity. The standard assumption used in evidence syntheses is that efficacy is constant across subgroups, although this is often criticised. Our findings suggest that for modest levels of comorbidities, this assumption is reasonable. Thus, trial efficacy findings can be combined with data on natural history and competing risks to assess the likely overall benefit of treatments in the context of comorbidity.

Quantile Treatment Effect of Zinc Lozenges on Common Cold Duration: A Novel Approach to Analyze the Effect of Treatment on Illness Duration

Calculation of the difference of means is the most common approach when analyzing treatment effects on continuous outcomes. Nevertheless, it is possible that the treatment has a different effect on patients who have a lower value of the outcome compared with patients who have a greater value of the outcome. The estimation of quantile treatment effects (QTEs) allows the analysis of treatment effects over the entire distribution of a continuous outcome, such as the duration of illness or the duration of hospital stay. Furthermore, most of these outcomes have asymmetric distributions with fat tails, and censored observations are not uncommon. These features can be accounted for in the analysis of the QTE. In this paper, we use the QTE approach to analyze the effect of zinc lozenges on common cold duration. We use the data set of the Mossad (1996) trial with zinc gluconate lozenges, and three data sets of trials with zinc acetate lozenges. In the Mossad (1996) trial, zinc gluconate lozenges shortened common cold duration on average by 4.0 days (95% CI 2.3-5.7 days). However, the QTE analysis indicates that 15- to 17-day colds were shortened by 8 days, and 2-day colds by just 1 day, for the group taking zinc lozenges. Thus, the overall 4.0-day average effect of zinc gluconate lozenges in the Mossad (1996) trial is inconsistent with our QTE findings for both short and long colds. Similar results were found in our QTE analysis of the pooled data sets of the three zinc acetate lozenge trials. The average effect of 2.7 days (95% CI 1.8-3.3 days) was inconsistent with the effects on short and long colds. The QTE approach may have broad usefulness for examining treatment effects on the duration of illness and hospital stay, and on other similar outcomes.

Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Explanation and Elaboration: A Report of the ISPOR CHEERS II Good Practices Task Force

Health economic evaluations are comparative analyses of alternative courses of action in terms of their costs and consequences. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement, published in 2013, was created to ensure health economic evaluations are identifiable, interpretable, and useful for decision making. It was intended as guidance to help authors report accurately which health interventions were being compared and in what context, how the evaluation was undertaken, what the findings were, and other details that may aid readers and reviewers in interpretation and use of the study. The new CHEERS 2022 statement replaces the previous CHEERS reporting guidance. It reflects the need for guidance that can be more easily applied to all types of health economic evaluation, new methods and developments in the field, and the increased role of stakeholder involvement including patients and the public. It is also broadly applicable to any form of intervention intended to improve the health of individuals or the population, whether simple or complex, and without regard to context (such as healthcare, public health, education, and social care). This Explanation and Elaboration Report presents the new CHEERS 2022 28-item checklist with recommendations and explanation and examples for each item. The CHEERS 2022 statement is primarily intended for researchers reporting economic evaluations for peer-reviewed journals and the peer reviewers and editors assessing them for publication. Nevertheless, we anticipate familiarity with reporting requirements will be useful for analysts when planning studies. It may also be useful for health technology assessment bodies seeking guidance on reporting, given that there is an increasing emphasis on transparency in decision making.

Development of the Instrument to assess the Credibility of Effect Modification Analyses (ICEMAN) in randomized controlled trials and meta-analyses

BACKGROUND

Most randomized controlled trials (RCTs) and meta-analyses of RCTs examine effect modification (also called a subgroup effect or interaction), in which the effect of an intervention varies by another variable (e.g., age or disease severity). Assessing the credibility of an apparent effect modification presents challenges; therefore, we developed the Instrument for assessing the Credibility of Effect Modification Analyses (ICEMAN).

METHODS

To develop ICEMAN, we established a detailed concept; identified candidate credibility considerations in a systematic survey of the literature; together with experts, performed a consensus study to identify key considerations and develop them into instrument items; and refined the instrument based on feedback from trial investigators, systematic review authors and journal editors, who applied drafts of ICEMAN to published claims of effect modification.

RESULTS

The final instrument consists of a set of preliminary considerations, core questions (5 for RCTs, 8 for meta-analyses) with 4 response options, 1 optional item for additional considerations and a rating of credibility on a visual analogue scale ranging from very low to high. An accompanying manual provides rationales, detailed instructions and examples from the literature. Seventeen potential users tested ICEMAN; their suggestions improved the user-friendliness of the instrument.

INTERPRETATION

The Instrument for assessing the Credibility of Effect Modification Analyses offers explicit guidance for investigators, systematic reviewers, journal editors and others considering making a claim of effect modification or interpreting a claim made by others.

A systematic review of subgroup analyses in randomised clinical trials in cardiovascular disease

Background:

Subgroup analyses are frequently used to assess heterogeneity of treatment effects in randomised clinical trials. Inconsistent, improper and incomplete implementation, reporting and interpretation have been identified as ongoing challenges. Further, subgroup analyses were frequently criticised because of unreliable or potentially misleading results. More recently, recommendations and guidelines have been provided to improve the reporting of data in this regard.

Methods:

This systematic review was based on a literature search within the digital archives of three selected medical journals, The New England Journal of Medicine, The Lancet and Circulation. We reviewed articles of randomised clinical trials in the domain of cardiovascular disease which were published in 2015 and 2016. We screened and evaluated the selected articles for the mode of implementation and reporting of subgroup analyses.

Results:

We were able to identify a total of 130 eligible publications of randomised clinical trials. In 89/130 (68%) articles, results of at least one subgroup analysis were presented. This was dependent on the considered journal (p < 0.001), the number of included patients (p < 0.001) and the lack of statistical significance of a trial’s primary analysis (p < 0.001). The number of reported subgroup analyses ranged from 1 to 101 (median = 13). We were able to comprehend the specification time of reported subgroup analyses for 71/89 (80%) articles, with 55/89 (62%) articles presenting exclusively pre-specified analyses. This information was not always traceable on the basis of provided trial protocols and often did not include the pre-definition of cut-off values for the categorization of subgroups. The use of interaction tests was reported in 84/89 (94%) articles, with 36/89 (40%) articles reporting heterogeneity of the treatment effect for at least one primary or secondary trial outcome. Subgroup analyses were reported more frequently for larger randomised clinical trials, and if primary analyses did not reach statistical significance. Information about the implementation of subgroup analyses was reported most consistently for articles from The New England Journal of Medicine, since it was also traceable on the basis of provided trial protocols. We were able to comprehend whether subgroup analyses were pre-specified in a majority of the reviewed publications. Even though results of multiple subgroup analyses were reported for most published trials, a corresponding adjustment for multiple testing was rarely considered.

Conclusion:

Compared to previous reviews in this context, we observed improvements in the reporting of subgroup analyses of cardiovascular randomised clinical trials. Nonetheless, critical shortcomings, such as inconsistent reporting of the implementation and insufficient pre-specification, persist.

Reporting and interpretation of subgroup analyses in heart failure randomized controlled trials

Aims

This study aimed to investigate the reporting of subgroup analyses in heart failure (HF) randomized controlled trials (RCTs) and to determine the strength and credibility of subgroup claims.

Methods and results

All primary HF RCTs published in nine high‐impact journals from 1 January 2008 to 31 December 2017 were included. Multivariable regression analysis was used to identify factors that may favour the reporting of results in specific subgroups. Strength of the subgroup effect claimed was classified into (i) strong, (ii) likely, or (iii) suggestive. Credibility of subgroup claim was scored using a pre‐specified 10 pointer criteria. Of the 261 HF RCTs studied, 107 (41%) reported subgroup analyses. Twenty‐five (23%) RCTs claimed a subgroup effect for the primary outcome of which six (24%) made a strong claim, eight (32%) claimed a likely effect, and 11 (44%) suggested a possible subgroup effect. Seven of the 25 RCTs did not employ interaction testing for subgroup claims of the primary outcome. Three out of 10 pre‐specified credibility criteria were satisfied by half of the trials. Fourteen trials justified the choice of subgroups, and 10 explicitly stated they were underpowered to detect differences within subgroups. Source of funding did not influence the frequency of reporting subgroup analyses (OR 0.53, 95% CI 0.78–3.62, P = 0.52).

Conclusions

Appropriate credibility criteria were rarely met even by HF RCTs that held strong subgroup claims. Subgroup analyses should be pre‐specified, be adequately powered, present interaction terms, and be replicated in independent data before being integrated into clinical decision making.

A tutorial on methodological studies: the what, when, how and why

BACKGROUND

Methodological studies - studies that evaluate the design, analysis or reporting of other research-related reports - play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

MAIN BODY

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a "frequently asked questions" format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

CONCLUSION

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Characteristics and interpretation of subgroup analyses based on tumour characteristics in randomised trials testing target-specific anticancer drugs: design of a systematic survey

BACKGROUND

Target-specific anticancer drugs are under rapid development. Little is known, however, about the risk of administering target-specific drugs to patients who have tumours with molecular alterations or other characteristics that can make the drug ineffective or even harmful. An increasing number of randomised clinical trials (RCTs) investigating target-specific anticancer drugs include subgroup analyses based on tumour characteristics. Such subgroup analyses have the potential to be more credible and influential than subgroup analyses based on traditional factors such as sex or tumour stage. In addition, they may more frequently lead to qualitative subgroup effects, that is, show benefit in one but harm in another subgroup of patients (eg, if the tumour characteristic makes the drug ineffective or even enhance tumour growth). If so, subgroup analyses based on tumour characteristics would be highly relevant for patient safety. The aim of this study is to systematically assess the frequency and characteristics of subgroup analyses based on tumour characteristics, the frequency of qualitative subgroup effects, their credibility, and the interpretations that investigators and guidelines developers report.

METHODS AND ANALYSIS

We will perform a systematic survey of 433 RCTs testing the effect of target-specific anticancer drugs. Teams of methodologically trained investigators and oncologists will identify eligible studies, extract relevant data and assess the credibility of putative subgroup effects using a recently developed formal instrument. We will systematically assess how trial investigators interpret apparent subgroup effects based on tumour characteristics and the extent to which they influence subsequent practice guidelines. Our results will provide empirical data characterising an increasingly used type of subgroup analysis in cancer trials and its potential impact on precision medicine to predict benefit or harm.

ETHICS AND DISSEMINATION

Formal ethical approval is not required for this study. We will disseminate the findings in a peer-reviewed and open-access journal publication.

Heterogeneous treatment effects of a text messaging smoking cessation intervention among university students

Introduction

Despite tobacco being an important preventable factor with respect to ill health and death, it is a legal substance that harms and kills many of those who use it. Text messaging smoking cessation interventions have been evaluated in a variety of contexts, and are generally considered to have a positive effect on smoking cessation success. In order for text messaging interventions to continue to be useful as prevalence of smoking decreases, it may be necessary to tailor the interventions to specific individuals. However, little is known with regard to who benefits the most and least from existing interventions.

Methods

In order to identify heterogenous treatment effects, we analyzed data from a randomized controlled trial of a text messaging smoking cessation intervention targeting university students in Sweden. We used a Bayesian hierarchical model where the outcome was modelled using logistic regression, and so-called horseshoe priors were used for coefficients. Predictive performance of the model, and heterogeneous treatment effects, were calculated using cross-validation over the trial data.

Results

Findings from the study of heterogenous treatment effects identified less effect of the intervention among university students with stronger dependence of nicotine and students who smoke a greater quantity of cigarettes per week. No heterogeneity was found with respect to sex, number of years smoking, or the use of snuff.

Discussion

Results emphasize that individuals with a more developed dependence of nicotine may have a harder time quitting smoking even with support. This questions the dissemination and development of text messaging interventions to university students in the future, as they may not be the optimal choice of intervention for those with a more developed dependence. On the other hand, text messaging interventions may be useful to disseminate among university students that are at risk of developing a strong dependence.

Trial registration

International Standard Randomized Controlled Trial Number (ISRCTN): 75766527; http://www.controlled-trials.com/ISRCTN75766527.

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement

Heterogeneity of treatment effect (HTE) refers to the nonrandom variation in the magnitude or direction of a treatment effect across levels of a covariate, as measured on a selected scale, against a clinical outcome. In randomized controlled trials (RCTs), HTE is typically examined through a subgroup analysis that contrasts effects in groups of patients defined "1 variable at a time" (for example, male vs. female or old vs. young). The authors of this statement present guidance on an alternative approach to HTE analysis, "predictive HTE analysis." The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risks with versus without the intervention, taking into account all relevant patient attributes simultaneously. The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed using a multidisciplinary technical expert panel, targeted literature reviews, simulations to characterize potential problems with predictive approaches, and a deliberative process engaging the expert panel. The authors distinguish 2 categories of predictive HTE approaches: a "risk-modeling" approach, wherein a multivariable model predicts the risk for an outcome and is applied to disaggregate patients within RCTs to define risk-based variation in benefit, and an "effect-modeling" approach, wherein a model is developed on RCT data by incorporating a term for treatment assignment and interactions between treatment and baseline covariates. Both approaches can be used to predict differential absolute treatment effects, the most relevant scale for clinical decision making. The authors developed 4 sets of guidance: criteria to determine when risk-modeling approaches are likely to identify clinically important HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. The PATH Statement, together with its explanation and elaboration document, may guide future analyses and reporting of RCTs.

The Predictive Approaches to Treatment effect Heterogeneity (PATH) Statement: Explanation and Elaboration

The PATH (Predictive Approaches to Treatment effect Heterogeneity) Statement was developed to promote the conduct of, and provide guidance for, predictive analyses of heterogeneity of treatment effects (HTE) in clinical trials. The goal of predictive HTE analysis is to provide patient-centered estimates of outcome risk with versus without the intervention, taking into account all relevant patient attributes simultaneously, to support more personalized clinical decision making than can be made on the basis of only an overall average treatment effect. The authors distinguished 2 categories of predictive HTE approaches (a "risk-modeling" and an "effect-modeling" approach) and developed 4 sets of guidance statements: criteria to determine when risk-modeling approaches are likely to identify clinically meaningful HTE, methodological aspects of risk-modeling methods, considerations for translation to clinical practice, and considerations and caveats in the use of effect-modeling approaches. They discuss limitations of these methods and enumerate research priorities for advancing methods designed to generate more personalized evidence. This explanation and elaboration document describes the intent and rationale of each recommendation and discusses related analytic considerations, caveats, and reservations.

Precision Medicine in Critical Illness: Sepsis and Acute Respiratory Distress Syndrome

Sepsis and the acute respiratory distress syndrome (ARDS) each cause substantial morbidity and mortality. In contrast to other lung diseases, the entire course of disease in these syndromes is measured in days to weeks rather than months to years, which raises unique challenges in achieving precision medicine. We review advances in sepsis and ARDS resulting from omics studies, including those involving genome-wide association, gene expression, targeted proteomics, and metabolomics approaches. We focus on promising evidence of biological subtypes in both sepsis and ARDS that consistently display high risk for death. In sepsis, a gene expression signature with dysregulated adaptive immune signaling has evidence for a differential response to systemic steroid therapy, whereas in ARDS, a hyperinflammatory pattern identified in plasma using targeted proteomics responded more favorably to randomized interventions including high positive end-expiratory pressure, volume conservative fluid therapy, and simvastatin therapy. These early examples suggest heterogeneous biology that may be challenging to detect by clinical factors alone and speak to the promise of a precision approach that targets the right treatment at the right time to the right patient.

Guidance from key organisations on exploring, confirming and interpreting subgroup effects of medical treatments: a scoping review

OBJECTIVES

With the increasing interest in personalised medicine, the use of subgroup analyses is likely to increase. Subgroup analyses are challenging and often misused, possibly leading to false interpretations of the effect. It remains unclear to what extent key organisations warn for such pitfalls and translate current methodological research to detect these effects into research guidelines. The aim of this scoping review is to determine and evaluate the current guidance used by organisations for exploring, confirming and interpreting subgroup effects.

DESIGN

Scoping review.

ELIGIBILITY CRITERIA

We identified four types of key stakeholder organisations: industry, health technology assessment organisations (HTA), academic/non-profit research organisations and regulatory bodies. After literature search and expert consultation, we identified international and national organisations of each type. For each organisation that was identified, we searched for official research guidance documents and contacted the organisation for additional guidance.

RESULTS

Twenty-seven (45%) of the 60 organisations that we included had relevant research guidance documents. We observed large differences between organisation types: 18% (n=2) of the industry organisations, 64% (n=9) of the HTA organisations, 38% (n=8) of academic/non-profit research organisations and 57% (n=8) of regulatory bodies provided guidance documents. The majority of the documents (n=33, 63%) mentioned one or more challenges in subgroup analyses, such as false positive findings or ecological bias with variations across the organisation types. Statistical recommendations were less common (n=19, 37%) and often limited to a formal test of interaction.

CONCLUSIONS

Almost half of the organisations included in this scoping review provided guidance on subgroup effect research in their guidelines. However, there were large differences between organisations in the amount and level of detail of their guidance. Effort is required to translate and integrate research findings on subgroup analysis to practical guidelines for decision making and to reduce the differences between organisations and organisation types.

Understanding of interaction (subgroup) analysis in clinical trials

BACKGROUND

When the treatment effect on the outcome of interest is influenced by a baseline/demographic factor, investigators say that an interaction is present. In randomized clinical trials (RCTs), this type of analysis is typically referred to as subgroup analysis. Although interaction (or subgroup) analyses are usually stated as a secondary study objective, it is not uncommon that these results lead to changes in treatment protocols or even modify public health policies. Nonetheless, recent reviews have indicated that their proper assessment, interpretation and reporting remain challenging.

RESULTS

Therefore, this article provides an overview of these challenges, to help investigators find the best strategy for application of interaction analyses on binary outcomes in RCTs. Specifically, we discuss the key points of formal interaction testing, including the estimation of both additive and multiplicative interaction effects. We also provide recommendations that, if adhered to, could increase the clarity and the completeness of reports of RCTs.

CONCLUSION

Altogether, this article provides a brief non-statistical guide for clinical investigators on how to perform, interpret and report interaction (subgroup) analyses in RCTs.

A systematic survey identified 36 criteria for assessing effect modification claims in randomized trials or meta-analyses

OBJECTIVE

The objective of the study was to systematically survey the methodological literature and collect suggested criteria for assessing the credibility of effect modification and associated rationales.

STUDY DESIGN AND SETTING

We searched MEDLINE, Embase, and WorldCat up to March 2018 for publications providing guidance for assessing the credibility of effect modification identified in randomized trials or meta-analyses. Teams of two investigators independently identified eligible publications and extracted credibility criteria and authors' rationale, reaching consensus through discussion. We created a taxonomy of criteria that we iteratively refined during data abstraction.

RESULTS

We identified 150 eligible publications that provided 36 criteria and associated rationales. Frequent criteria included significant test for interaction (n = 54), a priori hypothesis (n = 49), providing a causal explanation (n = 47), accounting for multiplicity (n = 45), testing a small number of effect modifiers (n = 38), and prespecification of analytic details (n = 39). For some criteria, we found more than one rationale; some criteria were connected through a common rationale. For some criteria, experts disagreed regarding their suitability (e.g., added value of stratified randomization; trustworthiness of biologic rationales).

CONCLUSION

Methodologists have expended substantial intellectual energy providing criteria for critical appraisal of apparent effect modification. Our survey highlights popular criteria, expert agreement and disagreement, and where more work is needed, including testing criteria in practice.

Personalized evidence based medicine: predictive approaches to heterogeneous treatment effects

The use of evidence from clinical trials to support decisions for individual patients is a form of "reference class forecasting": implicit predictions for an individual are made on the basis of outcomes in a reference class of "similar" patients treated with alternative therapies. Evidence based medicine has generally emphasized the broad reference class of patients qualifying for a trial. Yet patients in a trial (and in clinical practice) differ from one another in many ways that can affect the outcome of interest and the potential for benefit. The central goal of personalized medicine, in its various forms, is to narrow the reference class to yield more patient specific effect estimates to support more individualized clinical decision making. This article will review fundamental conceptual problems with the prediction of outcome risk and heterogeneity of treatment effect (HTE), as well as the limitations of conventional (one-variable-at-a-time) subgroup analysis. It will also discuss several regression based approaches to "predictive" heterogeneity of treatment effect analysis, including analyses based on "risk modeling" (such as stratifying trial populations by their risk of the primary outcome or their risk of serious treatment-related harms) and analysis based on "effect modeling" (which incorporates modifiers of relative effect). It will illustrate these approaches with clinical examples and discuss their respective strengths and vulnerabilities.

Perspective: Limiting Dependence on Nonrandomized Studies and Improving Randomized Trials in Human Nutrition Research: Why and How

A large majority of human nutrition research uses nonrandomized observational designs, but this has led to little reliable progress. This is mostly due to many epistemologic problems, the most important of which are as follows: difficulty detecting small (or even tiny) effect sizes reliably for nutritional risk factors and nutrition-related interventions; difficulty properly accounting for massive confounding among many nutrients, clinical outcomes, and other variables; difficulty measuring diet accurately; and suboptimal research reporting. Tiny effect sizes and massive confounding are largely unfixable problems that narrowly confine the scenarios in which nonrandomized observational research is useful. Although nonrandomized studies and randomized trials have different priorities (assessment of long-term causality compared with assessment of treatment effects), the odds for obtaining reliable information with the former are limited. Randomized study designs should therefore largely replace nonrandomized studies in human nutrition research going forward. To achieve this, many of the limitations that have traditionally plagued most randomized trials in nutrition, such as small sample size, short length of follow-up, high cost, and selective reporting, among others, must be overcome. Pivotal megatrials with tens of thousands of participants and lifelong follow-up are possible in nutrition science with proper streamlining of operational costs. Fixable problems that have undermined observational research, such as dietary measurement error and selective reporting, need to be addressed in randomized trials. For focused questions in which dietary adherence is important to maximize, trials with direct observation of participants in experimental in-house settings may offer clean answers on short-term metabolic outcomes. Other study designs of randomized trials to consider in nutrition include registry-based designs and "N-of-1" designs. Mendelian randomization designs may also offer some more reliable leads for testing interventions in trials. Collectively, an improved randomized agenda may clarify many things in nutrition science that might never be answered credibly with nonrandomized observational designs.

Reporting of heterogeneity of treatment effect in cohort studies: a review of the literature

Background

This article corresponds to a literature review and analyze how heterogeneity of treatment (HTE) is reported and addressed in cohort studies and to evaluate the use of the different measures to HTE analysis.

Methods

prospective cohort studies, in English language, measuring the effect of a treatment (pharmacological, interventional, or other) published among 119 core clinical journals (defined by the National Library of Medicine) in the last 16 years were selected in the following data source: Medline. One reviewer randomly sampled journal articles with 1: 1 stratification by journal type: high impact journals (the New England Journal of Medicine, JAMA, LANCET, Annals of Internal Medicine, BMJ and Plos Medicine) and low impact journal (the remaining journals) to identify 150 eligible studies. Two reviewers independently and in duplicate used standardized piloted forms to screen study reports for eligibility and to extract data. They also used explicit criteria to determine whether a cohort study reported HTE analysis. Logistic regression was used to examine the association of prespecified study characteristics with reporting versus not reporting of heterogeneity of treatment effect.

Results

One hundred fifty cohort studies were included of which 88 (58%) reported HTE analysis. High impact journals (Odds Ratio: 3.5, 95% CI: 1.78–7.5; P < 0.001), pharmacological studies (Odds Ratio: 0.26, 95% CI: 0.13–0.51; P < 0.001) and studies published after 2014 (Odds Ratio: 0.5, 95% CI: 0.25–0.97; P = 0.004) were associated with more frequent reporting of HTE. 27 (31%) studies which reported HTE used an interaction test.

Conclusion

More than half cohort studies report some measure of heterogeneity of treatment effect. Prospective cohort studies published in high impact journals, with large sample size, or studying a pharmacological treatment are associated with more frequent HTE reporting. The source of funding was not associated with HTE reporting. There is a need for guidelines on how to perform HTE analyses in cohort studies.

Electronic supplementary material

The online version of this article (10.1186/s12874-017-0466-6) contains supplementary material, which is available to authorized users.

Oral diabetes medication monotherapy and short-term mortality in individuals with type 2 diabetes and coronary artery disease

OBJECTIVE

To determine whether sulfonylurea use, compared with non-sulfonylurea oral diabetes medication use, was associated with 2-year mortality in individuals with well-controlled diabetes and coronary artery disease (CAD).

RESEARCH DESIGN AND METHODS

We studied 5352 US veterans with type 2 diabetes, obstructive CAD on coronary angiography, hemoglobin A1c ≤7.5% at the time of catheterization, and taking zero or one oral diabetes medication (categorized as no medications, non-sulfonylurea medication, or sulfonylurea). We estimated the association between medication category and 2-year mortality using inverse probability of treatment-weighted (IPW) standardized mortality differences and IPW multivariable Cox proportional hazards regression.

RESULTS

49%, 35%, and 16% of the participants were on no diabetes medications, non-sulfonylurea medications, and sulfonylureas, respectively. In individuals on no medications, non-sulfonylurea medications, and sulfonylureas, the unadjusted mortality rates were 6.6%, 5.2%, and 11.9%, respectively, and the IPW-standardized mortality rates were 5.9%, 6.5%, and 9.7%, respectively. The standardized absolute 2-year mortality difference between non-sulfonylurea and sulfonylurea groups was 3.2% (95% CI 0.7 to 5.7) (p=0.01). In Cox proportional hazards models, the point estimate suggested that sulfonylurea use might be associated with greater hazard of mortality than non-sulfonylurea medication use, but this finding was not statistically significant (HR 1.38 (95% CI 1.00 to 1.93), p=0.05). We did not observe significant mortality differences between individuals on no diabetes medications and non-sulfonylurea users.

CONCLUSIONS

Sulfonylurea use was common (nearly one-third of those taking medications) and was associated with increased 2-year mortality in individuals with obstructive CAD. The significance of the association between sulfonylurea use and mortality was attenuated in fully adjusted survival models. Caution with sulfonylurea use may be warranted for patients with well-controlled diabetes and CAD, and metformin or newer diabetes medications with cardiovascular safety data could be considered as alternatives when individualizing therapy.

Subgroup analyses with special reference to the effect of antiplatelet agents in acute coronary syndromes

Controlled trials estimate treatment effects averaged over the reference population of subjects. However, physicians are interested in whether the treatment effect varies across subgroups (effect heterogeneity) in order to target specific subgroups to maximise the benefit of treatment and minimise harm. Therefore, large clinical trials of antiplatelet agents include subgroup analyses that examine whether treatment effects differ between subgroups of subjects identified by baseline characteristics. Reporting subgroup is pervasive and often accompanied by claims of difference of treatment effects between subgroups with potential important implications for clinical practice. However, subgroup-specific analyses of clinical trial data have inherent limitations that reduce their reliability. These include reduced statistical power, failure to specify the subgroups of interest a priori, failure to account for examining large numbers of subgroups, lack of strong rationale for biological response modification, and performing analyses based on variables measured post randomisation or in trials showing no overall difference between treatments. Rules for interpretation of subgroup findings in subgroups have been suggested but are frequently not applied. In this article we draw attention to the pitfalls of subgroup analyses in the context of recent trials of antiplatelet agents.

The proposed 'concordance-statistic for benefit' provided a useful metric when modeling heterogeneous treatment effects

OBJECTIVES

Clinical prediction models that support treatment decisions are usually evaluated for their ability to predict the risk of an outcome rather than treatment benefit-the difference between outcome risk with vs. without therapy. We aimed to define performance metrics for a model's ability to predict treatment benefit.

STUDY DESIGN AND SETTING

We analyzed data of the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery (SYNTAX) trial and of three recombinant tissue plasminogen activator trials. We assessed alternative prediction models with a conventional risk concordance-statistic (c-statistic) and a novel c-statistic for benefit. We defined observed treatment benefit by the outcomes in pairs of patients matched on predicted benefit but discordant for treatment assignment. The 'c-for-benefit' represents the probability that from two randomly chosen matched patient pairs with unequal observed benefit, the pair with greater observed benefit also has a higher predicted benefit.

RESULTS

Compared to a model without treatment interactions, the SYNTAX score II had improved ability to discriminate treatment benefit (c-for-benefit 0.590 vs. 0.552), despite having similar risk discrimination (c-statistic 0.725 vs. 0.719). However, for the simplified stroke-thrombolytic predictive instrument (TPI) vs. the original stroke-TPI, the c-for-benefit (0.584 vs. 0.578) was similar.

CONCLUSION

The proposed methodology has the potential to measure a model's ability to predict treatment benefit not captured with conventional performance metrics.

Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials

IMPORTANCE

Many published randomized clinical trials (RCTs) make claims for subgroup differences.

OBJECTIVE

To evaluate how often subgroup claims reported in the abstracts of RCTs are actually supported by statistical evidence (P < .05 from an interaction test) and corroborated by subsequent RCTs and meta-analyses.

DATA SOURCES

This meta-epidemiological survey examines data sets of trials with at least 1 subgroup claim, including Subgroup Analysis of Trials Is Rarely Easy (SATIRE) articles and Discontinuation of Randomized Trials (DISCO) articles. We used Scopus (updated July 2016) to search for English-language articles citing each of the eligible index articles with at least 1 subgroup finding in the abstract.

STUDY SELECTION

Articles with a subgroup claim in the abstract with or without evidence of statistical heterogeneity (P < .05 from an interaction test) in the text and articles attempting to corroborate the subgroup findings.

DATA EXTRACTION AND SYNTHESIS

Study characteristics of trials with at least 1 subgroup claim in the abstract were recorded. Two reviewers extracted the data necessary to calculate subgroup-level effect sizes, standard errors, and the P values for interaction. For individual RCTs and meta-analyses that attempted to corroborate the subgroup findings from the index articles, trial characteristics were extracted. Cochran Q test was used to reevaluate heterogeneity with the data from all available trials.

MAIN OUTCOMES AND MEASURES

The number of subgroup claims in the abstracts of RCTs, the number of subgroup claims in the abstracts of RCTs with statistical support (subgroup findings), and the number of subgroup findings corroborated by subsequent RCTs and meta-analyses.

RESULTS

Sixty-four eligible RCTs made a total of 117 subgroup claims in their abstracts. Of these 117 claims, only 46 (39.3%) in 33 articles had evidence of statistically significant heterogeneity from a test for interaction. In addition, out of these 46 subgroup findings, only 16 (34.8%) ensured balance between randomization groups within the subgroups (eg, through stratified randomization), 13 (28.3%) entailed a prespecified subgroup analysis, and 1 (2.2%) was adjusted for multiple testing. Only 5 (10.9%) of the 46 subgroup findings had at least 1 subsequent pure corroboration attempt by a meta-analysis or an RCT. In all 5 cases, the corroboration attempts found no evidence of a statistically significant subgroup effect. In addition, all effect sizes from meta-analyses were attenuated toward the null.

CONCLUSIONS AND RELEVANCE

A minority of subgroup claims made in the abstracts of RCTs are supported by their own data (ie, a significant interaction effect). For those that have statistical support (P < .05 from an interaction test), most fail to meet other best practices for subgroup tests, including prespecification, stratified randomization, and adjustment for multiple testing. Attempts to corroborate statistically significant subgroup differences are rare; when done, the initially observed subgroup differences are not reproduced.

Sex based subgroup differences in randomized controlled trials: empirical evidence from Cochrane meta-analyses

OBJECTIVE

 To evaluate the frequency, validity, and relevance of statistically significant (P<0.05) sex-treatment interactions in randomized controlled trials in Cochrane meta-analyses.

DESIGN

 Meta-epidemiological study.

DATA SOURCES

 Cochrane Database of Systematic Reviews (CDSR) and PubMed.

ELIGIBILITY CRITERIA FOR STUDY SELECTION

 Reviews published in the CDSR with sex-treatment subgroup analyses in the forest plots, using data from randomized controlled trials.

DATA EXTRACTION

 Information on the study design and sex subgroup data were extracted from reviews and forest plots that met inclusion criteria. For each statistically significant sex-treatment interaction, the potential for biological plausibility and clinical significance was considered.

RESULTS

 Among the 41 reviews with relevant data, there were 109 separate treatment-outcome analyses ("topics"). Among the 109 topics, eight (7%) had a statistically significant sex-treatment interaction. The 109 topics included 311 randomized controlled trials (162 with both sexes, 46 with males only, 103 with females only). Of the 162 individual randomized controlled trials that included both sexes, 15 (9%) had a statistically significant sex-treatment interaction. Of four topics where the first published randomized controlled trial had a statistically significant sex-treatment interaction, no meta-analyses that included other randomized controlled trials retained the statistical significance and no meta-analyses showed statistical significance when data from the first published randomized controlled trial were excluded. Of the eight statistically significant sex-treatment interactions from the overall analyses, only three were discussed by the CDSR reviewers for a potential impact on different clinical management for males compared with females. None of these topics had a sex-treatment interaction that influenced treatment recommendations in recent guidelines. UpToDate, an online physician-authored clinical decision support resource, suggested differential management of men and women for one of these sex-treatment interactions.

CONCLUSION

 Statistically significant sex-treatment interactions are only slightly more frequent than what would be expected by chance and there is little evidence of subsequent corroboration or clinical relevance of sex-treatment interactions.

Vitamin E and the risk of pneumonia: using the I 2 statistic to quantify heterogeneity within a controlled trial

Analyses in nutritional epidemiology usually assume a uniform effect of a nutrient. Previously, four subgroups of the Alpha-Tocopherol, Beta-Carotene Cancer Prevention (ATBC) Study of Finnish male smokers aged 50-69 years were identified in which vitamin E supplementation either significantly increased or decreased the risk of pneumonia. The purpose of this present study was to quantify the level of true heterogeneity in the effect of vitamin E on pneumonia incidence using the I 2 statistic. The I 2 value estimates the percentage of total variation across studies that is explained by true differences in the treatment effect rather than by chance, with a range from 0 to 100 %. The I 2 statistic for the effect of vitamin E supplementation on pneumonia risk for five subgroups of the ATBC population was 89 % (95 % CI 78, 95 %), indicating that essentially all heterogeneity was true variation in vitamin E effect instead of chance variation. The I 2 statistic for heterogeneity in vitamin E effects on pneumonia risk was 92 % (95 % CI 80, 97 %) for three other ATBC subgroups defined by smoking level and leisure-time exercise level. Vitamin E decreased pneumonia risk by 69 % among participants who had the least exposure to smoking and exercised during leisure time (7·6 % of the ATBC participants), and vitamin E increased pneumonia risk by 68 % among those who had the highest exposure to smoking and did not exercise (22 % of the ATBC participants). These findings refute there being a uniform effect of vitamin E supplementation on the risk of pneumonia.

No improvement in the reporting of clinical trial subgroup effects in high-impact general medical journals

BACKGROUND

When subgroup analyses are not correctly analyzed and reported, incorrect conclusions may be drawn, and inappropriate treatments provided. Despite the increased recognition of the importance of subgroup analysis, little information exists regarding the prevalence, appropriateness, and study characteristics that influence subgroup analysis. The objective of this study is to determine (1) if the use of subgroup analyses and multivariable risk indices has increased, (2) whether statistical methodology has improved over time, and (3) which study characteristics predict subgroup analysis.

METHODS

We randomly selected randomized controlled trials (RCTs) from five high-impact general medical journals during three time periods. Data from these articles were abstracted in duplicate using standard forms and a standard protocol. Subgroup analysis was defined as reporting any subgroup effect. Appropriate methods for subgroup analysis included a formal test for heterogeneity or interaction across treatment-by-covariate groups. We used logistic regression to determine the variables significantly associated with any subgroup analysis or, among RCTs reporting subgroup analyses, using appropriate methodology.

RESULTS

The final sample of 416 articles reported 437 RCTs, of which 270 (62 %) reported subgroup analysis. Among these, 185 (69 %) used appropriate methods to conduct such analyses. Subgroup analysis was reported in 62, 55, and 67 % of the articles from 2007, 2010, and 2013, respectively. The percentage using appropriate methods decreased over the three time points from 77 % in 2007 to 63 % in 2013 (p < 0.05). Significant predictors of reporting subgroup analysis included industry funding (OR 1.94 (95 % CI 1.17, 3.21)), sample size (OR 1.98 per quintile (1.64, 2.40), and a significant primary outcome (OR 0.55 (0.33, 0.92)). The use of appropriate methods to conduct subgroup analysis decreased by year (OR 0.88 (0.76, 1.00)) and was less common with industry funding (OR 0.35 (0.18, 0.70)). Only 33 (18 %) of the RCTs examined subgroup effects using a multivariable risk index.

CONCLUSIONS

While we found no significant increase in the reporting of subgroup analysis over time, our results show a significant decrease in the reporting of subgroup analyses using appropriate methods during recent years. Industry-sponsored trials may more commonly report subgroup analyses, but without utilizing appropriate methods. Suboptimal reporting of subgroup effects may impact optimal physician-patient decision-making.

Best practice for analysis of shared clinical trial data

Background

Greater transparency, including sharing of patient-level data for further research, is an increasingly important topic for organisations who sponsor, fund and conduct clinical trials. This is a major paradigm shift with the aim of maximising the value of patient-level data from clinical trials for the benefit of future patients and society. We consider the analysis of shared clinical trial data in three broad categories: (1) reanalysis - further investigation of the efficacy and safety of the randomized intervention, (2) meta-analysis, and (3) supplemental analysis for a research question that is not directly assessing the randomized intervention.

Discussion

In order to support appropriate interpretation and limit the risk of misleading findings, analysis of shared clinical trial data should have a pre-specified analysis plan. However, it is not generally possible to limit bias and control multiplicity to the extent that is possible in the original trial design, conduct and analysis, and this should be acknowledged and taken into account when interpreting results. We highlight a number of areas where specific considerations arise in planning, conducting, interpreting and reporting analyses of shared clinical trial data. A key issue is that that these analyses essentially share many of the limitations of any post hoc analyses beyond the original specified analyses. The use of individual patient data in meta-analysis can provide increased precision and reduce bias. Supplemental analyses are subject to many of the same issues that arise in broader epidemiological analyses. Specific discussion topics are addressed within each of these areas.

Summary

Increased provision of patient-level data from industry and academic-led clinical trials for secondary research can benefit future patients and society. Responsible data sharing, including transparency of the research objectives, analysis plans and of the results will support appropriate interpretation and help to address the risk of misleading results and avoid unfounded health scares.

Subgroup analyses of randomized clinical trials in heart failure: facts and numbers

Subgroup analyses of major randomized clinical trials in heart failure are published frequently, but their impact on medical knowledge and practice guidelines has not been previously reported. In a novel analysis, we determined number of citations, impact factors, number of authors, and citations in guidelines of both parent trials and sub‐studies; we also qualitatively assessed whether the analyses were described as post‐hoc and non‐pre‐specified. A total of 229 sub‐studies evaluating outcomes in patient subgroups were published (median 6, range 0–36 per trial). The number of subjects in the parent trials positively correlated with number of sub‐studies (rho = 0.51, P = 0.009). The subgroups are frequently not pre‐specified. The impact factors of sub‐studies were lower in comparison to the parent trials as were the number of citations two years after the publication date; in addition, parent trials were cited more frequently in European and American professional guidelines compared with the sub‐studies. We maintain that the sub‐studies derived from major heart failure trials are frequently published, but their contribution to clinical guidelines and medical knowledge are highly debatable.

Subgroup analyses in confirmatory clinical trials: time to be specific about their purposes

Background

It is well recognized that treatment effects may not be homogeneous across the study population. Subgroup analyses constitute a fundamental step in the assessment of evidence from confirmatory (Phase III) clinical trials, where conclusions for the overall study population might not hold. Subgroup analyses can have different and distinct purposes, requiring specific design and analysis solutions. It is relevant to evaluate methodological developments in subgroup analyses against these purposes to guide health care professionals and regulators as well as to identify gaps in current methodology.

Methods

We defined four purposes for subgroup analyses: (1) Investigate the consistency of treatment effects across subgroups of clinical importance, (2) Explore the treatment effect across different subgroups within an overall non-significant trial, (3) Evaluate safety profiles limited to one or a few subgroup(s), (4) Establish efficacy in the targeted subgroup when included in a confirmatory testing strategy of a single trial. We reviewed the methodology in line with this “purpose-based” framework. The review covered papers published between January 2005 and April 2015 and aimed to classify them in none, one or more of the aforementioned purposes.

Results

In total 1857 potentially eligible papers were identified. Forty-eight papers were selected and 20 additional relevant papers were identified from their references, leading to 68 papers in total. Nineteen were dedicated to purpose 1, 16 to purpose 4, one to purpose 2 and none to purpose 3. Seven papers were dedicated to more than one purpose, the 25 remaining could not be classified unambiguously. Purposes of the methods were often not specifically indicated, methods for subgroup analysis for safety purposes were almost absent and a multitude of diverse methods were developed for purpose (1).

Conclusions

It is important that researchers developing methodology for subgroup analysis explicitly clarify the objectives of their methods in terms that can be understood from a patient’s, health care provider’s and/or regulator’s perspective. A clear operational definition for consistency of treatment effects across subgroups is lacking, but is needed to improve the usability of subgroup analyses in this setting. Finally, methods to particularly explore benefit-risk systematically across subgroups need more research.

Electronic supplementary material

The online version of this article (doi:10.1186/s12874-016-0122-6) contains supplementary material, which is available to authorized users.

Assessing Heterogeneity of Treatment Effects: Are Authors Misinterpreting Their Results?

OBJECTIVE

To determine whether investigations of heterogeneity of treatment effects (HTE) in randomized-controlled trials (RCTs) are prespecified and whether authors' interpretations of their analyses are consistent with the objective evidence.

DATA SOURCES/STUDY SETTING

Trials published in Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, Lancet, and New England Journal of Medicine in 1994, 1999, and 2004.

STUDY DESIGN

We reviewed 87 RCTs that reported formal tests for statistical interaction or heterogeneity (HTE analyses), derived from a probability sample of 541 articles.

DATA COLLECTION/EXTRACTION

We recorded reasons for performing HTE analysis; an objective classification of evidence for HTE (termed "clinicostatistical divergence" [CSD]); and authors' interpretations of findings. Authors' interpretations, compared with CSD, were coded as understated, overstated, or adequately stated.

PRINCIPLE FINDINGS

Fifty-three RCTs (61 percent) claimed prespecified covariates for HTE analyses. Trials showed strong (6), moderate (11), weak (25), or negligible (16) evidence for CSD (29 could not be classified due to inadequate information). Authors stated that evidence for HTE was sufficient to support differential treatment in subgroups (10); warranted more research (31); was absent (21); or provided no interpretation (25). HTE was overstated in 22 trials, adequately stated in 57 trials, and understated in 8 trials.

CONCLUSIONS

Inconsistencies in performance and reporting may limit the potential of HTE analysis as a tool for identifying HTE and individualizing care in diverse populations. Recommendations for future studies on the reporting and interpretation of HTE analyses are provided.

A systematic approach to subgroup analyses in a smoking cessation trial

BACKGROUND

Traditional approaches to subgroup analyses that test each moderating factor as a separate hypothesis can lead to erroneous conclusions due to the problems of multiple comparisons, model misspecification, and multicollinearity.

OBJECTIVE

To demonstrate a novel, systematic approach to subgroup analyses that avoids these pitfalls.

METHODS

A Best Approximating Model (BAM) approach that identifies multiple moderators and estimates their simultaneous impact on treatment effect sizes was applied to a randomized, controlled, 11-week, double-blind efficacy trial on smoking cessation of adult smokers with attention-deficit/hyperactivity disorder (ADHD), randomized to either OROS-methylphenidate (n = 127) or placebo (n = 128), and treated with nicotine patch. Binary outcomes measures were prolonged smoking abstinence and point prevalence smoking abstinence.

RESULTS

Although the original clinical trial data analysis showed no treatment effect on smoking cessation, the BAM analysis showed significant subgroup effects for the primary outcome of prolonged smoking abstinence: (1) lifetime history of substance use disorders (adjusted odds ratio [AOR] 0.27; 95% confidence interval [CI] 0.10-0.74), and (2) more severe ADHD symptoms (baseline score >36; AOR 2.64; 95% CI 1.17-5.96). A significant subgroup effect was also shown for the secondary outcome of point prevalence smoking abstinence--age 18 to 29 years (AOR 0.23; 95% CI 0.07-0.76).

CONCLUSIONS

The BAM analysis resulted in different conclusions about subgroup effects compared to a hypothesis-driven approach. By examining moderator independence and avoiding multiple testing, BAMs have the potential to better identify and explain how treatment effects vary across subgroups in heterogeneous patient populations, thus providing better guidance to more effectively match individual patients with specific treatments.

Subgroup Analyses in Reporting of Phase III Clinical Trials in Solid Tumors

Purpose

Treatment decisions in clinical oncology are guided by results from phase III randomized clinical trials (RCTs). The results of subgroup analyses may be potentially important in individualizing patient care. We investigated the appropriateness of the use and interpretation of subgroup analyses in oncology RCTs on the basis of the CONSORT statement requirements.

Methods

Phase III RCTs published between January 1, 2011, and December 31, 2013, were reviewed to identify eligible studies of solid tumor treatments. Information related to the subgroup analyses included prespecification, number, subgroup factors, interaction test use, and claim of subgroup difference.

Results

A total of 221 publications reporting data on 184,500 patients were analyzed. One hundred eighty-eight (85%) RCTs were reported with subgroup analyses. Of those, 146 (78%) trials were reported with at least six subgroups. For the majority of trials with subgroup analyses (173; 92%), the actual number of subgroup analyses conducted cannot be determined. Only 59 (31%) RCTs were reported with fully prespecified subgroups and only 64 (34%) trials were reported with interaction tests. In addition, 102 (54%) RCTs were reported with claims of subgroup differences. Of those, only 18 claims of RCTs (18%) were based on significant interaction test results.

Conclusion

The reporting of subgroup analyses in contemporary oncology RCTs is neither uniform nor complete; it requires improvement to ensure consistency and to provide critical information for guiding patient care. Major problems include testing of a large number of subgroups, subgroups without prespecifications, and inadequate use of interaction tests.

Estimates of absolute treatment benefit for individual patients required careful modeling of statistical interactions

OBJECTIVES

We aimed to compare modeling approaches to estimate the individual survival benefit of treatment with either coronary artery bypass graft surgery (CABG) or percutaneous coronary intervention (PCI) for patients with complex coronary artery disease.

STUDY DESIGN AND SETTING

We estimated survival with Cox regression models that included the treatment variable (CABG/PCI) interacting with either an internally developed overall prognostic index (PI) or with individual prognostic factors. We analyzed data of patients who were randomized in the Synergy between Percutaneous Coronary Intervention with Taxus and Cardiac Surgery trial (1,800 patients, 178 deaths).

RESULTS

A negligible interaction with the PI (P = 0.51) led to 4-year survival estimates in favor of CABG for all patients. In contrast, individual interactions indicated substantial relative treatment effect heterogeneity (overall interaction P = 0.004), and estimates of 4-year survival were numerically in favor of CABG for 1,275 of 1,800 patients (71%; 519 with 95% confidence). To test the more complex model with individual interactions, we first used penalized regression, resulting in smaller but largely consistent individual estimates of the survival difference between CABG and PCI. Second, strong treatment interactions were confirmed at external validation in 2,891 patients from a multinational registry.

CONCLUSION

Modeling strategies that omit interactions may result in misleading estimates of absolute treatment benefit for individual patients with the potential hazard of suboptimal decision making.

Regional variation in patients and outcomes in the Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist (TOPCAT) trial

BACKGROUND

Treatment of Preserved Cardiac Function Heart Failure With an Aldosterone Antagonist (TOPCAT) patients with heart failure and preserved left ventricular ejection fraction assigned to spironolactone did not achieve a significant reduction in the primary composite outcome (time to cardiovascular death, aborted cardiac arrest, or hospitalization for management of heart failure) compared with patients receiving placebo. In a post hoc analysis, an ≈4-fold difference was identified in this composite event rate between the 1678 patients randomized from Russia and Georgia compared with the 1767 enrolled from the United States, Canada, Brazil, and Argentina (the Americas).

METHODS AND RESULTS

To better understand this regional difference in clinical outcomes, demographic characteristics of these populations and their responses to spironolactone were explored. Patients from Russia/Georgia were younger, had less atrial fibrillation and diabetes mellitus, but were more likely to have had prior myocardial infarction or a hospitalization for heart failure. Russia/Georgia patients also had lower left ventricular ejection fraction and creatinine but higher diastolic blood pressure (all P<0.001). Hyperkalemia and doubling of creatinine were more likely and hypokalemia was less likely in patients receiving spironolactone in the Americas with no significant treatment effects in Russia/Georgia. All clinical event rates were markedly lower in Russia/Georgia, and there was no detectable impact of spironolactone on any outcomes. In contrast, in the Americas, the rates of the primary outcome, cardiovascular death, and hospitalization for heart failure were significantly reduced by spironolactone.

CONCLUSIONS

This post hoc analysis demonstrated greater potassium and creatinine changes and possible clinical benefits with spironolactone in patients with heart failure and preserved ejection fraction from the Americas.

CLINICAL TRIAL REGISTRATION URL

http://www.clinicaltrials.gov. Unique identifier: NCT00094302.

Randomized controlled trials versus rough set analysis: two competing approaches for evaluating clinical data

The present paper deals with the problem of evaluating empirical evidence for therapeutic decisions in medicine. The article discusses the views of Nancy Cartwright and John Worrall on the function that randomization plays in ascertaining causal relations with reference to the therapies applied. The main purpose of the paper is to present a general idea of alternative method of evaluating empirical evidence. The method builds on data analysis that makes use of rough set theory. The first attempts to apply the method show that it is an interesting alternative to randomized controlled trials.

Subgroup analyses in randomised controlled trials: cohort study on trial protocols and journal publications

OBJECTIVE

To investigate the planning of subgroup analyses in protocols of randomised controlled trials and the agreement with corresponding full journal publications.

DESIGN

Cohort of protocols of randomised controlled trial and subsequent full journal publications.

SETTING

Six research ethics committees in Switzerland, Germany, and Canada.

DATA SOURCES

894 protocols of randomised controlled trial involving patients approved by participating research ethics committees between 2000 and 2003 and 515 subsequent full journal publications.

RESULTS

Of 894 protocols of randomised controlled trials, 252 (28.2%) included one or more planned subgroup analyses. Of those, 17 (6.7%) provided a clear hypothesis for at least one subgroup analysis, 10 (4.0%) anticipated the direction of a subgroup effect, and 87 (34.5%) planned a statistical test for interaction. Industry sponsored trials more often planned subgroup analyses compared with investigator sponsored trials (195/551 (35.4%) v 57/343 (16.6%), P<0.001). Of 515 identified journal publications, 246 (47.8%) reported at least one subgroup analysis. In 81 (32.9%) of the 246 publications reporting subgroup analyses, authors stated that subgroup analyses were prespecified, but this was not supported by 28 (34.6%) corresponding protocols. In 86 publications, authors claimed a subgroup effect, but only 36 (41.9%) corresponding protocols reported a planned subgroup analysis.

CONCLUSIONS

Subgroup analyses are insufficiently described in the protocols of randomised controlled trials submitted to research ethics committees, and investigators rarely specify the anticipated direction of subgroup effects. More than one third of statements in publications of randomised controlled trials about subgroup prespecification had no documentation in the corresponding protocols. Definitive judgments regarding credibility of claimed subgroup effects are not possible without access to protocols and analysis plans of randomised controlled trials.

Evidence for the selective reporting of analyses and discrepancies in clinical trials: a systematic review of cohort studies of clinical trials

BACKGROUND

Most publications about selective reporting in clinical trials have focussed on outcomes. However, selective reporting of analyses for a given outcome may also affect the validity of findings. If analyses are selected on the basis of the results, reporting bias may occur. The aims of this study were to review and summarise the evidence from empirical cohort studies that assessed discrepant or selective reporting of analyses in randomised controlled trials (RCTs).

METHODS AND FINDINGS

A systematic review was conducted and included cohort studies that assessed any aspect of the reporting of analyses of RCTs by comparing different trial documents, e.g., protocol compared to trial report, or different sections within a trial publication. The Cochrane Methodology Register, Medline (Ovid), PsycInfo (Ovid), and PubMed were searched on 5 February 2014. Two authors independently selected studies, performed data extraction, and assessed the methodological quality of the eligible studies. Twenty-two studies (containing 3,140 RCTs) published between 2000 and 2013 were included. Twenty-two studies reported on discrepancies between information given in different sources. Discrepancies were found in statistical analyses (eight studies), composite outcomes (one study), the handling of missing data (three studies), unadjusted versus adjusted analyses (three studies), handling of continuous data (three studies), and subgroup analyses (12 studies). Discrepancy rates varied, ranging from 7% (3/42) to 88% (7/8) in statistical analyses, 46% (36/79) to 82% (23/28) in adjusted versus unadjusted analyses, and 61% (11/18) to 100% (25/25) in subgroup analyses. This review is limited in that none of the included studies investigated the evidence for bias resulting from selective reporting of analyses. It was not possible to combine studies to provide overall summary estimates, and so the results of studies are discussed narratively.

CONCLUSIONS

Discrepancies in analyses between publications and other study documentation were common, but reasons for these discrepancies were not discussed in the trial reports. To ensure transparency, protocols and statistical analysis plans need to be published, and investigators should adhere to these or explain discrepancies.

Is regadenoson an appropriate stressor for MPI in patients with left bundle branch block or pacemakers?

BACKGROUND

Patients with LBBB or ventricular pacemaker undergoing MPI are at risk for false positive MPI results in the setting of an elevated heart rate (HR) with exercise or dobutamine stress. The areas of increased apparent ischemia are typically the LAD and septal territories.

METHODS

In a subanalysis of the ADVANCE MPI 1 and 2 studies, perfusion on an initial adenosine and a second MPI study with regadenoson or adenosine was compared by visual and quantitative analysis. Among 2,015 patients, 64 had LBBB and 93 had pacemakers. The hemodynamic response during the second scan was compared in those with and without LBBB and PM.

RESULTS

Following regadenoson, peak HR in the LBBB group increased by a mean of 25.4 compared to 15.3 bpm following adenosine (P = .0083). In the pacemaker group HR was blunted, 11.8 and 8.1 following regadenoson and adenosine, respectively (P = .1262). However, the visually assessed summed difference score and the quantitatively assessed extent of ischemia for the LAD and septal territories and the entire LV did not differ between the initial adenosine and subsequent regadenoson scans.

CONCLUSIONS

The significant increase in HR observed with regadenoson compared to adenosine did not translate into greater perfusion defects in the LAD or septal territories in patients undergoing regadenoson stress.

Detecting moderator effects using subgroup analyses

In the analysis of prevention and intervention studies, it is often important to investigate whether treatment effects vary among subgroups of patients defined by individual characteristics. These "subgroup analyses" can provide information about how best to use a new prevention or intervention program. However, subgroup analyses can be misleading if they test data-driven hypotheses, employ inappropriate statistical methods, or fail to account for multiple testing. These problems have led to a general suspicion of findings from subgroup analyses. This article discusses sound methods for conducting subgroup analyses to detect moderators. Multiple authors have argued that, to assess whether a treatment effect varies across subgroups defined by patient characteristics, analyses should be based on tests for interaction rather than treatment comparisons within the subgroups. We discuss the concept of heterogeneity and its dependence on the metric used to describe treatment effects. We discuss issues of multiple comparisons related to subgroup analyses and the importance of considering multiplicity in the interpretation of results. We also discuss the types of questions that would lead to subgroup analyses and how different scientific goals may affect the study at the design stage. Finally, we discuss subgroup analyses based on post-baseline factors and the complexity associated with this type of subgroup analysis.

Reporting characteristics of non-primary publications of results of randomized trials: a cross-sectional review

Background

For a randomized trial, the primary publication is usually the one which reports the results of the primary outcome and provides consolidated data from all study centers. Other aspects of a randomized trial’s findings (that is, non-primary results) are often reported in subsequent publications.

Methods

We carried out a cross-sectional review of the characteristics and type of information reported in non-primary reports (n = 69) of randomized trials (indexed in PubMed core clinical journals in 2009) and whether they report pre-specified or exploratory analyses. We also compared consistency of information in non-primary publications with that reported in the primary publication.

Results

The majority (n = 56; 81%) of non-primary publications were large, multicenter trials, published in specialty journals. Most reported subgroup analyses (n = 27; 39%), analyzing a specific subgroup of patients from the randomized trial, or reported on secondary outcomes (n = 29; 42%); 19% (n = 13) reported extended follow-up. Less than half reported details of trial registration (n = 30; 43%) or the trial protocol (n = 27; 39%) and in 41% (n = 28) it was unclear from reading the abstract that the report was not the primary publication for the trial. Non-primary publications often analyzed and reported multiple different outcomes (16% reported >20 outcomes) and in 10% (n = 7) it was unclear how many outcomes had actually been assessed; in 42% (n = 29) it was unclear whether the analyses reported were pre-specified or exploratory. Only 39% (n = 27) of non-primary publications described the primary outcome of the randomized trial, 6% (n = 4) reported its numerical results and 9% (n = 6) details of how participants were randomized.

Conclusion

Non-primary publications often lack important information about the randomized trial and the type of analyses conducted and whether these were pre-specified or exploratory to enable readers to accurately identify and assess the validity and reliably of the study findings. We provide recommendations for what information authors should include in non-primary reports of randomized trials.