Clinical trials: discerning hype from substance

Cross-validated risk scores adaptive enrichment (CADEN) design

We propose a Cross-validated ADaptive ENrichment design (CADEN) in which a trial population is enriched with a subpopulation of patients who are predicted to benefit from the treatment more than an average patient (the sensitive group). This subpopulation is found using a risk score constructed from the baseline (potentially high-dimensional) information about patients. The design incorporates an early stopping rule for futility. Simulation studies are used to assess the properties of CADEN against the original (non-enrichment) cross-validated risk scores (CVRS) design which constructs a risk score at the end of the trial. We show that when there exists a sensitive group of patients, CADEN achieves a higher power and a reduction in the expected sample size compared to the CVRS design. We illustrate the application of the design in two real clinical trials. We conclude that the new design offers improved statistical efficiency over the existing non-enrichment method, as well as increased benefit to patients. The method has been implemented in an R package caden.

Finding the (biomarker-defined) subgroup of patients who benefit from a novel therapy: No time for a game of hide and seek

An important element of precision medicine is the ability to identify, for a specific therapy, those patients for whom benefits of that therapy meaningfully exceed the risks. To achieve this goal, treatment effect usually is examined across subgroups defined by a variety of factors, including demographic, clinical, or pathologic characteristics or by molecular attributes of patients or their disease. Frequently such subgroups are defined by the measurement of biomarkers. Even though such examination is necessary when pursuing this goal, the evaluation of treatment effect across a variety of subgroups is statistically fraught due to both the danger of inflated false-positive error rate from multiple testing and the inherent insensitivity to how treatment effects differ across subgroups.Pre-specification of subgroup analyses with appropriate control of false-positive (i.e. type I) error is recommended when possible. However, when subgroups are specified by biomarkers, which could be measured by different assays and might lack established interpretation criteria, such as cut-offs, it might not be possible to fully specify those subgroups at the time a new therapy is ready for definitive evaluation in a Phase 3 trial. In these situations, further refinement and evaluation of treatment effect in biomarker-defined subgroups might have to take place within the trial. A common scenario is that evidence suggests that treatment effect is a monotone function of a biomarker value, but optimal cut-offs for therapy decisions are not known. In this setting, hierarchical testing strategies are widely used, where testing is first conducted in a particular biomarker-positive subgroup and then is conducted in the expanded pool of biomarker-positive and biomarker-negative patients, with control for multiple testing. A serious limitation of this approach is the logical inconsistency of excluding the biomarker-negatives when evaluating effects in the biomarker-positives, yet allowing the biomarker-positives to drive the assessment of whether a conclusion of benefit could be extrapolated to the biomarker-negative subgroup.Examples from oncology and cardiology are described to illustrate the challenges and pitfalls. Recommendations are provided for statistically valid and logically consistent subgroup testing in these scenarios as alternatives to reliance on hierarchical testing alone, and approaches for exploratory assessment of continuous biomarkers as treatment effect modifiers are discussed.

Pro-Con Debate: Interdisciplinary Perspectives on Industry-Sponsored Research

Pro: Nearly all new devices and drugs come from industry that provides two-thirds of the funding for medical research, and a much higher fraction of clinical research. Realistically, without corporate-funded studies, perioperative research would stagnate with little innovation and few new products. Opinions are ubiquitous and normal, but do not constitute epidemiologic bias. Competent clinical research includes many protections against selection and measurement bias, and the publication process provides at least moderate protection against misinterpretation of results. Trial registries largely prevent selective data presentation. Sponsored trials are particularly protected against inappropriate corporate influence because they are usually codesigned with the US Food and Drug Administration, and analyses are based on formal predefined statistical plans, as well as being conducted with rigorous external monitoring. Novel products, which are essential for advances in clinical care, largely come from industry, and industry appropriately funds much of the required research. We should celebrate industry's contribution to improvements in clinical care. Con: While industry funding contributes to research and discovery, examples of industry-funded research demonstrate bias. In the setting of financial pressures and potential conflict of interest, bias can influence the type of study design, hypotheses being tested, rigor and transparency in data analysis, interpretation, as well as reporting of the results. Unlike public granting agencies, industry does not necessarily provide funding based on unbiased peer review following an open call for proposals. The focus on success can influence the choice of a comparator, which might not be ideal among the possible alternatives, the language used in the publication, and even the ability to publish. Unpublished negative trials can result in selected information being withheld from the scientific community and the public. Appropriate safeguards are needed to ensure that research addresses the most important and relevant questions, that results are available even when they do not support the use of a product produced by the funding company, that populations studied reflect the relevant patients, that the most rigorous approaches are applied, that studies have the appropriate power to address the question posed, and that conclusions are presented in an unbiased manner.

Pro-Con Debate: Interdisciplinary Perspectives on Industry-Sponsored Research

PRO: Nearly all new devices and drugs come from industry that provides two-thirds of the funding for medical research, and a much higher fraction of clinical research. Realistically, without corporate-funded studies, perioperative research would stagnate with little innovation and few new products. Opinions are ubiquitous and normal but do not constitute epidemiologic bias. Competent clinical research includes many protections against selection and measurement bias, and the publication process provides at least moderate protection against misinterpretation of results. Trial registries largely prevent selective data presentation. Sponsored trials are particularly protected against inappropriate corporate influence because they are usually codesigned with the US Food and Drug Administration, and analyses are based on formal predefined statistical plans, as well as being conducted with rigorous external monitoring. Novel products, which are essential for advances in clinical care, largely come from industry, and industry appropriately funds much of the required research. We should celebrate industry's contribution to improvements in clinical care. CON: While industry funding contributes to research and discovery, examples of industry-funded research demonstrate bias. In the setting of financial pressures and potential conflict of interest, bias can influence the type of study design, hypotheses being tested, rigor and transparency in data analysis, interpretation, as well as reporting of the results. Unlike public granting agencies, industry does not necessarily provide funding based on unbiased peer review following an open call for proposals. The focus on success can influence the choice of a comparator, which might not be ideal among the possible alternatives, the language used in the publication, and even the ability to publish. Unpublished negative trials can result in selected information being withheld from the scientific community and the public. Appropriate safeguards are needed to ensure that research addresses the most important and relevant questions, that results are available even when they do not support the use of a product produced by the funding company, that populations studied reflect the relevant patients, that the most rigorous approaches are applied, that studies have the appropriate power to address the question posed, and that conclusions are presented in an unbiased manner.

Application of Bayesian approaches in drug development: starting a virtuous cycle

The pharmaceutical industry and its global regulators have routinely used frequentist statistical methods, such as null hypothesis significance testing and p values, for evaluation and approval of new treatments. The clinical drug development process, however, with its accumulation of data over time, can be well suited for the use of Bayesian statistical approaches that explicitly incorporate existing data into clinical trial design, analysis and decision-making. Such approaches, if used appropriately, have the potential to substantially reduce the time and cost of bringing innovative medicines to patients, as well as to reduce the exposure of patients in clinical trials to ineffective or unsafe treatment regimens. Nevertheless, despite advances in Bayesian methodology, the availability of the necessary computational power and growing amounts of relevant existing data that could be used, Bayesian methods remain underused in the clinical development and regulatory review of new therapies. Here, we highlight the value of Bayesian methods in drug development, discuss barriers to their application and recommend approaches to address them. Our aim is to engage stakeholders in the process of considering when the use of existing data is appropriate and how Bayesian methods can be implemented more routinely as an effective tool for doing so.

Exploratory analyses of clinical trial data used for health technology assessments: a retrospective evaluation

OBJECTIVES

To examine the validity and statistical limitations of exploratory analyses of clinical trial data commonly requested by agencies responsible for determining which medical products may be financed or reimbursed by a healthcare system.

DESIGN

This was a retrospective review of efficacy and safety analyses conducted for German Health Technology Assessment (HTA) evaluations with a decision date between 2015 and 2020, and an illustrative safety-related exploratory analysis of data from two phase III clinical trials of verubecestat (an anti-amyloid drug whose development was stopped for lack of efficacy) as would be mandated by the German HTA agency.

RESULTS

We identified 422 HTA evaluations of 404 randomised controlled clinical trials. For 140 trials (34.7%), the evaluation was based on subpopulations of participants in the originating confirmatory trial (175 subpopulations were assessed). In 57% (100 of 175), the subpopulation sample size was 50% or less of the original study population. Detailed analysis of five evaluations based on subpopulations of the original trial is presented. The safety-related exploratory analysis of verubecestat led to 206 statistical analyses for treatments and 812 treatment-by-subgroup interaction tests. Of 31 safety endpoints with an elevated HR (suggesting association with drug treatment), the HR for 81% of these (25 of 31) was not elevated in both trials. Of the 812 treatment-by-subgroup interactions evaluated, 26 had an elevated HR for a subgroup in one trial, but only 1 was elevated in both trials.

CONCLUSIONS

Many HTA evaluations rely on subpopulation analyses and numerous post hoc statistical hypothesis tests. Subpopulation analysis may lead to loss of statistical power and uncontrolled influences of random imbalances. Multiple testing may introduce spurious findings. Decisions about benefits of medical products should therefore not rely on exploratory analyses of clinical trial data but rather on prospective clinical studies and careful synthesis of all available evidence based on prespecified criteria.

A randomized clinical trial of the efficacy and safety of sitagliptin as initial oral therapy in youth with type 2 diabetes

OBJECTIVE

To assess the efficacy and safety of DPP-4 inhibition with sitagliptin in youth with type 2 diabetes (T2D).

STUDY DESIGN

This was a 54-week, double-blind, randomized, controlled clinical trial evaluating the safety and efficacy of DPP-4 inhibition with sitagliptin 100 mg once daily as initial oral therapy in youth with T2D. The 190 participants, aged 10-17 years, had HbA1c 6.5%-10% (7.0%-10% if on insulin). All were negative for pancreatic autoantibodies and overweight/obese at screening or diagnosis. The trial was placebo controlled for the first 20 weeks, after which metformin replaced placebo. The primary efficacy endpoint was change from baseline in HbA1c at Week 20.

RESULTS

Treatment groups were well balanced at baseline (mean ± SD HbA1c = 7.5% ± 1.0, BMI percentile = 97.1% ± 6.8, age = 14.0 years ± 2.0 [57.4% <15], 60.5% female). At Week 20, least squares mean changes from baseline in HbA1c were -0.01% (sitagliptin) and 0.18% (placebo); between-group difference (95% CI) = -0.19% (-0.68, 0.30), p = 0.448. At Week 54, the changes in HbA1c were 0.45% (sitagliptin) and -0.11 (placebo/metformin). There were no notable between-group differences in the adverse event profiles through Week 54.

CONCLUSIONS

DPP-4 inhibition with sitagliptin did not provide significant improvement in glycemic control. In this study, sitagliptin was generally well tolerated with a safety profile similar to that reported in adults. (ClinicalTrials.gov: NCT01485614; EudraCT: 2011-002528-42).

The frequent insignificance of a "significant" p-value

Null hypothesis significance testing (NHST) and p-values are widespread in the cardiac surgical literature but are frequently misunderstood and misused. The purpose of the review is to discuss major disadvantages of p-values and suggest alternatives. We describe diagnostic tests, the prosecutor's fallacy in the courtroom, and NHST, which involve inter-related conditional probabilities, to help clarify the meaning of p-values, and discuss the enormous sampling variability, or unreliability, of p-values. Finally, we use a cardiac surgical database and simulations to explore further issues involving p-values. In clinical studies, p-values provide a poor summary of the observed treatment effect, whereas the three-number summary provided by effect estimates and confidence intervals is more informative and minimizes over-interpretation of a "significant" result. p-values are an unreliable measure of the strength of evidence; if used at all they give only, at best, a very rough guide to decision making. Researchers should adopt Open Science practices to improve the trustworthiness of research and, where possible, use estimation (three-number summaries) or other better techniques.

SPIRIT-PRO Extension explanation and elaboration: guidelines for inclusion of patient-reported outcomes in protocols of clinical trials

Patient-reported outcomes (PROs) are used in clinical trials to provide valuable evidence on the impact of disease and treatment on patients' symptoms, function and quality of life. High-quality PRO data from trials can inform shared decision-making, regulatory and economic analyses and health policy. Recent evidence suggests the PRO content of past trial protocols was often incomplete or unclear, leading to research waste. To address this issue, international, consensus-based, PRO-specific guidelines were developed: the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT)-PRO Extension. The SPIRIT-PRO Extension is a 16-item checklist which aims to improve the content and quality of aspects of clinical trial protocols relating to PRO data collection to minimise research waste, and ultimately better inform patient-centred care. This SPIRIT-PRO explanation and elaboration (E&E) paper provides information to promote understanding and facilitate uptake of the recommended checklist items, including a comprehensive protocol template. For each SPIRIT-PRO item, we provide a detailed description, one or more examples from existing trial protocols and supporting empirical evidence of the item's importance. We recommend this paper and protocol template be used alongside the SPIRIT 2013 and SPIRIT-PRO Extension paper to optimise the transparent development and review of trial protocols with PROs.

Shifting the focus away from binary thinking of statistical significance and towards education for key stakeholders: revisiting the debate on whether it's time to de-emphasize or get rid of statistical significance

There has been a long-standing controversy among scientists regarding the appropriate use of P-values and statistical significance in clinical research. This debate has resurfaced through recent calls to modify the threshold of P-value required to declare significance, or to retire statistical significance entirely. In this article, we revisit the issue by discussing: i) the connection between statistical thinking and evidence-based practice; ii) some history of statistical significance and P-values; iii) some practical challenges with statistical significance or P-value thresholds in clinical research; iv) the on-going debate on what to do with statistical significance; v) suggestions to shift the focus away from binary thinking of statistical significance and towards education for key stakeholders on research essentials including statistical thinking, critical thinking, good reporting, basic clinical research concepts and methods, and more. We then conclude with remarks and illustrations of the potential deleterious public health consequences of poor methods including selective choice of analysis approach and misguided reliance on binary use of P-values to report and interpret scientific findings.

Design, Analysis, and Pitfalls of Clinical Trials Using Ex Situ Liver Machine Perfusion: The International Liver Transplantation Society Consensus Guidelines

BACKGROUND

Recent trials in liver machine perfusion (MP) have revealed unique challenges beyond those seen in most clinical studies. Correct trial design and interpretation of data are essential to avoid drawing conclusions that may compromise patient safety and increase costs.

METHODS

The International Liver Transplantation Society, through the Special Interest Group "DCD, Preservation and Machine Perfusion," established a working group to write consensus statements and guidelines on how future clinical trials in liver perfusion should be designed, with particular focus on relevant clinical endpoints and how different techniques of liver perfusion should be compared. Protocols, abstracts, and full published papers of clinical trials using liver MP were reviewed. The use of a simplified Grading of Recommendations Assessment, Development, and Evaluation working group (GRADE) system was attempted to assess the level of evidence. The working group presented its conclusions at the International Liver Transplantation Society consensus conference "DCD, Liver Preservation, and Machine Perfusion" held in Venice, Italy, on January 31, 2020.

RESULTS

Twelve recommendations were proposed with the main conclusions that clinical trials investigating the effect of MP in liver transplantation should (1) make the protocol publicly available before the start of the trial, (2) be adequately powered, and (3) carefully consider timing of randomization in function of the primary outcome.

CONCLUSIONS

There are issues with using accepted primary outcomes of liver transplantation trials in the context of MP trials, and no ideal endpoint could be defined by the working group. The setup of an international registry was considered vital by the working group.

The ACTTION Guide to Clinical Trials of Pain Treatments, part II: mitigating bias, maximizing value

Summaries of the articles included in part II of the ACTTION Guide to Clinical Trials of Pain Treatments are followed by brief overviews of methodologic considerations involving precision pain medicine, pragmatic clinical trials, real world evidence, and patient engagement in clinical trials.

Improving the Standards of Reporting of Clinical Trial Data

Although checklists and guidelines for reporting and interpretation of clinical trial results are of immense value there is still room for a biased presentation in journal publications. Two important sources of bias that remain are as follows: (1) The absence of a principle guiding the display of point estimates in abstracts. For example, bias arises, even for a primary endpoint, when the reported point estimate is preferentially selected and does not correspond to the prespecified method of analysis. The benefit of treatment on an endpoint is often communicated through point estimates, and as abstracts contain the main takeaways, establishing ground rules for what to include and what not to include is crucial. (2) A commingling in the body of the publication of results from α-controlled endpoints, non-α-controlled endpoints, and post hoc analyses. The total number of non-α-controlled and post hoc analyses are unknown. Blending a favored selection of these with α-controlled results provides opportunities to overstate or understate findings as desired. Publicly available results provide the grist for the changes proposed to improve reporting standards. Additional changes are recommended as well, including a threshold of significance more stringent than 0.05 for non-α-controlled analyses. For safety, the proposal is to display the data via the mean cumulative function graph for prespecified adverse events of interest. The bottom line is that more objective reporting can be achieved if journals establish standards for reporting of point estimates in abstracts and require a hierarchical display of results in the main body.

Ethical considerations in the design, execution, and analysis of clinical trials of chronic pain treatments

Introduction:

In the field of pain research, clinical trials may randomize over 500 subjects and include more than 150 sites spanning over a dozen countries.

Methods:

This review examines the ethical considerations affecting clinical trial design, execution, and analysis of trials for chronic pain. The Belmont Report has been the touchstone for human studies protection efforts since 1979. Commissioned by the U.S. government in response to ethical failures in medical research, such as the Tuskegee Syphilis Study, the report emphasizes 3 basic principles: respect for persons, beneficence, and justice. Trial design and sample size have important ethical implications.

Conclusions:

Measures to enhance trial transparency and combat publication and many other types of bias should be implemented.

Reporting of Design Features and Analysis Details in Randomized Clinical Trials of Procedural Treatments for Cancer Pain: An ACTTION Systematic Review

BACKGROUND AND OBJECTIVES

The objective of this study was to assess the reporting of randomized clinical trials investigating procedural treatments (eg, nerve blocks, targeted drug delivery) for cancer pain, with a focus on aspects that are particularly challenging in these trials.

METHODS

This article presents results from a systematic review of reporting of randomized clinical trials of procedural interventions for cancer pain. Articles were identified by searching PubMed from 1966 to June 2014. Data related to quality of reporting are presented for early (1985-2004) and late periods (2005-2014).

RESULTS

A total of 35 published trials were included. Approximately two-thirds of the articles clearly indicated the level of blinding. Only 26% reported a primary outcome measure. Less than half explicitly reported the number of patients who completed the trial, and only 1 reported a method that was used to accommodate missing data. Almost one-third of articles included a responder analysis, all of which specified the definition of a responder.

CONCLUSIONS

The goal of highlighting these deficiencies in reporting is to promote transparent reporting of details affecting the completion and interpretation of procedural cancer pain trials so that their quality can be more easily evaluated.

Trials in adult critical care that show increased mortality of the new intervention: Inevitable or preventable mishaps?

Several promising therapies assessed in the adult critically ill in large, multicenter randomized controlled trials (RCTs) were associated with significantly increased mortality in the intervention arms. Our hypothesis was that there would be wide ranges in sponsorship (industry or not), type(s) of intervention(s), use of DSMBs, presence of interim analyses and early stopping rules, absolute risk increase (ARI), and whether or not adequate prior proof-of-principle Phase II studies were done of RCTs that found increased mortality rates of the intervention compared to control groups. We reviewed RCTs that showed a statistically significant increased mortality rate in the intervention compared to control group(s). We recorded source of sponsorship, sample sizes, types of interventions, mortality rates, ARI (as well as odds ratios, relative risks and number needed to harm), whether there were pre-specified interim analyses and early stopping rules, and whether or not there were prior proof-of-principle (also known as Phase II) RCTs. Ten RCTs (four industry sponsored) of many interventions (high oxygen delivery, diaspirin cross-linked hemoglobin, growth hormone, methylprednisolone, hetastarch, high-frequency oscillation ventilation, intensive insulin, NOS inhibition, and beta-2 adrenergic agonist, TNF-α receptor) included 19,126 patients and were associated with wide ranges of intervention versus control group mortality rates (25.7–59 %, mean 29.9 vs 17–49 %, mean 25 %, respectively) yielding ARIs of 2.6–29 % (mean 5 %). All but two RCTs had pre-specified interim analyses, and seven RCTs were stopped early. All RCTs were preceded by published proof-of-principle RCT(s), two by the same group. Seven interventions (except diaspirin cross-linked hemoglobin and the NOS inhibitor) were available for use clinically at the time of the pivotal RCT. Common, clinically available interventions used in the critically ill were associated with increased mortality in large, pivotal RCTs even though safety was often addressed by interim analyses and early stopping rules.

Treatment effect heterogeneity for univariate subgroups in clinical trials: Shrinkage, standardization, or else

Treatment effect heterogeneity is a well-recognized phenomenon in randomized controlled clinical trials. In this paper, we discuss subgroup analyses with prespecified subgroups of clinical or biological importance. We explore various alternatives to the naive (the traditional univariate) subgroup analyses to address the issues of multiplicity and confounding. Specifically, we consider a model-based Bayesian shrinkage (Bayes-DS) and a nonparametric, empirical Bayes shrinkage approach (Emp-Bayes) to temper the optimism of traditional univariate subgroup analyses; a standardization approach (standardization) that accounts for correlation between baseline covariates; and a model-based maximum likelihood estimation (MLE) approach. The Bayes-DS and Emp-Bayes methods model the variation in subgroup-specific treatment effect rather than testing the null hypothesis of no difference between subgroups. The standardization approach addresses the issue of confounding in subgroup analyses. The MLE approach is considered only for comparison in simulation studies as the "truth" since the data were generated from the same model. Using the characteristics of a hypothetical large outcome trial, we perform simulation studies and articulate the utilities and potential limitations of these estimators. Simulation results indicate that Bayes-DS and Emp-Bayes can protect against optimism present in the naïve approach. Due to its simplicity, the naïve approach should be the reference for reporting univariate subgroup-specific treatment effect estimates from exploratory subgroup analyses. Standardization, although it tends to have a larger variance, is suggested when it is important to address the confounding of univariate subgroup effects due to correlation between baseline covariates. The Bayes-DS approach is available as an R package (DSBayes).

Deficiencies in reporting of statistical methodology in recent randomized trials of nonpharmacologic pain treatments: ACTTION systematic review

OBJECTIVE

The goal of this study was to assess the quality of reporting of statistical methods in randomized clinical trials (RCTs), including identification of primary analyses, missing data accommodation, and multiplicity adjustment, in studies of nonpharmacologic, noninterventional pain treatments (e.g., physical therapy, cognitive behavioral therapy, acupuncture, and massage).

STUDY DESIGN

Systematic review of 101 articles reporting RCTs of pain treatments that were published between January 2006 and June 2013 in the European Journal of Pain, the Journal of Pain, and Pain.

SETTING

Systematic review.

RESULTS

Sixty-two percent of studies identified a primary outcome variable, 46% identified a primary analysis, and of those with multiple primary analyses, only 21% adjusted for multiplicity. Slightly over half (55%) of studies reported using at least one method to accommodate missing data. Only four studies reported prespecifying at least one of these four study methods.

CONCLUSION

This review identified deficiencies in the reporting of primary analyses and methods to adjust for multiplicity and accommodate missing data in articles disseminating results of nonpharmacologic, noninterventional trials. Investigators should be encouraged to indicate whether their analyses were prespecified and to clearly and completely report statistical methods in clinical trial publications to maximize the interpretability of trial results.

Data interpretation in analgesic clinical trials with statistically nonsignificant primary analyses: an ACTTION systematic review

Peer-reviewed publications of randomized clinical trials (RCTs) are the primary means of disseminating research findings. "Spin" in RCT publications is misrepresentation of statistically nonsignificant research findings to suggest treatment benefit. Spin can influence the way readers interpret clinical trials and use the information to make decisions about treatments and medical policies. The objective of this study was to determine the frequency with which 4 types of spin were used in publications of analgesic RCTs with nonsignificant primary analyses in 6 major pain journals. In the 76 articles included in our sample, 28% of the abstracts and 29% of the main texts emphasized secondary analyses with P values <.05; 22% of abstracts and 29% of texts emphasized treatment benefit based on nonsignificant primary results; 14% of abstracts and 18% of texts emphasized within-group improvements over time, rather than primary between-group comparisons; and 13% of abstracts and 10% of texts interpreted a nonsignificant difference between groups in a superiority study as comparable effectiveness. When considering the article conclusion sections, 21% did not mention the nonsignificant primary result, 22% were presented with no uncertainty or qualification, 30% did not acknowledge that future research was required, and 8% recommended the intervention for clinical use.

PERSPECTIVE

This article identifies relatively frequent "spin" in analgesic RCTs. These findings highlight a need for authors, reviewers, and editors to be more cognizant of how analgesic RCT results are presented and attempt to minimize spin in future clinical trial publications.

Reporting of primary analyses and multiplicity adjustment in recent analgesic clinical trials: ACTTION systematic review and recommendations

Performing multiple analyses in clinical trials can inflate the probability of a type I error, or the chance of falsely concluding a significant effect of the treatment. Strategies to minimize type I error probability include prespecification of primary analyses and statistical adjustment for multiple comparisons, when applicable. The objective of this study was to assess the quality of primary analysis reporting and frequency of multiplicity adjustment in 3 major pain journals (ie, European Journal of Pain, Journal of Pain, and PAIN®). A total of 161 randomized controlled trials investigating noninvasive pharmacological treatments or interventional treatments for pain, published between 2006 and 2012, were included. Only 52% of trials identified a primary analysis, and only 10% of trials reported prespecification of that analysis. Among the 33 articles that identified a primary analysis with multiple testing, 15 (45%) adjusted for multiplicity; of those 15, only 2 (13%) reported prespecification of the adjustment methodology. Trials in clinical pain conditions and industry-sponsored trials identified a primary analysis more often than trials in experimental pain models and non-industry-sponsored trials, respectively. The results of this systematic review demonstrate deficiencies in the reporting and possibly the execution of primary analyses in published analgesic trials. These deficiencies can be rectified by changes in, or better enforcement of, journal policies pertaining to requirements for the reporting of analyses of clinical trial data.

Evaluating effects of developmental education for college students using a regression discontinuity design

BACKGROUND

Annually, American colleges and universities provide developmental education (DE) to millions of underprepared students; however, evaluation estimates of DE benefits have been mixed.

OBJECTIVES

Using a prototypic exemplar of DE, our primary objective was to investigate the utility of a replicative evaluative framework for assessing program effectiveness.

RESEARCH DESIGN

Within the context of the regression discontinuity (RD) design, this research examined the effectiveness of a DE program for five, sequential cohorts of first-time college students. Discontinuity estimates were generated for individual terms and cumulatively, across terms.

SUBJECTS

Participants were 3,589 first-time community college students.

MEASURES

DE program effects were measured by contrasting both college-level English grades and a dichotomous measure of pass/fail, for DE and non-DE students.

RESULTS

Parametric and nonparametric estimates of overall effect were positive for continuous and dichotomous measures of achievement (grade and pass/fail). The variability of program effects over time was determined by tracking results within individual terms and cumulatively, across terms. Applying this replication strategy, DE's overall impact was modest (an effect size of approximately .20) but quite consistent, based on parametric and nonparametric estimation approaches. A meta-analysis of five RD results yielded virtually the same estimate as the overall, parametric findings. Subset analysis, though tentative, suggested that males benefited more than females, while academic gains were comparable for different ethnicities.

CONCLUSION

The cumulative, within-study comparison, replication approach offers considerable potential for the evaluation of new and existing policies, particularly when effects are relatively small, as is often the case in applied settings.

Portfolio of clinical research in adult cardiovascular disease as reflected in ClinicalTrials.gov

Background

Cardiovascular medicine is widely regarded as a vanguard for evidence‐based drug and technology development. Our goal was to describe the cardiovascular clinical research portfolio from ClinicalTrials.gov.

Methods and Results

We identified 40 970 clinical research studies registered between 2007 and 2010 in which patients received diagnostic, therapeutic, or other interventions per protocol. By annotating 18 491 descriptors from the National Library of Medicine's Medical Subject Heading thesaurus and 1220 free‐text terms to select those relevant to cardiovascular disease, we identified studies that related to the diagnosis, treatment, or prevention of diseases of the heart and peripheral arteries in adults (n=2325 [66%] included from review of 3503 potential studies). The study intervention involved a drug in 44.6%, a device or procedure in 39.3%, behavioral intervention in 8.1%, and biological or genetic interventions in 3.0% of the trials. More than half of the trials were postmarket approval (phase 4, 25.6%) or not part of drug development (no phase, 34.5%). Nearly half of all studies (46.3%) anticipated enrolling 100 patients or fewer. The majority of studies assessed biomarkers or surrogate outcomes, with just 31.8% reporting a clinical event as a primary outcome.

Conclusions

Cardiovascular studies registered on ClinicalTrials.gov span a range of study designs. Data have limited verification or standardization and require manual processes to describe and categorize studies. The preponderance of small and late‐phase studies raises questions regarding the strength of evidence likely to be generated by the current portfolio and the potential efficiency to be gained by more research consolidation.

SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials

High quality protocols facilitate proper conduct, reporting, and external review of clinical trials. However, the completeness of trial protocols is often inadequate. To help improve the content and quality of protocols, an international group of stakeholders developed the SPIRIT 2013 Statement (Standard Protocol Items: Recommendations for Interventional Trials). The SPIRIT Statement provides guidance in the form of a checklist of recommended items to include in a clinical trial protocol. This SPIRIT 2013 Explanation and Elaboration paper provides important information to promote full understanding of the checklist recommendations. For each checklist item, we provide a rationale and detailed description; a model example from an actual protocol; and relevant references supporting its importance. We strongly recommend that this explanatory paper be used in conjunction with the SPIRIT Statement. A website of resources is also available (www.spirit-statement.org). The SPIRIT 2013 Explanation and Elaboration paper, together with the Statement, should help with the drafting of trial protocols. Complete documentation of key trial elements can facilitate transparency and protocol review for the benefit of all stakeholders.

DHEA for poor responders: can treatment be justified in the absence of evidence?

Evidence-based medicine is the application of science to the practice of healthcare, leading to reproducibility and transparency in the science supporting healthcare practice. Dehydroepiandrosterone (DHEA) has been proposed as a treatment for improving ovarian reserve in poor responding women undergoing IVF. However, there is a lack of evidence of its effectiveness as data is derived mostly from retrospective studies and one questionable randomized trial. DHEA appears to be devoid of major side effects, however, due lack of data supporting its effectiveness, its wide scale use cannot be currently recommended.

Final tips in interpreting evidence-based medicine

In this issue, the Southern Medical Journal presents a series of articles to help students of medicine understand the principles of evidence-based medicine. These articles are not meant to be a comprehensive review, but rather an easy-to-read primer. In this final article, the authors offer suggestions to aid the reader in navigating the ever-expanding amount of information. These tips address a number of points that are commonly encountered in the medical literature, but are not all-inclusive.

An adaptive strategy for assessing regional consistency in multiregional clinical trials

Background Unexpected regional difference in treatment effect has been reported in recent multiregional clinical trials (MRCTs). This may cause difficulty in interpreting results and can have regulatory implications such as marketing approvals and/or product labels in various markets. Careful consideration of consistency across regions and appropriate plans to address potential regional difference are necessary at the design stage. However, assessment of consistency in treatment effect is generally not the primary objective, and therefore, when there is no strong a priori reason to expect a regional difference, a MRCT is not usually designed to address the regional consistency. Unexpected regional finding may arise and increase the risk of ambiguous or controversial results at the end of the study.

Purpose To mitigate this risk, we propose an adaptive strategy for regional assessment based upon accumulated blinded data.

Methods If review of accumulated blinded data shows unexpectedly severe imbalance in an intrinsic or extrinsic factor, and further assessment indicates that this factor could be a potential effect modifier as supported by biological plausibility or blinded correlation analysis, a stratified regional analysis controlled for this factor may be specified and documented before database lock.

Results The proposed adaptive strategy can help with the interpretation of unexpected regional finding. A recent trial is used to illustrate the approach.

Limitations Even if the imbalanced factor may appear to explain away the regional difference, establishment of causal effect is usually difficult and requires more involved effort.

Conclusions This approach, by prespecifying the stratified analysis, can reduce the risk of post hoc exaggerated emphases across many possible exploratory analyses and provide greater confidence in the validity of the conclusions. If a causal effect can be established that the apparent regional difference is likely caused by this intrinsic or extrinsic factor, this prespecified analysis can also help guide clinical practice.

Secondary analysis of clinical trials--a cautionary note

There is concern in published reports and reviews that patients are being harmed or denied effective treatment by the use of questionable results from secondary analyses of data from clinical trials. A well-reported secondary analysis must make clear to the reader the uncertainty of the result--so clear, in fact, that it should be an obvious part of the conclusions that implementation should await confirmation as the primary outcome in an adequately powered trial. Those who write, review and publish these reports have a responsibility to ensure that reports accurately describe the sources of uncertainty, explain complex methods and their weaknesses with clarity, and convince readers to require better evidence before changing their practice.

Accelerating the development of improved analgesic treatments: the ACTION public-private partnership

There has been considerable progress identifying pathophysiologic mechanisms of neuropathic pain, but analgesic medications with improved efficacy, safety, and tolerability still represent an unmet public health need. Numerous treatments examined in recent randomized clinical trials (RCTs) have failed to show efficacy for neuropathic pain, including treatments that had previously demonstrated efficacy. This suggests that at least some negative results reflect limited assay sensitivity of RCTs to distinguish efficacious treatments from placebo. Patient characteristics, clinical trial research designs and methods, outcome measures, approaches to data analysis, and statistical power may all play a role in accounting for difficulties in demonstrating the benefits of efficacious analgesic treatments vs placebo. The identification of specific clinical trial characteristics associated with assay sensitivity in existing data has the potential to provide an evidence-based approach to the design of analgesic clinical trials. The US Food and Drug Administration recently launched the Analgesic Clinical Trial Innovations, Opportunities, and Networks (ACTION) public-private partnership, which is designed to facilitate the discovery and development of analgesics with improved efficacy, safety, and tolerability for acute and chronic pain conditions. ACTION will establish a collaborative effort to prioritize research objectives, develop a standardized analgesic database platform, and conduct methodologically focused studies to increase the assay sensitivity and efficiency of analgesic clinical trials. The results of these activities have the potential to inform and accelerate the development of improved pain management interventions of all types, not just pharmacologic treatments.

Secondary use of randomized controlled trials to evaluate drug safety: a review of methodological considerations

Background Randomized clinical trials (RCTs) are often positioned at the top of evidence hierarchies. Meta-analyses of RCTs aim to integrate the state of knowledge on a given scientific question, particularly for rare drug-related outcomes. However, although RCTs are valuable tools in our armamentarium, they are rarely designed to evaluate drug safety and are thus susceptible to limitations that may hamper the ability of both RCTs and meta-analyses to fully characterize the safety profiles of drugs. Their potential limitations might be exacerbated in the study of rare outcomes, often encountered in drug safety assessment, when even minor deviations from the intended randomization could impact the stability of the risk estimates.

Purpose This article considers the methodological caveats of both RCTs and meta-analyses of RCTs pertinent to the study of drug-related harms. It is intended to stimulate discussion about the impact of these caveats on interpreting findings of RCTs and meta-analyses for drug safety, which would foster more robust, critical evaluations, and thus enhance clinical and regulatory decision-making.

Methods Pertinent issues that can influence the interpretation of drug-related harms discussed in this article were based on authors’ expertise and review of the literature.

Results Investigators and clinicians should be cognizant of the potential limitations of the secondary use of RCTs and meta-analyses in the assessment of drug-related harms and, when applicable, should consider potential remedies to overcome these limitations.

Limitations Only few practical examples are included in the article due to the fact that many of the discussed caveats are not examined and/or reported in many publications. In addition, the confidential nature of data reviewed at a regulatory agency forestalls an in depth discussion of examples pertaining to specific drugs. Furthermore, our ability to quantify the extent of encountering, or the actual impact of, the caveats addressed in this review on the RCTs findings is limited. It is worth noting that the mere encounter of a given caveat does not mean that it will obviate the utility of drug safety information from a given trial. The extent of its impact is expected to vary based on the specifics of the trial, the drugs studied, the indications, and the nature of the adverse events.

Conclusions Although some of the limitations described are inherent in RCTs, some of the sources of bias highlighted in this article could be minimized by careful RCT design, planned follow-up, and improved collection of information on adverse events. As future research sheds more light on pertinent knowledge gaps and issues, the ability to maximize the use of RCTs and meta-analyses of RCTs to address drug safety questions of interest will be greatly enhanced.

Some essential considerations in the design and conduct of non-inferiority trials

BACKGROUND

Suppose a standard therapy (Standard) has been established to provide a clinically important reduction in risk of irreversible morbidity or mortality. In that setting, the safety and efficacy of an experimental intervention likely would be assessed in a clinical trial providing a comparison with Standard rather than a placebo arm. Such a trial often is designed to assess whether the efficacy of the experimental intervention is not unacceptably worse than that of Standard, and is called a non-inferiority trial. Formally, the non-inferiority trial usually is designed to rule out a non-inferiority margin, defined as the minimum threshold for what would constitute an unacceptable loss of efficacy.

PURPOSE

Even though the literature has many important articles identifying various approaches to the design and conduct of non-inferiority trials, confusion remains especially regarding key considerations for selecting the non-inferiority margin. The purpose of this article is to provide improved clarity regarding these considerations.

METHODS

We present scientific insights into many factors that should be addressed in the design and conduct of non-inferiority trials to enhance their integrity and reliability, and provide motivation for key considerations that guide the selection of non-inferiority margins. We also provide illustrations and insights from recent experiences.

RESULTS

Two considerations are essential, and should be addressed in separate steps, in the formulation of the non-inferiority margin. First, the margin should be formulated using adjustments to account for bias or lack of reliability in the estimate of the effect of Standard in the non-inferiority trial setting. Second, the non-inferiority margin should be formulated to achieve preservation of an appropriate percentage of the effect of Standard.

LIMITATIONS

The considerations, in particular regarding the importance of preservation of effect, might not apply to settings where it would be ethical as well as clinically relevant to include both Standard and placebo arms in the trial for direct comparisons with the experimental intervention arm.

CONCLUSIONS

Non-inferiority trials with non-rigorous margins allow substantial risk for accepting inadequately effective experimental regimens, leading to the risk of erosion in quality of health care. The design and conduct of non-inferiority trials, including selection of non-inferiority margins, should account for many factors that can induce bias in the estimated effect of Standard in the non-inferiority trial and thus lead to bias in the estimated effect of the experimental treatment, for the need to ensure the experimental treatment preserves a clinically acceptable fraction of Standard's effect, and for the particular vulnerability of the integrity of a non-inferiority trial to the irregularities in trial conduct. Due to the inherent uncertainties in non-inferiority trials, alternative designs should be pursued whenever possible.

Adjuvant chemotherapy in 2011 for patients with soft-tissue sarcoma

The mainstay of treatment for adults with soft-tissue sarcomas is wide surgical excision. Half of all patients with adequate local control of high-grade sarcomas develop distant metastases and, despite additional treatment, ultimately die from their disease. This daunting reality has resulted in a three-decade research effort to assess the efficacy of adjuvant therapy for adult soft-tissue sarcomas. The multitude of histopathological subtypes, each with its own biology and clinical behavior, and the rarity of adult soft-tissue sarcomas as a whole greatly complicate such an assessment. This Perspectives article examines data that support or refute the use of adjuvant chemotherapy in the treatment of soft-tissue sarcomas.