Learning from past mistakes: assessing trial quality, power and eligibility in non-renal systemic lupus erythematosus randomized controlled trials

Pro-Con Debate: Interdisciplinary Perspectives on Industry-Sponsored Research

Pro: Nearly all new devices and drugs come from industry that provides two-thirds of the funding for medical research, and a much higher fraction of clinical research. Realistically, without corporate-funded studies, perioperative research would stagnate with little innovation and few new products. Opinions are ubiquitous and normal, but do not constitute epidemiologic bias. Competent clinical research includes many protections against selection and measurement bias, and the publication process provides at least moderate protection against misinterpretation of results. Trial registries largely prevent selective data presentation. Sponsored trials are particularly protected against inappropriate corporate influence because they are usually codesigned with the US Food and Drug Administration, and analyses are based on formal predefined statistical plans, as well as being conducted with rigorous external monitoring. Novel products, which are essential for advances in clinical care, largely come from industry, and industry appropriately funds much of the required research. We should celebrate industry's contribution to improvements in clinical care. Con: While industry funding contributes to research and discovery, examples of industry-funded research demonstrate bias. In the setting of financial pressures and potential conflict of interest, bias can influence the type of study design, hypotheses being tested, rigor and transparency in data analysis, interpretation, as well as reporting of the results. Unlike public granting agencies, industry does not necessarily provide funding based on unbiased peer review following an open call for proposals. The focus on success can influence the choice of a comparator, which might not be ideal among the possible alternatives, the language used in the publication, and even the ability to publish. Unpublished negative trials can result in selected information being withheld from the scientific community and the public. Appropriate safeguards are needed to ensure that research addresses the most important and relevant questions, that results are available even when they do not support the use of a product produced by the funding company, that populations studied reflect the relevant patients, that the most rigorous approaches are applied, that studies have the appropriate power to address the question posed, and that conclusions are presented in an unbiased manner.

Pro-Con Debate: Interdisciplinary Perspectives on Industry-Sponsored Research

PRO: Nearly all new devices and drugs come from industry that provides two-thirds of the funding for medical research, and a much higher fraction of clinical research. Realistically, without corporate-funded studies, perioperative research would stagnate with little innovation and few new products. Opinions are ubiquitous and normal but do not constitute epidemiologic bias. Competent clinical research includes many protections against selection and measurement bias, and the publication process provides at least moderate protection against misinterpretation of results. Trial registries largely prevent selective data presentation. Sponsored trials are particularly protected against inappropriate corporate influence because they are usually codesigned with the US Food and Drug Administration, and analyses are based on formal predefined statistical plans, as well as being conducted with rigorous external monitoring. Novel products, which are essential for advances in clinical care, largely come from industry, and industry appropriately funds much of the required research. We should celebrate industry's contribution to improvements in clinical care. CON: While industry funding contributes to research and discovery, examples of industry-funded research demonstrate bias. In the setting of financial pressures and potential conflict of interest, bias can influence the type of study design, hypotheses being tested, rigor and transparency in data analysis, interpretation, as well as reporting of the results. Unlike public granting agencies, industry does not necessarily provide funding based on unbiased peer review following an open call for proposals. The focus on success can influence the choice of a comparator, which might not be ideal among the possible alternatives, the language used in the publication, and even the ability to publish. Unpublished negative trials can result in selected information being withheld from the scientific community and the public. Appropriate safeguards are needed to ensure that research addresses the most important and relevant questions, that results are available even when they do not support the use of a product produced by the funding company, that populations studied reflect the relevant patients, that the most rigorous approaches are applied, that studies have the appropriate power to address the question posed, and that conclusions are presented in an unbiased manner.

Lessons learned from the failure of several recent trials with biologic treatment in systemic lupus erythematosus

INTRODUCTION

Treatment of systemic lupus erythematosus (SLE) represents a challenge due to variable disease manifestations, clinical course, and outcome. Long-term outcome in SLE remain unsatisfactory and a search for new therapeutic options is definitely warranted. Despite expectations, most clinical trials performed in SLE and lupus nephritis in the last decade did not reach primary outcome, and the only drug that has been licensed is belimumab.

AREAS COVERED

Results of negative trials testing monoclonal antibodies and other biologic agents in SLE are briefly summarized. Reasons for the failure of the trials are listed and discussed.

EXPERT OPINION

Future studies should recruit patients with similar organ involvement, better defined disease manifestations, higher activity, and similar severity. In addition to testing higher efficacy if given as add-on treatment to standard-of-care, the trials should be aimed at reducing dosing, or completely eliminating some parts of the current standard treatment, especially corticosteroids. Median follow-up of the patients should be longer. Moreover, specific biomarkers are needed to help to identify eligible patients and to better define response to treatment. An urgent unmet need is testing these new drugs in patients with severe SLE (including those refractory to current treatment).

Randomized Trials, Meta-Analyses, and Systematic Reviews: Using Examples from Rheumatology

This article introduces contemporary ideas and standards for clinical research in rheumatology for randomized trials, systematic reviews, and meta-analyses. Examples of different randomized trials in rheumatic diseases are provided to understand the methods for trials and the rationale for outcomes within trials. Insights from meta-analyses and systematic literature reviews, including network meta-analyses within rheumatology treatment, are provided. Ethical considerations, sample size calculations, and types of randomized controlled trials are discussed.

Sample size of surgical randomized controlled trials: a lack of improvement over time

BACKGROUND

Interpretation of randomized controlled trials (RCTs) without a significant difference regarding the primary outcome (negative RCTs) is frequently challenging, due to concerns about sample size and thus sufficient statistical power. We aimed to assess the adequacy of sample size and corresponding power of surgical RCTs.

METHODS

We previously identified all surgical RCTs available in PubMed in two distinct years a decade apart (1999 and 2009). For all "negative" trials, we estimated whether the sample size of the trial was appropriate to detect a difference in the primary outcome measure. The main outcome measure was a sufficient sample size to detect large, medium, and small treatment effects. We also performed a post hoc power analysis based on the actual observed effect difference.

RESULTS

A total of 228 negative RCTs (74 in 1999 and 121 in 2009) were included. The median sample size was 76 (± 222) and 80 (± 163) in 1999 and 2009, respectively. Sample size calculation was increasingly reported from 40% in 1999 to 54% in 2009 (P = 0.02). The proportion of studies adequately powered to detect large (57% versus 68%), medium (26% versus 25%), or small (8% versus 7%) differences did not differ significantly between 1999 and 2009, respectively. To reach sufficient power, the required increases in sample size were 130%, 240%, and 1032% for large, medium, and small differences, respectively. Reporting a sample size calculation was the only independent predictor for adequate power.

CONCLUSIONS

Despite slight improvement in the reporting of a sample size calculation, about a third of surgical trials remains underpowered to demonstrate differences that are likely to be clinically significant. Increased attention of researchers, medical ethical boards, and journal editors is required to reduce potentially wasted resources on underpowered trials.

Characteristics of funding of clinical trials: cross-sectional survey and proposed guidance

OBJECTIVES

To provide a detailed and current characterisation of funding of a representative sample clinical trials. We also aimed to develop guidance for standardised reporting of funding information.

METHODS

We addressed the extent to which clinical trials published in 2015 in any of the 119 Core Clinical Journals included a statement on the funding source (eg, whether a not-for-profit organisation was supported by a private-for-profit organisation), type of funding, amount and role of funder. We used a stepwise approach to develop a guidance and an instrument for standardised reporting of funding information.

RESULTS

Of 200 trials, 178 (89%) included a funding statement, of which 171 (96%) reported being funded. Funding statements in the 171 funded trials indicated the source in 100%, amount in 1% and roles of funders in 50%. The most frequent sources were governmental (58%) and private-for-profit (40%). Of 54 funding statements in which the source was a not-for-profit organisation, we found evidence of undisclosed support of those from private-for-profit organisation(s) in 26 (48%). The most frequently reported roles of funders in the 171 funded trials related to study design (42%) and data analysis, interpretation or management (41%). Of 139 randomised controlled trials (RCTs) addressing pharmacological or surgical interventions, 29 (21%) reported information on the supplier of the medication or device. The proposed guidance addresses both the funding information that RCTs should report and the reporting process. Attached to the guidance is a fillable PDF document for use as an instrument for standardised reporting of funding information.

CONCLUSION

Although the majority of RCTs report funding, there is considerable variability in the reporting of funding source, amount and roles of funders. A standardised approach to reporting of funding information would address these limitations. Future research should explore the implications of funding by not-for-profit organisations that are supported by for-profit organisations.

Financial conflicts of interest and their association with outcome and quality of fibromyalgia drug therapy randomized controlled trials

AIMS

To evaluate the association of financial conflicts of interest (FCOI) with the characteristics, outcome and reported methodological quality of fibromyalgia drug therapy randomized controlled trials (FM-RCTs).

METHODS

A cross-sectional study of original, parallel-group, drug therapy FM-RCTs published between 1997 and 2011 from Medline and Cochrane Central Register of Controlled Trials was conducted. Two reviewers independently assessed each RCT for funding source, authors' FCOI(s), study characteristics, reporting of methodological measures important for internal validity and outcome (positive [statistically significant result favoring experimental drug for the primary outcome] or non-positive).

RESULTS

Forty-seven RCTs were eligible with funding source as: 26 (55.3%) industry; eight (17%) non-profit source(s); five (10.6%) mixed; and eight (17%) unspecified. Industry-funded RCTs were more likely to be multicenter and enroll greater number of patients. Reporting of key methodological measures was suboptimal; however, industry and non-profit funded RCTs did not differ in their reporting. Thirty (63.8%) RCTs had ≥ one author who disclosed an FCOI (receipt of research grant [21, 44.7%], industry sponsor employee [20, 42.6%], receipt of consultancy fee/honorarium [16, 34%] and stock ownership [11, 23.4%]). Although industry funding and certain authors' FCOIs (employment and receipt of consultancy fee/honorarium) were univariately associated with positive outcome, such association was not observed after adjusting for study sample size.

CONCLUSIONS

The majority of FM-RCTs were industry-sponsored, and had at least one author with an FCOI. Reporting of key methodological measures was suboptimal. After adjusting for study sample size, no association of industry funding or author's FCOI with study outcome was seen.

Caution regarding the choice of standard deviations to guide sample size calculations in clinical trials

Background

The method used to determine choice of standard deviation (SD) is inadequately reported in clinical trials. Underestimations of the population SD may result in underpowered clinical trials.

Purpose

This study demonstrates how using the wrong method to determine population SD can lead to inaccurate sample sizes and underpowered studies, and offers recommendations to maximize the likelihood of achieving adequate statistical power.

Methods

We review the practice of reporting sample size and its effect on the power of trials published in major journals. Simulated clinical trials were used to compare the effects of different methods of determining SD on power and sample size calculations.

Results

Prior to 1996, sample size calculations were reported in just 1%–42% of clinical trials. This proportion increased from 38% to 54% after the initial Consolidated Standards of Reporting Trials (CONSORT) was published in 1996, and from 64% to 95% after the revised CONSORT was published in 2001. Nevertheless, underpowered clinical trials are still common. Our simulated data showed that all minimal and 25th-percentile SDs fell below 44 (the population SD), regardless of sample size (from 5 to 50). For sample sizes 5 and 50, the minimum sample SDs underestimated the population SD by 90.7% and 29.3%, respectively. If only one sample was available, there was less than 50% chance that the actual power equaled or exceeded the planned power of 80% for detecting a median effect size (Cohen’s d = 0.5) when using the sample SD to calculate the sample size. The proportions of studies with actual power of at least 80% were about 95%, 90%, 85%, and 80% when we used the larger SD, 80% upper confidence limit (UCL) of SD, 70% UCL of SD, and 60% UCL of SD to calculate the sample size, respectively. When more than one sample was available, the weighted average SD resulted in about 50% of trials being underpowered; the proportion of trials with power of 80% increased from 90% to 100% when the 75th percentile and the maximum SD from 10 samples were used. Greater sample size is needed to achieve a higher proportion of studies having actual power of 80%.

Limitations

This study only addressed sample size calculation for continuous outcome variables.

Conclusions

We recommend using the 60% UCL of SD, maximum SD, 80th-percentile SD, and 75th-percentile SD to calculate sample size when 1 or 2 samples, 3 samples, 4–5 samples, and more than 5 samples of data are available, respectively. Using the sample SD or average SD to calculate sample size should be avoided.

Association of industry funding with the outcome and quality of randomized controlled trials of drug therapy for rheumatoid arthritis

OBJECTIVE

To assess the association of industry funding with the characteristics, outcome, and reported quality of randomized controlled trials (RCTs) of drug therapy for rheumatoid arthritis (RA).

METHODS

The Medline and Cochrane Central Register of Controlled Trials databases were searched to identify original RA drug therapy RCTs published in 2002-2003 and 2006-2007. Two reviewers independently assessed each RCT for the funding source, characteristics, outcome (positive [statistically significant result favoring experimental drug for the primary outcome] or not positive), and reporting of methodologic measures whose inadequate performance may have biased the assessment of treatment effect. RCTs that were registered at ClinicalTrials.gov and completed during the study years were assessed for publication bias.

RESULTS

Of the 103 eligible RCTs identified, 58 (56.3%) were funded by industry, 19 (18.4%) were funded by nonprofit sources, 6 (5.8%) had mixed funding, and funding for 20 (19.4%) was not specified. Industry-funded RCTs had significantly more study centers and subjects, while nonprofit agency-funded RCTs had longer duration and were more likely to study different treatment strategies. Outcome could be assessed for 86 (83.5%) of the 103 RCTs studied. The funding source was not associated with a higher likelihood of positive outcomes favoring the sponsored experimental drug (75.5% of industry-funded RCTs had a positive outcome, compared with 68.8% of non-industry-funded RCTs, 40% of RCTs with mixed funding, and 81.2% of RCTs for which funding was not specified). Industry-funded RCTs showed a trend toward a higher likelihood of nonpublication (P=0.093). Industry-funded RCTs were more frequently associated with double-blinding, an adequate description of participant flow, and performance of an intent-to-treat analysis.

CONCLUSION

Industry funding was not associated with a higher likelihood of positive outcomes of published RCTs of drug therapy for RA, and industry-funded RCTs performed significantly better than non-industry-funded RCTs in terms of reporting the use of some key methodologic quality measures.

Quality of reporting of neonatal and infant trials in high-impact journals

OBJECTIVES

To perform a systematic review of the quality of reporting for randomized controlled trials (RCTs) with infants and neonates that were published in high-impact journals and to identify RCT characteristics associated with quality of reporting.

METHODS

RCTs that enrolled infants younger than 12 months and were published in 2005-2009 in 6 pediatric or general medical journals were reviewed. Eligible RCTs were evaluated for the presence of 11 quality criteria selected from the Consolidated Standards of Reporting Trials guidelines. The relationships between quality of reporting and key study characteristics were tested with nonparametric statistics.

RESULTS

Two reviewers had very good agreement regarding the eligibility of studies (κ = 0.85) and the presence of quality criteria (κ = 0.82). Among 179 eligible RCTs, reporting of the individual quality criteria varied widely. Only 50% included a flow diagram, but 99% reported the number of study participants. Higher quality of reporting was associated with greater numbers of study participants, publication in a general medical journal, and greater numbers of centers (P < .0001 for each comparison). Geographic region and positive study outcomes were not associated with reporting quality.

CONCLUSIONS

The quality of reporting of infant and neonatal RCTs is inconsistent, particularly in pediatric journals. Therefore, readers cannot assess accurately the validity of many RCT results. Strict adherence to the Consolidated Standards of Reporting Trials guidelines should lead to improved reporting.

Canadian Recommendations for Clinical Trials of Pharmacologic Interventions in Rheumatoid Arthritis: Inclusion Criteria and Study Design: Table 1

Objective.

Current clinical trial designs for pharmacologic interventions in rheumatoid arthritis (RA) do not reflect the innovations in RA diagnosis, treatment, and care in countries where new drugs are most often used. The objective of this project was to recommend revised entry criteria and other study design features for RA clinical trials.

Methods.

Recommendations were developed using a modified nominal group consensus method. Canadian Rheumatology Research Consortium (CRRC) members were polled to rank the greatest challenges to clinical trial recruitment in their practices. Initial recommendations were developed by an expert panel of rheumatology trialists and other experts. A scoping study methodology was then used to examine the evidence available to support or refute each initial recommendation. The potential influence of CRRC recommendations on primary outcomes in future trials was examined. Recommendations were finalized using a consensus process.

Results.

Recommendations for clinical trial inclusion criteria addressed measures of disease activity [Disease Activity Score 28 using erythrocyte sedimentation rate (DAS28-ESR) > 3.2 PLUS ≥ 3 tender joints using 28-joint count (TJC28) PLUS ≥ 3 swollen joint (SJC28) OR C-reactive protein (CRP) or ESR > upper limit of normal PLUS ≥ 3 TJC28 PLUS ≥ 3 SJC28], functional classification, disease classification and duration, and concomitant RA treatments. Additional recommendations regarding study design addressed rescue strategies and longterm extension.

Conclusion.

There is an urgent need to modify clinical trial inclusion criteria and other study design features to better reflect the current characteristics of people living with RA in the countries where the new drugs will be used.

The financing of drug trials by pharmaceutical companies and its consequences. Part 1: a qualitative, systematic review of the literature on possible influences on the findings, protocols, and quality of drug trials

BACKGROUND

In recent years, a number of studies have shown that clinical drug trials financed by pharmaceutical companies yield favorable results for company products more often than independent trials do. Moreover, pharmaceutical companies have been found to influence drug trials in various ways. This paper provides an overview of the findings of current, systematic studies on this topic.

METHODS

Publications retrieved from a systematic Medline search on this topic from 1 November 2002 to 16 December 2009 were independently evaluated and selected by two of the authors. These publications were supplemented by further ones found in their references sections.

RESULTS

57 publications were included for evaluation in Parts 1 and 2 of this article. Published drug trials that were financed by pharmaceutical companies, or whose authors declared a financial conflict of interest, were found to yield favorable results for the drug manufacturer more frequently than independently financed trials whose authors had no such conflicts. The results were also interpreted favorably more often than in independently financed trials. Furthermore, there was evidence that pharmaceutical companies influenced study protocols in a way that was favorable to themselves. The methodological quality of trials financed by pharmaceutical companies was not found to be any worse than that of trials financed in other ways.

CONCLUSION

Published drug trials that are financed by pharmaceutical companies may present a distorted picture. This cannot be explained by any difference in methodological quality between such trials and trials financed in other ways.

Characteristics of opioid-using pregnant women who accept or refuse participation in a clinical trial: screening results from the MOTHER study

BACKGROUND

Although concerns arise about the generalizability of results from Randomized Controlled Trials (RCTs), few studies systematically examine this issue.

OBJECTIVES

This study compared the characteristics of 427 opioid-using pregnant women who did (n = 208) and did not consent (n = 219) to enrollment in a multicenter clinical trial of agonist medications (i.e., the MOTHER study).

METHODS

Logistic regression models were used to compare consenters and non-consenters to examine the effect of screening variables on the likelihood of consenting.

RESULTS

Of nine characteristics examined, most differences did not reach statistical significance. Consenting participants were less likely than non-consenting women to be currently enrolled in a methadone maintenance program (74.5% vs. 84.5%, p =.01).

CONCLUSION AND SCIENTIFIC SIGNIFICANCE

These data show that the recruited sample of drug-dependent pregnant women enrolled in an intensive RCT is representative of the larger population of treated opioid-dependent patients and supports the generalizability of randomized controlled trials in this population.

Two negative randomized controlled trials in lupus: now what?

Recently, two large randomized controlled trials of distinct biologic therapies in systemic lupus erythematosus, B-cell depletion with rituximab and co-stimulatory blockade with CTLA4Ig (abatacept), failed to meet primary endpoints. Given the great need for new treatments in lupus, these results were met with disappointment and have left the rheumatology and immunology community searching for an explanation. Are these experimental agents ineffective in lupus or are there trial design issues or other considerations? In this commentary, we discuss our perspective on these results within the context of current understanding of the pathophysiology of lupus and the mechanism of action of biologic therapies.