Baseline P value distributions in randomized trials were uniform for continuous but not categorical variables

Quantitative assessment of baseline imbalances in evolocumab and alirocumab trials: a meta-epidemiological study

BACKGROUND

Baseline imbalances have been identified in randomized trials of evolocumab and alirocumab. Our aim was to quantitatively assess (1) the presence of systematic baseline differences, and (2) the relationship of baseline differences with effects on low-density lipoprotein-cholesterol (LDL-c) and clinical outcomes in the trials.

METHODS

We performed a meta-epidemiological study. PubMed, Embase, regulatory reports, ClinicalTrials.gov and company websites were searched for trials. Seven baseline characteristics (mean age, LDL-c, BMI, percentage males, diabetics, smokers, and hypertensives) and five outcomes (LDL-c, major adverse cardiac events, serious adverse events, any adverse events, all-cause mortality) were extracted. We calculated (1) range and distribution of baseline imbalances (sign-test), (2) pooled baseline differences and heterogeneity (meta-analysis), (3) differences in SDs around continuous variables (sign-test and pooling), and (4) the relationship of baseline differences with outcomes (meta-regression). The comparisons of PCSK9-inhibitor groups with either placebo or ezetimibe were analysed separately and combined.

RESULTS

We identified 43 trials with 63,193 participants. Baseline characteristics were frequently missing. Many trials showed small baseline imbalances, but some large imbalances. Only baseline BMI showed a statistically significant lower pooled mean for the drug versus placebo groups (MD -0.16; 95% CI -0.24 to -0.09). Heterogeneity in baseline imbalances was present in six placebo- and five ezetimibe-comparisons. Heterogeneity was statistically significant for BMI, males, diabetics and hypertensives in the combined comparisons. There was a statistically significant preponderance for larger SDs in the PCSK9-inhibitor versus control groups (sign-test age 0.014; LDL-c 0.014; BMI 0.049). Meta-regression showed clinically relevant relationships of baseline imbalances in age, BMI and diabetics with the risk of any adverse events and the risk of mortality. Two relationships were statistically significant: A higher mean BMI in the drug versus control group with a decreased risk of mortality (beta - 0.56; 95% CI -1.10 to -0.02), and a higher proportion of diabetics with an increased risk of any adverse events (beta 0.02; 95% 0.01 to 0.04).

CONCLUSIONS

Heterogeneous baseline imbalances and systematically different SDs were present in evolocumab and alirocumab trials, so study groups cannot be assumed to be comparable. These findings raise concerns about the design and conduct of the randomization procedures.

Statistical techniques to assess publication integrity in groups of randomized trials: a narrative review

OBJECTIVES

To describe statistical tools available for assessing publication integrity of groups of randomized controlled trials (RCTs).

STUDY DESIGN AND SETTING

Narrative review.

RESULTS

Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomization occurred. For continuous variables, the tools assess baseline means, baseline P values, and the occurrence of identical means and/or standard deviation. For categorical variables, they assess baseline P values, frequency counts for individual or all variables, numbers of trial participants randomized or withdrawing, and compare reported with independently calculated P values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission.

CONCLUSION

Statistical tools are useful for the assessment of publication integrity of groups of RCTs.

Post-publication research integrity concerns in randomized clinical trials: A scoping review of the literature

BACKGROUND

Post-publication handling of integrity concerns in randomized clinical trials (RCTs) is a contentious matter.

OBJECTIVES

We undertook a scoping systematic review to map the literature regarding post-publication integrity issues in RCTs.

SEARCH STRATEGY AND SELECTION CRITERIA

Following prospective registration (https://osf.io/pgxd8) we initially searched PubMed and Scopus but subsequently extended it to include the Cochrane Library, and Google Scholar databases without language, article type or publication time restriction until November 2022. Reviewers independently selected published articles covering any aspect of post-publication research integrity concerns in RCTs.

DATA COLLECTION AND ANALYSIS

The study findings grouped within domains relating to issues concerning post-publication integrity were extracted in duplicate, verified by a third reviewer, and then tabulated.

MAIN RESULTS

The initial search captured 3159 citations, of which 89 studies were included in the review. Cross-sectional studies constituted the majority of included studies (n = 34, 38.2%), followed by systematic reviews (n = 10, 11.2%), methodology reviews/studies (n = 9, 10.1%) and other types of descriptive studies (n = 8, 9.0%). A total of 21 articles (23.6%) covered the domain on general issues, 25 (28.1%) in the journal's instructions and policies domain, eight (9.0%) in the editorial and peer review domain, one (1.1%) in the correspondence and complaints (post-publication peer review) domain, 12 (13.5%) in the investigation for concerns domain, six (6.7%) in the post-investigation decisions and sanctions domain, none in the critical appraisal guidance domain, five (5.6%) in the integrity assessment in systematic reviews domain, and 26 (29.2%) in the recommendations for future research domain. A total of 12 of the selected articles (13.5%) covered two (n = 9) or three (n = 3) different domains.

CONCLUSIONS

Various research integrity domains and issues covering post-publication aspects of RCT integrity were captured and gaps were identified, mostly related with the necessary implications for all stakeholders to improve research transparency. There is an urgent need for a multistakeholder consensus towards creating specific statements for addressing post-publication integrity concerns in RCTs.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Automated detection of over- and under-dispersion in baseline tables in randomised controlled trials

Background: Papers describing the results of a randomised trial should include a baseline table that compares the characteristics of randomised groups. Researchers who fraudulently generate trials often unwittingly create baseline tables that are implausibly similar (under-dispersed) or have large differences between groups (over-dispersed). I aimed to create an automated algorithm to screen for under- and over-dispersion in the baseline tables of randomised trials. Methods: Using a cross-sectional study I examined 2,245 randomised controlled trials published in health and medical journals on PubMed Central. I estimated the probability that a trial's baseline summary statistics were under- or over-dispersed using a Bayesian model that examined the distribution of t-statistics for the between-group differences, and compared this with an expected distribution without dispersion. I used a simulation study to test the ability of the model to find under- or over-dispersion and compared its performance with an existing test of dispersion based on a uniform test of p-values. My model combined categorical and continuous summary statistics, whereas the uniform test used only continuous statistics. Results: The algorithm had a relatively good accuracy for extracting the data from baseline tables, matching well on the size of the tables and sample size. Using t-statistics in the Bayesian model out-performed the uniform test of p-values, which had many false positives for skewed, categorical and rounded data that were not under- or over-dispersed. For trials published on PubMed Central, some tables appeared under- or over-dispersed because they had an atypical presentation or had reporting errors. Some trials flagged as under-dispersed had groups with strikingly similar summary statistics. Conclusions: Automated screening for fraud of all submitted trials is challenging due to the widely varying presentation of baseline tables. The Bayesian model could be useful in targeted checks of suspected trials or authors.

Distributions of baseline categorical variables were different from the expected distributions in randomized trials with integrity concerns

BACKGROUND AND OBJECTIVES

Comparing observed and expected distributions of baseline continuous variables in randomized controlled trials (RCTs) can be used to assess publication integrity. We explored whether baseline categorical variables could also be used.

METHODS

The observed and expected (binomial) distribution of all baseline categorical variables were compared in four sets of RCTs: two controls, and two with publication integrity concerns. We also compared baseline calculated and reported P-values.

RESULTS

The observed and expected distributions of baseline categorical variables were similar in the control datasets, both for frequency counts (and percentages) and for between-group differences in frequency counts. However, in both sets of RCTs with publication integrity concerns, about twice as many variables as expected had between-group differences in frequency counts of one or 2, and far fewer variables than expected had between-group differences of >4 (P < 0.001 for both datasets). Furthermore, about one in six reported P-values for baseline categorial variables differed by > 0.1 from the calculated P-value in trials with publication integrity concerns.

CONCLUSION

Comparing the observed and expected distributions and reported and calculated P-values of baseline categorical variables may help in the assessment of publication integrity of a body of RCTs.

Evidence and risk indicators of non-random sampling in clinical trials in implant dentistry: A systematic appraisal

Aim

Analysis of distribution of p‐values of continuous differences between test and controls after randomization provides evidence of unintentional error, non‐random sampling, or data fabrication in randomized controlled trials (RCTs). We assessed evidence of highly unusual distributions of baseline characteristics of subjects enrolled in clinical trials in implant dentistry.

Materials and methods

RCTs published between 2005 and 2020 were systematically searched in Pubmed, Embase, and Cochrane databases. Baseline patient data were extracted from full text articles by two independent assessors. The hypothesis of non‐random sampling was tested by comparing the expected and the observed distribution of the p‐values of differences between test and controls after randomization.

Results

One‐thousand five‐hundred and thirty‐eight unique RCTs were identified, of which 409 (26.6%) did not report baseline characteristics of the population, and 671 (43.6%) reported data in forms other than mean and standard deviation and could not be used to assess their random sampling. Four‐hundred and fifty‐eight trials with 1449 baseline variables in the form of mean and standard deviation were assessed. The study observed an over‐representation of very small p‐values [<.001, 1.38%, 95% confidence interval (CI) 0.85–2.12 compared to the expected 0.10%, 95% CI 0.00–0.26]. No evidence of over‐representation of larger p‐values was observed. Unusual distributions were present in 2.38% of RCTs and more frequent in non‐registered trials, in studies supported by non‐industry funding, and in multi‐centre RCTs.

Conclusions

The inability to assess random sampling due to insufficient reporting in 26.6% of trials requires attention. In trials reporting suitable baseline data, unusual distributions were uncommon, and no evidence of data fabrication was detected, but there was evidence of non‐random sampling. Continued efforts are necessary to ensure high integrity and trust in the evidence base of the field.

Errors in the implementation, analysis, and reporting of randomization within obesity and nutrition research: a guide to their avoidance

Randomization is an important tool used to establish causal inferences in studies designed to further our understanding of questions related to obesity and nutrition. To take advantage of the inferences afforded by randomization, scientific standards must be upheld during the planning, execution, analysis, and reporting of such studies. We discuss ten errors in randomized experiments from real-world examples from the literature and outline best practices for their avoidance. These ten errors include: representing nonrandom allocation as random, failing to adequately conceal allocation, not accounting for changing allocation ratios, replacing subjects in nonrandom ways, failing to account for non-independence, drawing inferences by comparing statistical significance from within-group comparisons instead of between-groups, pooling data and breaking the randomized design, failing to account for missing data, failing to report sufficient information to understand study methods, and failing to frame the causal question as testing the randomized assignment per se. We hope that these examples will aid researchers, reviewers, journal editors, and other readers to endeavor to a high standard of scientific rigor in randomized experiments within obesity and nutrition research.

Assessing Research Misconduct in Randomized Controlled Trials

Randomized controlled trials (RCTs) serve as the pillar of evidence-based medicine and guide medical practice. Compromised data integrity in RCTs undermines the authority of this valuable tool for science and puts patients at risk. Although a large number of retractions due to data issues in obstetrics and gynecology have occurred in the past few years, many problematic RCTs could still go uncovered because in general there is insufficient willingness to envisage and confront research misconduct. In this article, we discuss the necessity of assessing research misconduct, summarize methods that have been applied in detecting previous cases of misconduct, and propose potential solutions. There is no established mechanism to monitor feedback on published articles and the current system that handles potential research misconduct is unsatisfactory. Fortunately, there are methods to assess data integrity in RCTs both with and without individual participant data. Investigations into research misconduct can be facilitated by assessing all publications from a leading author or author group to identify duplication and patterns of ongoing misconduct. There is a pressing need to improve the mechanism that investigates data manipulation. The mechanism that handles misconduct should prioritize the interests of patients and readers rather than trial authors and their institutions. An equally urgent issue is to establish mechanisms that prevent compromised trials from polluting evidence synthesis or misguiding practice.

Methods to assess research misconduct in health-related research: A scoping review

OBJECTIVE

To give an overview of the available methods to investigate research misconduct in health-related research.

STUDY DESIGN AND SETTING

In this scoping review, we conducted a literature search in MEDLINE, Embase, The Cochrane CENTRAL Register of Studies Online (CRSO), and The Virtual Health Library portal up to July 2020. We included papers that mentioned and/or described methods for screening or assessing research misconduct in health-related research. We categorized identified methods into the following four groups according to their scopes: overall concern, textual concern, image concern, and data concern.

RESULTS

We included 57 papers reporting on 27 methods: two on overall concern, four on textual concern, three on image concern, and 18 on data concern. Apart from the methods to locate textual plagiarism and image manipulation, all other methods, be it theoretical or empirical, are based on examples, are not standardized, and lack formal validation.

CONCLUSION

Existing methods cover a wide range of issues regarding research misconduct. Although measures to counteract textual plagiarism are well implemented, tools to investigate other forms of research misconduct are rudimentary and labour-intensive. To cope with the rising challenge of research misconduct, further development of automatic tools and routine validation of these methods is needed.

TRIAL REGISTRATION NUMBER

Center for Open Science (OSF) (https://osf.io/mq89w).

An investigation of seven other publications by the first author of a retracted paper due to doubts about data integrity

BACKGROUND

In 2019, a randomized controlled trial (RCT) authored by Dr. Ismail was retracted due to concerns about data integrity. Since there are no policies in place to investigate other publications of authors of retracted studies, we investigated Dr. Ismail's other trials.

METHODS

We searched for RCTs authored by Dr. Ismail. We made pairwise comparisons of values in baseline and outcome tables between trials. We assessed whether the distributions of baseline characteristics were compatible with properly conducted randomization, using Monte Carlo simulations and the Kolmogorov-Smirnov test. We read the publications carefully for unusual features.

RESULTS

Dr. Ismail was author in eight published and one unpublished trial. In three of his first author studies we found multiple identical values in the baseline and/or outcome tables from different trials. At least some of the trials were unlikely to have followed a proper randomization process. There were a number of other unusual features in the papers we reviewed.

CONCLUSIONS

It is probable that other trials published by Dr. Ismail contain questionable data. We call for a thorough investigation of the original trial data and related official documents. Our exercise suggests that the practice to assess research integrity should include all publications of authors with retracted fabricated articles.

Participant withdrawals were unusually distributed in randomized trials with integrity concerns: a statistical investigation

OBJECTIVES

Comparing observed and expected distributions of categorical outcome variables in randomized controlled trials (RCTs) has been previously used to assess publication integrity. We applied this technique to withdrawals from RCTs.

STUDY DESIGN AND SETTING

We compared the observed distribution of withdrawals with the expected binomial distribution in six sets of RCTs: four control sets and two sets with concerns about their publication integrity.

RESULTS

In the control data sets (n = 13, 115, 71, and 36 trials, respectively), the observed distributions of withdrawals were consistent with the expected distributions, both for the numbers of withdrawals per trial arm and for the differences in withdrawals between trial arms in two-arm RCTs. In contrast, in both sets of RCTs with concerns regarding publication integrity (n = 151 and 35 trials, respectively), there were striking differences between the observed and expected distributions of trial withdrawals. Two-arm RCTs from the two sets with publication integrity concerns were 2.6 (95% confidence interval 2.0-3.3) times more likely to have a difference of 0 or 1 withdrawals between trial arms than control RCTs (P < 0.001). Simulating a 50% higher rate of withdrawals in active treatment arms in the largest set of control RCTs still produced an observed distribution of withdrawals per trial arm consistent with the expected distribution.

CONCLUSION

Comparing the observed and expected distribution of trial withdrawals may be a useful technique when considering publication integrity of a body of RCTs.

Article placement order in rheumatology journals: a content analysis

OBJECTIVES

To analyse variables associated with article placement order in serial rheumatology journals.

DESIGN

Content analysis.

SETTING

Original articles published in seven rheumatology journals from 2013 to 2018.

PRIMARY AND SECONDARY OUTCOME MEASURES

The following data were extracted from 6787 articles: order number of article in issue, gender of first and last author, geographical region, industry funding, research design and disease category. Cumulative density function plots were used to determine whether article placement distribution was different from the expected distribution. ORs for articles published in the first three places of an issue compared with the last three places were calculated. Altmetric Score and downloads were meta-analysed.

RESULTS

Article placement order did not associate with author gender or geographical region but was associated with funding source and research design. In addition, articles about rheumatoid arthritis were more likely to be ordered at the front of issues (p<0.001). Articles about crystal arthritis, systemic lupus erythematosus, vasculitis, pain syndromes and paediatric rheumatic diseases were more likely to be ordered at the end of issues (all p<0.001). Association of article placement order with disease category was observed only in journals with tables of contents grouped by disease. Articles ordered in the first three places had higher Altmetric and download rates, than articles in the last three places.

CONCLUSIONS

Author gender and geographical region do not influence article placement order in serial rheumatology journals. However, bias for certain disease categories is reflected in article placement order. Editorial decisions about article placement order can influence the prominence of diseases.