Raw data were not disclosed in 95% of PubMed-indexed heart failure meta-analyses in 2021: A systematic analysis of transparency

BACKGROUND

The rising concern of irreproducible and non-transparent studies poses a significant challenge in modern medical literature. The impact of this issue on cardiology, particularly in the subfield of heart failure, remains poorly understood. To address this knowledge gap, we assessed the quality of evidence presented in recent heart failure meta-analyses by exploring several crucial transparency indicators.

METHODS

We conducted a cross-sectional study and searched PubMed for meta - analyses themed around heart failure. We included the 100 most recent publications from 2021 and investigated the presence of several indices that are associated with transparency and reproducibility.

RESULTS

The vast majority of the papers did not include their raw data (95/100, 95%) nor their analytic code (99/100, 99%). Less than half (42/100, 42%) preregistered their protocol, while only 65/100 (65%) adhered to a reporting guidelines method. Bias calculation for the respective studies included in each meta - analysis was present in 83/100 (83%) papers and publication bias was measured in approximately half (56/100, 56%).

CONCLUSIONS

Our study indicates that meta-analyses in the field of heart failure present important information of transparency infrequently. Therefore, reproduction and validation of their findings seems to be practically impossible.

Trials that turn from retrospectively registered to prospectively registered: a cohort study of "retroactively prospective" clinical trial registration using history data

BACKGROUND

Prospective registration of clinical trials is mandated by various regulations. However, clinical trial registries like ClinicalTrials.gov allow registry entries to be updated at any time, and key study elements, including the start date, may change before the first patient is enrolled. If a trial changes its start date after recruiting began, however, it may indicate a reason for concern. This study aimed to measure the rate of "retroactively prospective" trials. This refers to trials that are originally registered retrospectively, with the start date before the registration date, but that retroactively change their start date to be after the registration date, making them appear as if they were prospectively registered.

METHODS

We retrieved clinical trial history data for all clinical trials registered on ClinicalTrials.gov with a first registration date in the year 2015 (N = 11,908). Using automated analyses, we determined the timepoints of registration in relation to the start date of the trial over time. For retroactively prospective trials and a set of control trials, we manually checked the accompanying publications to determine which start date they report and whether they report changes to the start date.

RESULTS

We found 235 clinical trials to be retroactively prospective, comprising 2.0% of all clinical trials in our sample of 11,908 trials. Among the 113 retroactively prospective clinical trials with an accompanying publication, 12 (10.6%) explicitly stated in the publication that they had been prospectively registered.

CONCLUSIONS

Retroactively prospective trial registration happens in one in 50 trials. While these changes to the start date could be mistakes or legitimate edits based on the most up-to-date information, they could also indicate a retrospectively registered trial that has been made to appear as a prospectively registered trial, which would lead to biases unapparent to reviewers. Our results point to the need for more transparent reporting of changes to a trial's details and have implications for the review and conduct of clinical trials, with our fully automated and freely available tools allowing reviewers or editors to detect these changes.

TRIAL REGISTRATION

The preregistered protocol of our study is available via https://osf.io/rvq53 . The most recent version of the protocol lists all deviations from the original study plan, including the rationale behind the changes, and additional analyses that were conducted.

Consumption of ultra-processed foods and multiple health outcomes: An umbrella study of meta-analyses

Consumption of ultra-processed foods (UPFs) is associated with various adverse health outcomes, which significantly influence the global disease burden. This umbrella review aimed to fill the knowledge gap and guide public health practices by summarizing the association between UPFs and multiple health outcomes. A total of four databases were systematically searched from inception to December 2022, and 14 eligible systematic reviews (SRs) with meta-analyses (MAs) were identified. The SRs were published in 10 journals from 2020 to 2023, with 54,147-5,750,133 participants and 5-61 studies. The overall corrected covered area (CCA) was corresponded to a slight overlap. The results showed that an increased UPFs consumption is associated with multiple health outcomes (e.g., obesity, diabetes, hypertension, mortality). Only two SRs were "Moderate" regarding the overall methodological quality, while the other twelve were "Low" or "Critically low". Therefore, well-conducted SRs with high-quality prospective cohorts with a particular focus on special populations are needed to verify these findings further.

Navigating PROSPERO4animals: 10 top tips for efficient pre-registration of your animal systematic review protocol

Systematic reviews are an essential tool in identifying knowledge gaps and synthesizing evidence from in vivo animal research to improve human health. The review process follows an explicit and systematic methodology to minimize bias, but is not immune to biases or methodological flaws. Pre-registering a systematic review protocol has several benefits, including avoiding unplanned duplication of reviews, reducing reporting biases, and providing structure throughout the review process. It also helps to align the opinions of review team members and can shield researchers from post-hoc critique. PROSPERO4animals is the international prospective register of systematic reviews (PROSPERO) for the preregistration of systematic review of animal studies. As administrators, here we provide 10 tips to facilitate pre-registration in PROSPERO4animals. These tips address common difficulties that both beginners and experienced researchers may face when pre-registering their systematic review protocols. This article aims to help authors write and register a detailed systematic review protocol on PROSPERO4animals.

Evaluation of 'implications for research' statements in systematic reviews of interventions in advanced cancer patients - a meta-research study

BACKGROUND

Implications for research (IfR) sections are an important part of systematic reviews (SRs) to inform health care researchers and policy makers. PRISMA 2020 recommends reporting IfR, while Cochrane Reviews require a separate chapter on IfR. However, it is unclear to what extent SRs discuss IfR. We aimed i) to assess whether SRs include an IfR statement and ii) to evaluate which elements informed IfR statements.

METHODS

We conducted a meta-research study based on SRs of interventions in advanced cancer patients from a previous project (CRD42019134904). As suggested in the Cochrane Handbook, we assessed if the following predefined variables were referred to in IfR statements: patient, intervention, control, outcome (PICO) and study design; concepts underlying Grading of Recommendations, Assessment, Development and Evaluation (GRADE) domains: risk of bias, inconsistency, indirectness, imprecision, publication bias. Data were independently extracted by three reviewers after piloting the data extraction form. Discrepancies were resolved in weekly in-depth discussions.

RESULTS

We included 261 SRs. The majority evaluated a pharmacological intervention (n = 244, 93.5%); twenty-nine were Cochrane Reviews (11.1%). Four out of five SRs included an IfR statement (n = 210, 80.5%). IfR statements commonly addressed 'intervention' (n = 121, 57.6%), 'patient ' (n = 113, 53.8%), and 'study design' (n = 107, 51.0%). The most frequent PICO and study design combinations were 'patient and intervention ' (n = 71, 33.8%) and 'patient, intervention and study design ' (n = 34, 16.2%). Concepts underlying GRADE domains were rarely used for informing IfR recommendations: 'risk of bias ' (n = 2, 1.0%), and 'imprecision ' (n = 1, 0.5%), 'inconsistency ' (n = 1, 0.5%). Additional elements informing IfR were considerations on cost effectiveness (n = 9, 4.3%), reporting standards (n = 4, 1.9%), and individual patient data meta-analysis (n = 4, 1.9%).

CONCLUSION

Although about 80% of SRs included an IfR statement, the reporting of PICO elements varied across SRs. Concepts underlying GRADE domains were rarely used to derive IfR. Further work needs to assess the generalizability beyond SRs in advanced cancer patients. We suggest that more specific guidance on which and how IfR elements to report in SRs of interventions needs to be developed. Utilizing PICO elements and concepts underlying GRADE according to the Cochrane Handbook to state IfR seems to be a reasonable approach in the interim.

REGISTRATION

CRD42019134904.

The current state of preclinical modeling of human diabetic cardiomyopathy using rodents

The prevalence of diabetic cardiomyopathy (DCM), a specific cardiovascular complication of diabetes mellitus, has recently increased. Its pathogenesis is not fully understood, and no consensus regarding therapeutic options has been reached. Experimental studies on rodents are expected to yield further data at the preclinical stage. The present paper describes and quantitatively compares the experimental protocols intended to mimic human DCM. Experimental articles (conducted between 1990 and 2022) were identified from online electronic databases according to the PRISMA Protocol. The Cochrane Q-test was used to estimate study heterogeneity; the quality of each individual study was assessed using SYRCLE's risk of bias tool for animal studies. Sensitivity analysis was performed according to the leave-one-out method. Publication bias across studies was assessed using Egger's weighted regression and Duval and Tweedie 'trim and fill' method. A wide spectrum of protocols - from 651 papers, was examined (type 1 or 2 diabetes mellitus, as well as obesity models). They were found to vary in their presentation of DCM according to a variety of hemodynamic, echocardiographic, histopathologic and metabolic parameters. Particular attention was paid to comorbid conditions, and cardiac performance featured as systolic, diastolic dysfunction, or refractory heart failure. The majority of models displayed diastolic dysfunction, as well as myocardial fibrosis and left ventricle hypertrophy, which mimics early stage DCM. Unlike in humans, animal DCM rarely progressed to the symptomatic heart failure with reduced ejection fraction (HFrEF). The ability of individual procedures to reflect refractory heart failure or biventricular dysfunction - in the end-stage DCM has remained unclear.

Heterogeneity of outcomes in randomized controlled trials on implant prosthodontic therapy is hindering comparative effectiveness research: meta-research study

BACKGROUND

Consistency in outcomes across clinical trials allows for comparing and combining results from different studies. A core outcome set (COS), representing a minimally agreed standardized group of outcomes that should be monitored and measured through research in a specific field of medicine, is not yet available for trials in implant prosthodontic (dental implant) therapy. This meta-research study aimed to analyze outcomes used in clinical trials on implant prosthodontic therapy.

METHODS

We searched the Cochrane Oral Health Group (COHG) register to identify systematic reviews of interventions in implant prosthodontic therapy published by October 2023. From the randomized controlled trials (RCTs) included in the relevant reviews, we extracted data on the characteristics of the included trials and the outcomes used. We categorized outcomes into domains.

RESULTS

From 182 systematic reviews in the COHG register, we included 11 systematic reviews on dental implant therapy. The reviews included 117 unique RCTs with 4725 participants, published from 1995 to 2020, which analyzed 74 different outcomes. Using different definitions, implant failure was analyzed in 73 RCTs. Seventeen RCTs did not define implant failure. Failure was most often (30 RCTs) followed up for one year. Only one RCT assessed implant failure after five years. Trials used 17 definitions of implant failure, while 17 trials did not report on the criteria of implant failure. Complications were analyzed in 48 RCTs, although they were not clearly defined in 12 RCTs. Failure of prosthodontic supra-structure was analyzed in 74 RCTs, with definitions of failure and criteria not clearly defined in 44 RCTs. Trials considered adverse events, peri-implant tissue health, patient attitudes, and other outcomes, including cost, aesthetics, or procedure duration. These outcomes were often different between trials. Twenty-six outcomes were used only once per study.

CONCLUSIONS

Clinical trials in implant prosthodontics used different outcomes, different definitions of outcomes and used different times to monitor them. Standardization of outcomes is necessary to allow comparability and evidence synthesis about the effectiveness of implant prosthodontic therapy.

Poor statistical reporting, inadequate data presentation and spin persist despite Journal awareness and updated Information for Authors

Sound reporting of research results is fundamental to good science. Unfortunately, poor reporting is common and does not improve with editorial educational strategies. We investigated whether publicly highlighting poor reporting at a journal can lead to improved reporting practices. We also investigated whether reporting practices that are required or strongly encouraged in journal Information for Authors are enforced by journal editors and staff. A 2016 audit highlighted poor reporting practices in the Journal of Neurophysiology. In August 2016 and 2018, the American Physiological Society updated the Information for Authors, which included the introduction of several required or strongly encouraged reporting practices. We audited Journal of Neurophysiology papers published in 2019 and 2020 (downloaded through the library of the University of New South Wales) on reporting items selected from the 2016 audit, the newly introduced reporting practices, and items from previous audits. Summary statistics (means, counts) were used to summarize audit results. In total, 580 papers were audited. Compared to results from the 2016 audit, several reporting practices remained unchanged or worsened. For example, 60% of papers erroneously reported standard errors of the mean, 23% of papers included undefined measures of variability, 40% of papers failed to define a statistical threshold for their tests, and when present, 64% of papers with p-values between 0.05 and 0.1 misinterpreted them as statistical trends. As for the newly introduced reporting practices, required practices were consistently adhered to by 34 to 37% of papers, while strongly encouraged practices were consistently adhered to by 9 to 26% of papers. Adherence to the other audited reporting practices was comparable to our previous audits. Publicly highlighting poor reporting practices did little to improve research reporting. Similarly, requiring or strongly encouraging reporting practices was only partly effective. Although the present audit focused on a single journal, this is likely not an isolated case. Stronger, more strategic measures are required to improve poor research reporting.

Recognizing the value of meta-research and making it easier to find

Meta-research is a bourgeoning field studying topics with significant relevance to health sciences librarianship, such as research reproducibility, peer review, and open access. As a discipline that studies research itself and the practices of researchers, meta-research spans disciplines and encompasses a broad spectrum of topics and methods. The breadth of meta-research presents a significant challenge for identifying published meta-research studies. Introducing a subject heading for meta-research in the controlled vocabularies of literature databases has the potential to increase the visibility of meta-research, further advance the field, and expand its impact on research practices. Given the relatively recent designation of meta-research as a field and its expanding use as a term, now is the time to develop appropriate indexing vocabulary. We seek to call attention to the value of meta-research for health sciences librarianship, describe the challenges of identifying meta-research literature with currently available key terms, and highlight the need to establish controlled vocabulary specific to meta-research.

Methodology reporting improved over time in 176,469 randomized controlled trials

OBJECTIVES

To describe randomized controlled trial (RCT) methodology reporting over time.

STUDY DESIGN AND SETTING

We used a deep learning-based sentence classification model based on the Consolidated Standards of Reporting Trials (CONSORT) statement, considered minimum requirements for reporting RCTs. We included 176,469 RCT reports published between 1966 and 2018. We analyzed the reporting trends over 5-year time periods, grouping trials from 1966 to 1990 in a single stratum. We also explored the effect of journal impact factor (JIF) and medical discipline.

RESULTS

Population, Intervention, Comparator, Outcome (PICO) items were commonly reported during each period, and reporting increased over time (e.g., interventions: 79.1% during 1966-1990 to 87.5% during 2010-2018). Reporting of some methods information has increased, although there is room for improvement (e.g., sequence generation: 10.8-41.8%). Some items are reported infrequently (e.g., allocation concealment: 5.1-19.3%). The number of items reported and JIF are weakly correlated (Pearson's r (162,702) = 0.16, P < 0.001). The differences in the proportion of items reported between disciplines are small (<10%).

CONCLUSION

Our analysis provides large-scale quantitative support for the hypothesis that RCT methodology reporting has improved over time. Extending these models to all CONSORT items could facilitate compliance checking during manuscript authoring and peer review, and support metaresearch.

Credibility at stake: only two-thirds of randomized trials of nutrition interventions are registered and lack transparency in outcome and treatment effect definitions

OBJECTIVES

This study aimed to investigate the adherence of randomized controlled trials of nutrition interventions to transparency practices informing assessments of selective reporting biases, including the availability of a trial registration entry, protocol and statistical analysis plan (SAP).

STUDY DESIGN AND SETTING

Retrospective observational study with cross-sectional design. We systematically searched for trials published from 1 July 2019, to 30 June 2020, and included a randomly selected sample of 400 studies. We searched for registry entries, protocols, and SAPs for all included studies. We extracted data to characterize the disclosure of sufficient information in the available materials to inform assessments of selective reporting biases, considering the definition of outcome domain, measure, metric, method of aggregation, time point, analysis population, methods to handle missing data and method of adjustment.

RESULTS

Most trials (69%) were registered, but these often lacked sufficient specification of outcomes and intended treatment effects. Protocols and SAPs provided more details but were less often available (14% and 3%, respectively), and even then, almost all studies presented limited information to inform the assessments of risk of bias due to the selection of the reported result.

CONCLUSION

Lack of full specification of outcomes and intended treatment effects hinder a full adherence of randomized controlled trials of nutrition interventions to transparency practices and may affect their credibility.

Analysis of the adequacy of control arms in oncology randomised clinical trials published between 2017 and 2021: a meta-research study

INTRODUCTION

Randomised controlled trials (RCTs) are usually considered the highest level of evidence for clinical practice. Patients assigned to control arm in RCTs should always receive the best available treatments to protect participants while also allowing for proper interpretation and applicability of study results. Here we analysed RCTs published in oncology between 2017 and 2021 to describe the frequency of suboptimal control arms.

METHODS

We identified phase III studies testing active treatments in patients with solid tumours among 11 major oncology journals. Each control arm was analysed, and the standard of care was determined according to international guidelines and scientific evidence at accrual beginning and until accrual completion. We identified studies with suboptimal control arm from the beginning (type 1) and studies with an initially optimal control arm which became outdated during the accrual period (type 2).

RESULTS

This analysis included 387 studies. Forty-three (11.1%) control arms were judged as suboptimal: 24 (6.2%) type 1 and 19 (4.9%) type 2. These rates were higher in industry-sponsored compared to academic trials: 9.3% versus 1.9% for type 1 (p = 0.003); 7.9% versus 0.6% for type 2 (p = 0.001). Rates of suboptimal control arms were higher in studies with positive results: 8.1% versus 4.0% for type 1 (p = 0.09); 7.6% versus 1.7% for type 2 (p = 0.007).

CONCLUSIONS

Many trials have suboptimal control arms, even in journals with high-impact factors, leading to suboptimal treatment of control patients and biased evaluation of trial results.

Pilot trials may improve the quality of full-scale trials: a meta-research study

OBJECTIVES

Evidence on the value of pilot trials for subsequent trial's quality is scarce. This study aims to determine if a pilot trial improves the quality of the full-scale trial.

STUDY DESIGN AND SETTING

We searched PubMed for pilot trials and their subsequent full-scale trials. The meta-analysis of the full-scale trials was used to identify other full-scale trials on the same research topic but without a pilot trial. Markers of trial quality included publication outcomes and Cochrane Risk of Bias (RoB) assessment.

RESULTS

Fifty-eight full-scale trials with a pilot trial and 151 full-scale trials without were identified from 47 meta-analyses. Trials with a pilot trial were published 0.9 years sooner (mean ± standard deviation: 1.7 ± 1.0 vs. 2.6 ± 2.0, P = 0.005) and in peer-reviewed journals with higher impact factors (60.9 ± 75.0 vs. 24.8 ± 50.3, P < 0.001). A pilot trial's presence was associated with lower risk of bias in full-scale trial random sequence generation (OR [95% CI]: 4.05 [1.27-12.91]), allocation concealment (2.89 [1.07-7.83]), and participants/researchers masking (4.31 [1.37-13.50]), but not outcome assessment masking (1.03 [0.49-2.18]), incomplete outcome data (1.27 [0.47-3.42]), and selective reporting (1.23 [0.44-3.46]).

CONCLUSION

Conducting a pilot trial may enhance the quality of the subsequent full-scale trial.

Mapping and systematic appraisal of umbrella reviews in epidemiological research: a protocol for a meta-epidemiological study

INTRODUCTION

Umbrella review is one of the terms used to describe an overview of systematic reviews. During the last years, a rapid increase in the number of umbrella reviews on epidemiological studies has been observed, but there is no systematic assessment of their methodological and reporting characteristics. Our study aims to fill this gap by performing a systematic mapping of umbrella reviews in epidemiological research.

METHODS

We will perform a meta-epidemiological study including a systematic review in MEDLINE and EMBASE to identify all the umbrella reviews that focused on systematic reviews of epidemiological studies and were published from inception until December 31, 2022. We will consider eligible any research article which was designed as an umbrella review and summarized systematic reviews and meta-analyses of epidemiological studies. From each eligible article, we will extract information about the research topic, the methodological characteristics, and the reporting characteristics. We will examine whether the umbrella reviews assessed the strength of the available evidence and the rigor of the included systematic reviews. We will also examine whether these characteristics change across time.

DISCUSSION

Our study will systematically appraise the methodological and reporting characteristics of published umbrella reviews in epidemiological literature. The findings of our study can be used to improve the design and conduct of future umbrella reviews, to derive a standardized set of reporting and methodological guidelines for umbrella reviews, and to allow further meta-epidemiological work.

SYSTEMATIC REVIEW REGISTRATION

osf.io/sxzc6.

Dissemination of knowledge from Cochrane Public Health reviews: a bibliographic study

BACKGROUND

Appropriate dissemination of public health evidence is of high importance to ensure that scientific knowledge reaches potential stakeholders and relevant population groups. A wide distrust towards science and its findings indicates that communication thereof remains below its potential. Cochrane Public Health provides an important source of high-quality scientific evidence in the field of public health via reviews with systematic methodology. The aims of this study were to identify (1) dissemination strategies and (2) stakeholders of Cochrane Public Health reviews.

METHODS

This is a bibliographic study with a cross-sectional design. All 68 records (reviews or review protocols) listed on the Cochrane Public Health website ( https://ph.cochrane.org/cph-reviews-and-topics ) up to 8 March 2022 were included. Record characteristics, dissemination strategies, and potential stakeholder details were coded by one author, and 10% of records were checked by another author. Data were analyzed using descriptive statistics or narratively into common themes.

RESULTS

The 68 records were published between 2010 and 2022 and included 15 review protocols and 53 reviews with systematic methodology (46 systematic, 6 rapid, and 1 scoping review). All 53 reviews were disseminated via open-access plain language summaries (PLS) in English with translations into 3-13 other languages. Other dissemination strategies included information on Cochrane websites (e.g., clinical answers or guidelines) available for 41/53 reviews and Cochrane news or blogs that mentioned 19/53 reviews. Overall, 23/68 records mentioned the actual stakeholder involvement in review production, protocol development, or formulation of dissemination plans. The potential stakeholders included several highly diverse groups, such as the general population or specific communities (e.g., racial minority groups), policy and decision makers, and researchers and professionals in various fields (e.g., nutrition, physical activity, education, or care).

CONCLUSIONS

This study shows that Cochrane Public Health reviews are disseminated predominantly via PLS in different languages and via review information on Cochrane websites. Planned dissemination strategies were rarely reported although actual stakeholders were involved in the planning and production of some reviews. The relevance of Cochrane Public Health reviews for non-academic stakeholders and the general population highlights the need for the dissemination of evidence from such reviews beyond academia.

SYSTEMATIC REVIEW REGISTRATION

The study was prospectively registered at the Open Science Framework ( https://osf.io/ga9pt/ ).

NEgatiVE results in Radiomics research (NEVER): A meta-research study of publication bias in leading radiology journals

PURPOSE

The purpose of this study was to conduct a meta-research of radiomics-related articles for the publication of negative results, with a focus on the leading clinical radiology journals due to their purportedly high editorial standards.

METHODS

A literature search was performed in PubMed to identify original research studies on radiomics (last search date: August 16th, 2022). The search was restricted to studies published in Q1 clinical radiology journals indexed by Scopus and Web of Science. Following an a priori power analysis based on our null hypothesis, a random sampling of the published literature was conducted. Besides the six baseline study characteristics, a total of three items about publication bias were evaluated. Agreement between raters was analyzed. Disagreements were resolved through consensus. Statistical synthesis of the qualitative evaluations was presented.

RESULTS

Following a priori power analysis, we included a random sample of 149 publications in this study. Most of the publications were retrospective (95%; 142/149), based on private data (91%; 136/149), centered on a single institution (75%; 111/149), and lacked external validation (81%; 121/149). Slightly fewer than half (44%; 66/149) made no comparison to non-radiomic approaches. Overall, only one study (1%; 1/149) reported negative results for radiomics, yielding a statistically significant binomial test (p < 0.0001).

CONCLUSION

The top clinical radiology journals almost never publish negative results, having a strong bias toward publishing positive results. Almost half of the publications did not even compare their approach with a non-radiomic method.

Publication bias impacts on effect size, statistical power, and magnitude (Type M) and sign (Type S) errors in ecology and evolutionary biology

Collaborative efforts to directly replicate empirical studies in the medical and social sciences have revealed alarmingly low rates of replicability, a phenomenon dubbed the 'replication crisis'. Poor replicability has spurred cultural changes targeted at improving reliability in these disciplines. Given the absence of equivalent replication projects in ecology and evolutionary biology, two inter-related indicators offer the opportunity to retrospectively assess replicability: publication bias and statistical power. This registered report assesses the prevalence and severity of small-study (i.e., smaller studies reporting larger effect sizes) and decline effects (i.e., effect sizes decreasing over time) across ecology and evolutionary biology using 87 meta-analyses comprising 4,250 primary studies and 17,638 effect sizes. Further, we estimate how publication bias might distort the estimation of effect sizes, statistical power, and errors in magnitude (Type M or exaggeration ratio) and sign (Type S). We show strong evidence for the pervasiveness of both small-study and decline effects in ecology and evolution. There was widespread prevalence of publication bias that resulted in meta-analytic means being over-estimated by (at least) 0.12 standard deviations. The prevalence of publication bias distorted confidence in meta-analytic results, with 66% of initially statistically significant meta-analytic means becoming non-significant after correcting for publication bias. Ecological and evolutionary studies consistently had low statistical power (15%) with a 4-fold exaggeration of effects on average (Type M error rates = 4.4). Notably, publication bias reduced power from 23% to 15% and increased type M error rates from 2.7 to 4.4 because it creates a non-random sample of effect size evidence. The sign errors of effect sizes (Type S error) increased from 5% to 8% because of publication bias. Our research provides clear evidence that many published ecological and evolutionary findings are inflated. Our results highlight the importance of designing high-power empirical studies (e.g., via collaborative team science), promoting and encouraging replication studies, testing and correcting for publication bias in meta-analyses, and adopting open and transparent research practices, such as (pre)registration, data- and code-sharing, and transparent reporting.

Synthesis methods other than meta-analysis were commonly used but seldom specified: survey of systematic reviews

OBJECTIVES

To examine the specification and use of summary and statistical synthesis methods, focusing on synthesis methods other than meta-analysis.

STUDY DESIGN AND SETTING

We coded the specification and use of summary and synthesis methods in 100 randomly sampled systematic reviews (SRs) of public health and health systems interventions published in 2018 from the Health Evidence and Health Systems Evidence databases.

RESULTS

Sixty of the 100 SRs used other synthesis methods for some (27/100) or all syntheses (33/100). Of these, 54/60 used vote counting: three based on direction of effect, 36 on statistical significance, and 15 were unclear. Eight SRs summarized effect estimates (for example, using medians). Seventeen SRs used the term 'narrative synthesis' (or equivalent) without describing methods; in practice 15 of these used vote counting. 58/100 SRs used meta-analysis. In SRs providing a rationale for not proceeding with meta-analysis, the most common reason was due to diversity in study characteristics (33/39).

CONCLUSION

Statistical synthesis methods other than meta-analysis are commonly used, but few SRs describe the methods. Improved description of methods is required to allow users to appropriately interpret findings, critique methods used and verify the results. Greater awareness of the serious limitations of vote counting based on statistical significance is required.

Indicators of questionable research practices were identified in 163,129 randomized controlled trials

OBJECTIVES

To explore indicators of the following questionable research practices (QRPs) in randomized controlled trials (RCTs): (1) risk of bias in four domains (random sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcome assessment); (2) modifications in primary outcomes that were registered in trial registration records (proxy for selective reporting bias); (3) ratio of the achieved to planned sample sizes; and (4) statistical discrepancy.

STUDY DESIGN AND SETTING

Full texts of all human RCTs published in PubMed in 1996-2017 were automatically identified and information was collected automatically. Potential indicators of QRPs included author-specific, publication-specific, and journal-specific characteristics. Beta, logistic, and linear regression models were used to identify associations between these potential indicators and QRPs.

RESULTS

We included 163,129 RCT publications. The median probability of bias assessed using Robot Reviewer software ranged between 43% and 63% for the four risk of bias domains. A more recent publication year, trial registration, mentioning of CONsolidated Standards Of Reporting Trials-checklist, and a higher journal impact factor were consistently associated with a lower risk of QRPs.

CONCLUSION

This comprehensive analysis provides an insight into indicators of QRPs. Researchers should be aware that certain characteristics of the author team and publication are associated with a higher risk of QRPs.

Screening for in vitro systematic reviews: a comparison of screening methods and training of a machine learning classifier

OBJECTIVE

Existing strategies to identify relevant studies for systematic review may not perform equally well across research domains. We compare four approaches based on either human or automated screening of either title and abstract or full text, and report the training of a machine learning algorithm to identify in vitro studies from bibliographic records.

METHODS

We used a systematic review of oxygen-glucose deprivation (OGD) in PC-12 cells to compare approaches. For human screening, two reviewers independently screened studies based on title and abstract or full text, with disagreements reconciled by a third. For automated screening, we applied text mining to either title and abstract or full text. We trained a machine learning algorithm with decisions from 2000 randomly selected PubMed Central records enriched with a dataset of known in vitro studies.

RESULTS

Full-text approaches performed best, with human (sensitivity: 0.990, specificity: 1.000 and precision: 0.994) outperforming text mining (sensitivity: 0.972, specificity: 0.980 and precision: 0.764). For title and abstract, text mining (sensitivity: 0.890, specificity: 0.995 and precision: 0.922) outperformed human screening (sensitivity: 0.862, specificity: 0.998 and precision: 0.975). At our target sensitivity of 95% the algorithm performed with specificity of 0.850 and precision of 0.700.

CONCLUSION

In this in vitro systematic review, human screening based on title and abstract erroneously excluded 14% of relevant studies, perhaps because title and abstract provide an incomplete description of methods used. Our algorithm might be used as a first selection phase in in vitro systematic reviews to limit the extent of full text screening required.

Three out of four published systematic reviews on COVID-19 treatments were not registered and one-third of those registered were published: a meta-research study

OBJECTIVES

The aim of this study is to describe (1) registered and (2) published systematic reviews (SRs) on COVID-19 treatments, and to analyze (3) the proportion of publications among registered SRs and (4) the proportion of registrations among published SRs.

STUDY DESIGN AND SETTING

This meta-research study (CRD42021240423) is part of CEOsys (http://www.covid-evidenz.de/). Two reviewers identified protocols in PROSPERO (registered January 2020 to September 2020) and SRs published as preprint or peer-reviewed article in L·OVE (Living OVerview of the Evidence) COVID-19 (by May 2021). SRs of all types assessing COVID-19 treatments in humans were included.

RESULTS

We included 239 PROSPERO protocols and 346 SRs published in L·OVE. In both samples, the affiliation of the corresponding author with an Asian institution, standard SR as review type, and meta-analysis as synthesis method were the most frequent characteristics. Living SRs made up ≤10%. A total of 71 of 239 (29.7%) PROSPERO protocols were published as SR by February 2022, that is, after at least 17 months of follow-up (25 of 71 as preprints, 35.2%). In L·OVE, 261 of 346 (75.4%) SRs published by May 2021 were not registered in PROSPERO.

CONCLUSION

Overall, one-third PROSPERO protocols were published and three-fourth published SRs were not registered. We strongly encourage authors to register and publish their SRs promptly to reduce research waste and to allocate resources efficiently during the pandemic and beyond.

Meta-research evaluating redundancy and use of systematic reviews when planning new studies in health research: a scoping review

BACKGROUND

Several studies have documented the production of wasteful research, defined as research of no scientific importance and/or not meeting societal needs. We argue that this redundancy in research may to a large degree be due to the lack of a systematic evaluation of the best available evidence and/or of studies assessing societal needs.

OBJECTIVES

The aim of this scoping review is to (A) identify meta-research studies evaluating if redundancy is present within biomedical research, and if so, assessing the prevalence of such redundancy, and (B) to identify meta-research studies evaluating if researchers had been trying to minimise or avoid redundancy.

ELIGIBILITY CRITERIA

Meta-research studies (empirical studies) were eligible if they evaluated whether redundancy was present and to what degree; whether health researchers referred to all earlier similar studies when justifying and designing a new study and/or when placing new results in the context of earlier similar trials; and whether health researchers systematically and transparently considered end users' perspectives when justifying and designing a new study.

SOURCES OF EVIDENCE

The initial overall search was conducted in MEDLINE, Embase via Ovid, CINAHL, Web of Science, Social Sciences Citation Index, Arts & Humanities Citation Index, and the Cochrane Methodology Register from inception to June 2015. A 2nd search included MEDLINE and Embase via Ovid and covered January 2015 to 26 May 2021. No publication date or language restrictions were applied.

CHARTING METHODS

Charting methods included description of the included studies, bibliometric mapping, and presentation of possible research gaps in the identified meta-research.

RESULTS

We identified 69 meta-research studies. Thirty-four (49%) of these evaluated the prevalence of redundancy and 42 (61%) studies evaluated the prevalence of a systematic and transparent use of earlier similar studies when justifying and designing new studies, and/or when placing new results in context, with seven (10%) studies addressing both aspects. Only one (1%) study assessed if the perspectives of end users had been used to inform the justification and design of a new study. Among the included meta-research studies evaluating whether redundancy was present, only two of nine health domains (medical areas) and only two of 10 research topics (different methodological types) were represented. Similarly, among the included meta-research studies evaluating whether researchers had been trying to minimise or avoid redundancy, only one of nine health domains and only one of 10 research topics were represented.

CONCLUSIONS THAT RELATE TO THE REVIEW QUESTIONS AND OBJECTIVES

Even with 69 included meta-research studies, there was a lack of information for most health domains and research topics. However, as most included studies were evaluating across different domains, there is a clear indication of a high prevalence of redundancy and a low prevalence of trying to minimise or avoid redundancy. In addition, only one meta-research study evaluated whether the perspectives of end users were used to inform the justification and design of a new study.

SYSTEMATIC REVIEW REGISTRATION

Protocol registered at Open Science Framework: https://osf.io/3rdua/ (15 June 2021).

Spin and reporting in systematic reviews with meta-analysis of randomized clinical trials in restorative dentistry

OBJECTIVE

To estimate the prevalence of spin and completeness of reporting of systematic reviews with metanalysis (SRMAs) in restorative dentistry.

METHODS

Inclusion criteria were SRMAs of randomized clinical trials of restorative dentistry on survival, success, or failure rates of treatment in humans, with no language or year restriction. SRMAs performed with non-RCTs were excluded. PubMed/MEDLINE, Web of Science, Scopus, Embase, and Cochrane Collaboration Library were searched from inception to April 2022. Outcomes were the prevalence of spin (primary outcome) and completeness of reporting (secondary outcome) in the abstract and full text. Data were reported through means and standard deviations or absolute and relative frequencies. Spin in each item was considered low when occurring in less than 25% of the papers, moderate (25 to 75%), or high (more than 75%).

RESULTS

We identified 7029 studies and 49 unique manuscripts were included. There was a moderate presence of spin in the abstracts and low in full texts. In the abstracts, 65.9% did not report adverse events; while in the abstract and full text, more than 16% reported a conclusion containing recommendations for clinical practice not supported by the findings. Regarding completeness of reporting, there was poor reporting for most items in the abstract while there was an adequate report in full texts, except for register name and registration number (not reported in 32.7%).

CONCLUSIONS

Abstract of SRMAs in restorative dentistry should be better reported. Spin and poor reporting were more frequent in the abstracts, which misleads readers and could lead to inadequate clinical recommendations.

CLINICAL SIGNIFICANCE

Spin and incomplete reporting are a threat to evidence-based practice, especially in systematic reviews. Therefore, care providers, researchers, and other stakeholders should be aware of the possibility of spin in systematic reviews and other sources to prevent misinterpretation, which could lead to inadequate decisions and treatments.

Searching for an experimental rodent model of heart failure with preserved ejection fraction: Re-visited

Heart failure with preserved ejection fraction (HFpEF) currently accounts for over 50% of all heart failure cases. It displays a large number of comorbidities, and the pathophysiological mechanisms underlying the disease remain uncertain and treatment options are limited. The heterogeneity and complexity of the disease, and its specific comorbidities, can limit the number of animal models that could ideally mimic it. The current study describes and compares the efficacy of the most popular approaches from a quantitative point of view. A review and meta-analysis of more than 500 experimental protocols was performed with special attention to these models created to induce heart failure by the most common comorbidities associated with human HFpEF, e.g., hypertension, diabetes, obesity and aging. The analysis included a wide spectrum of outcomes (alterations in body weight, lung and left ventricle weights, laboratory, hemodynamic, echocardiographic, and histopathological data as well as animal mortality) and possible covariates that could determine the utility of the particular model, such as animal age, species, experimental period and genetic modification. A wide range of systemic hypertension as well as diabetes (obesity) - related animal models are used for pre-clinical studies on heart failure, but some of them fail to replicate HFpEF. Future studies should include an evaluation of other features besides diastolic dysfunction that confirm that this is an HFpEF model, or the potential to progress to heart failure with reduced ejection fraction (HFrEF).

A longitudinal assessment of trial protocols approved by research ethics committees: The Adherance to SPIrit REcommendations in the UK (ASPIRE-UK) study

Background

To assess the quality of reporting of RCT protocols approved by UK research ethics committees before and after the publication of the Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) guideline.

Methods

We had access to RCT study protocols that received ethical approval in the UK in 2012 (n=103) and 2016 (n=108). From those, we assessed the adherence to the 33 SPIRIT items (i.e. a total of 64 components of the 33 SPIRIT items). We descriptively analysed the adherence to SPIRIT guidelines as proportion of adequately reported items (median and interquartile range [IQR]) and stratified the results by year of approval and sponsor.

Results

The proportion of reported SPIRIT items increased from a median of 64.9% (IQR, 57.6–69.2%) in 2012 to a median of 72.5% (IQR, 65.3–78.3%) in 2016. Industry-sponsored RCTs reported more SPIRIT items in 2012 (median 67.4%; IQR, 64.1–69.4%) compared to non-industry-sponsored trials (median 59.8%; IQR, 46.5–67.7%). This gap between industry- and non-industry-sponsored trials increased in 2016 (industry-sponsored: median 75.6%; IQR, 71.2–79.0% vs non-industry-sponsored: median 65.3%; IQR, 51.6–76.3%).

Conclusions

The adherence to SPIRIT guidelines has improved in the UK from 2012 to 2016 but remains on a modest level, especially for non-industry-sponsored RCTs.

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-022-06516-1.

Joining the meta-research movement: A bibliometric case study of the journal Perspectives on Medical Education

PURPOSE

To conduct a bibliometric case study of the journal Perspectives on Medical Education (PME) to provide insights into the journal's inner workings and to "take stock" of where PME is today, where it has been, and where it might go.

METHODS

Data, including bibliographic metadata, reviewer and author details, and downloads, were collected for manuscripts submitted to and published in PME from the journal's Editorial Manager and Web of Science. Gender of authors and reviewers was predicted using Genderize.io. To visualize and analyze collaboration patterns, citation relationships and term co-occurrence social network analyses (SNA) were conducted. VOSviewer was used to visualize the social network maps.

RESULTS

Between 2012-2019 PME received, on average, 260 manuscripts annually (range = 73-402). Submissions were received from authors in 81 countries with the majority in the United States (US), United Kingdom, and the Netherlands. PME published 518 manuscripts with authors based in 31 countries, the majority being in the Netherlands, US, and Canada. PME articles were downloaded 717,613 times (mean per document: 1388). In total 1201 (55% women) unique peer reviewers were invited and 649 (57% women) completed reviews; 1227 (49% women) unique authors published in PME. SNA revealed that PME authors were quite collaborative, with most authoring articles with others and only a minority (n = 57) acting as single authors.

DISCUSSION

This case study provides a glimpse into PME and offers evidence for PME's next steps. In the future, PME is committed to growing the journal thoughtfully; diversifying and educating editorial teams, authors, and reviewers, and liberating and sharing journal data.

Adherence to the PRISMA statement and its association with risk of bias in systematic reviews published in rehabilitation journals: A meta-research study

•

The mean overall adherence to the PRISMA checklist across the sample of systematic reviews published in rehabilitation journals was 61.4%.

•

A high overall risk of bias was a significant predictor of lower adherence (B=−7.1%; 95%CI −12.1, −2.0).

•

Studies published in fourth quartile journals displayed a lower overall adherence than those published in the first quartile.

•

No association between adherence and publication options and publication year was found.

•

The overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) when the SR protocol was registered

Background

Systematic reviews (SRs) and meta-analyses are essential resources for the clinicians. They allow to evaluate the strengths and the weaknesses of the evidence to support clinical decision-making if they are adequately reported. Little is known in the rehabilitation field about the completeness of reporting of SRs and its relationship with the risk of bias (ROB).

Objectives

Primary: 1) To evaluate the completeness of reporting of systematic reviews (SRs) published in rehabilitation journals by evaluating their adherence to the PRISMA 2009 checklist, 2) To investigate the relationship between ROB and completeness of reporting. Secondary: To study the association between completeness of reporting and journals and study characteristics.

Methods

A random sample of 200 SRs published between 2011 and 2020 in 68 rehabilitation journals was indexed under the “rehabilitation” category in the InCites database. Two independent reviewers evaluated adherence to the PRISMA checklist and assessed ROB using the ROBIS tool. Overall adherence and adherence to each PRISMA item and section were calculated. Regression analyses investigated the association between completeness of reporting, ROB, and other characteristics (impact factor, publication options, publication year, and study protocol registration).

Results

The mean overall PRISMA adherence across the 200 studies considered was 61.4%. Regression analyses show that having a high overall ROB is a significant predictor of lower adherence (B=-7.1%; 95%CI -12.1, -2.0). Studies published in fourth quartile journals displayed a lower overall adherence (B= -7.2%; 95%CI -13.2, -1.3) than those published in first quartile journals; the overall adherence increased (B= 11.9%; 95%CI 5.9, 18.0) if the SR protocol was registered. No association between adherence, publication options, and publication year was found.

Conclusion

Reporting completeness in rehabilitation SRs is suboptimal and is associated with ROB, impact factor, and study registration. Authors of SRs should improve adherence to the PRISMA guideline, and journal editors should implement strategies to optimize the completeness of reporting.

Risk of bias in observational studies using routinely collected data of comparative effectiveness research: a meta-research study

BACKGROUND

To assess the completeness of reporting, research transparency practices, and risk of selection and immortal bias in observational studies using routinely collected data for comparative effectiveness research.

METHOD

We performed a meta-research study by searching PubMed for comparative effectiveness observational studies evaluating therapeutic interventions using routinely collected data published in high impact factor journals from 01/06/2018 to 30/06/2020. We assessed the reporting of the study design (i.e., eligibility, treatment assignment, and the start of follow-up). The risk of selection bias and immortal time bias was determined by assessing if the time of eligibility, the treatment assignment, and the start of follow-up were synchronized to mimic the randomization following the target trial emulation framework.

RESULT

Seventy-seven articles were identified. Most studies evaluated pharmacological treatments (69%) with a median sample size of 24,000 individuals. In total, 20% of articles inadequately reported essential information of the study design. One-third of the articles (n = 25, 33%) raised some concerns because of unclear reporting (n = 6, 8%) or were at high risk of selection bias and/or immortal time bias (n = 19, 25%). Only five articles (25%) described a solution to mitigate these biases. Six articles (31%) discussed these biases in the limitations section.

CONCLUSION

Reporting of essential information of study design in observational studies remained suboptimal. Selection bias and immortal time bias were common methodological issues that researchers and physicians should be aware of when interpreting the results of observational studies using routinely collected data.

Impact of the COVID-19 pandemic on publication dynamics and non-COVID-19 research production

BACKGROUND

The COVID-19 pandemic has severely affected health systems and medical research worldwide but its impact on the global publication dynamics and non-COVID-19 research has not been measured. We hypothesized that the COVID-19 pandemic may have impacted the scientific production of non-COVID-19 research.

METHODS

We conducted a comprehensive meta-research on studies (original articles, research letters and case reports) published between 01/01/2019 and 01/01/2021 in 10 high-impact medical and infectious disease journals (New England Journal of Medicine, Lancet, Journal of the American Medical Association, Nature Medicine, British Medical Journal, Annals of Internal Medicine, Lancet Global Health, Lancet Public Health, Lancet Infectious Disease and Clinical Infectious Disease). For each publication, we recorded publication date, publication type, number of authors, whether the publication was related to COVID-19, whether the publication was based on a case series, and the number of patients included in the study if the publication was based on a case report or a case series. We estimated the publication dynamics with a locally estimated scatterplot smoothing method. A Natural Language Processing algorithm was designed to calculate the number of authors for each publication. We simulated the number of non-COVID-19 studies that could have been published during the pandemic by extrapolating the publication dynamics of 2019 to 2020, and comparing the expected number to the observed number of studies.

RESULTS

Among the 22,525 studies assessed, 6319 met the inclusion criteria, of which 1022 (16.2%) were related to COVID-19 research. A dramatic increase in the number of publications in general journals was observed from February to April 2020 from a weekly median number of publications of 4.0 (IQR: 2.8-5.5) to 19.5 (IQR: 15.8-24.8) (p < 0.001), followed afterwards by a pattern of stability with a weekly median number of publications of 10.0 (IQR: 6.0-14.0) until December 2020 (p = 0.045 in comparison with April). Two prototypical editorial strategies were found: 1) journals that maintained the volume of non-COVID-19 publications while integrating COVID-19 research and thus increased their overall scientific production, and 2) journals that decreased the volume of non-COVID-19 publications while integrating COVID-19 publications. We estimated using simulation models that the COVID pandemic was associated with a 18% decrease in the production of non-COVID-19 research. We also found a significant change of the publication type in COVID-19 research as compared with non-COVID-19 research illustrated by a decrease in the number of original articles, (47.9% in COVID-19 publications vs 71.3% in non-COVID-19 publications, p < 0.001). Last, COVID-19 publications showed a higher number of authors, especially for case reports with a median of 9.0 authors (IQR: 6.0-13.0) in COVID-19 publications, compared to a median of 4.0 authors (IQR: 3.0-6.0) in non-COVID-19 publications (p < 0.001).

CONCLUSION

In this meta-research gathering publications from high-impact medical journals, we have shown that the dramatic rise in COVID-19 publications was accompanied by a substantial decrease of non-COVID-19 research. META-RESEARCH REGISTRATION: https://osf.io/9vtzp/ .

Software review: The JATSdecoder package-extract metadata, abstract and sectioned text from NISO-JATS coded XML documents; Insights to PubMed central's open access database

JATSdecoder is a general toolbox which facilitates text extraction and analytical tasks on NISO-JATS coded XML documents. Its function JATSdecoder() outputs metadata, the abstract, the sectioned text and reference list as easy selectable elements. One of the biggest repositories for open access full texts covering biology and the medical and health sciences is PubMed Central (PMC), with more than 3.2 million files. This report provides an overview of the PMC document collection processed with JATSdecoder(). The development of extracted tags is displayed for the full corpus over time and in greater detail for some meta tags. Possibilities and limitations for text miners working with scientific literature are outlined. The NISO-JATS-tags are used quite consistently nowadays and allow a reliable extraction of metadata and text elements. International collaborations are more present than ever. There are obvious errors in the date stamps of some documents. Only about half of all articles from 2020 contain at least one author listed with an author identification code. Since many authors share the same name, the identification of person-related content is problematic, especially for authors with Asian names. JATSdecoder() reliably extracts key metadata and text elements from NISO-JATS coded XML files. When combined with the rich, publicly available content within PMCs database, new monitoring and text mining approaches can be carried out easily. Any selection of article subsets should be carefully performed with in- and exclusion criteria on several NISO-JATS tags, as both the subject and keyword tags are used quite inconsistently.

Saffron (Crocus sativus L.) and health outcomes: a meta-research review of meta-analyses and an evidence mapping study

BACKGROUND

Although a number of systematic reviews and meta-analyses of saffron (Crocus sativus L.) have been published, no study has comprehensively summarized the clinical evidence from meta-analyses, or assessed the reporting or methodological quality of these reviews.

PURPOSE

The present meta-research study was designed to fill the gaps in knowledge to inform future studies and allow enhanced clinical decision-making on saffron.

METHODS

The PubMed, Cochrane Library, Embase, and CNKI databases were systematically searched from inception to April 3 rd, 2021, for meta-analyses of clinical trials that assessed the efficacy and safety of saffron. PRISMA 2009 and AMSTAR-2 were employed to assess the reporting and methodological quality of meta-analyses identified in the search, respectively. The present study was registered on PROSPERO with registration number CRD42020220274.

RESULTS

Nineteen eligible systematic reviews with meta-analyses published in English were identified from 235 records. These meta-analyses were published in 12 peer-reviewed journals from 2013 to 2021. The heterogeneous results indicated that saffron significantly reduced fasting blood glucose, waist circumference, diastolic blood pressure, concentrations of total cholesterol and low-density lipoprotein cholesterol, and improved symptoms of depression, cognitive function and sexual dysfunction compared with controls (mainly placebos). Common side effects of saffron consumption included nausea, dry mouth, poor appetite, and headache, but no serious adverse reactions were reported. Primary analysis and sensitivity analysis confirmed that the reporting and methodological quality of reviews included in the study were highly correlated (p < 0.001). The quality of meta-analyses of saffron requires improvement by including a structured abstract, a prospective protocol and registration, explanation of the study designs within each study that is reviewed, the searches, risk of bias assessment, literature selection, and reporting of funding sources.

CONCLUSION

The available evidence indicates that saffron is a safe plant for administration as a medicine and can improve diverse clinical outcomes, but the scientific quality of the published systematic reviews needs to be improved. Moreover, the clinical effects of saffron need to be confirmed through high-quality randomized trials in multiple countries with large sample sizes.

Restricting evidence syntheses of interventions to English-language publications is a viable methodological shortcut for most medical topics: a systematic review

OBJECTIVES

To assess the impact of restricting systematic reviews of conventional or alternative medical treatments or diagnostic tests to English-language publications.

STUDY DESIGN AND SETTING

We systematically searched MEDLINE (Ovid), the Science Citation Index Expanded (Web of Science), and Current Contents Connect (Web of Science) up to April 24, 2020. Eligible methods studies assessed the impact of restricting systematic reviews to English-language publications on effect estimates and conclusions. Two reviewers independently screened the literature; one investigator performed the data extraction, a second investigator checked for completeness and accuracy. We synthesized the findings narratively.

RESULTS

Eight methods studies (10 publications) met the inclusion criteria; none addressed language restrictions in diagnostic test accuracy reviews. The included studies analyzed nine to 147 meta-analyses and/or systematic reviews. The proportions of non-English-language publications ranged from 2% to 100%. Based on five methods studies, restricting literature searches or inclusion criteria to English-language publications led to a change in statistical significance in 23/259 meta-analyses (9%). Most commonly, the statistical significance was lost, but had no impact on the conclusions of systematic reviews.

CONCLUSION

Restricting systematic reviews to English-language publications appears to have little impact on the effect estimates and conclusions of systematic reviews.

The impact of gender on researchers' assessment: A randomized controlled trial

OBJECTIVES

This randomized controlled trial aimed to test whether women or men would be preferred with identical curriculum vitae (CV); and the impact of the career stage in the evaluators' choice.

STUDY DESIGN AND SETTING

A simulated post-doctoral process was carried forward to be assessed for judgment. Level 1 and 2 Brazilian fellow researchers in the field of Dentistry were invited to act as external reviewers in a post-doctoral process and were randomly assigned to receive a CV from a woman or a man. They were required to rate the CV from 0 to 10 in scientific contribution, leadership potential, ability to work in groups, and international experience.

RESULTS

For all categories of CVs evaluated, CVs from men received higher scores compared to the CVs from women. Robust variance Poisson regressions demonstrated that men were more likely to receive higher scores in all categories, despite applicants' career stage. For example, CVs from men were nearly three quarters more likely to be seen as having leadership potential than equivalent CVs from women.

CONCLUSION

Gender bias is powerfully prevalent in academia in the dentistry field, despite researchers' career stage. Actions like implicit bias training must be urgently implemented to avoid (or at least decrease) that more women are harmed.

Intent to share Annals of Internal Medicine's trial data was not associated with data re-use

OBJECTIVE

To explore the impact of the Annals of Internal Medicine (AIM) data-sharing policy for randomized controlled trials (RCTs) in terms of output from data-sharing (i.e. publications re-using the data).

STUDY DESIGN AND SETTING

Retrospective study. RCTs published in the AIM between 2007 and 2017 were retrieved on PubMed. Publications where the data had been re-used were identified on Web of Science. Searches were performed by two independent reviewers. The primary outcome was any published re-use of the data (re-analysis, secondary analysis, or meta-analysis of individual participant data [MIPD]), where the first, last and corresponding authors were not among the authors of the RCT. Analyses used Cox (primary analysis) models adjusting for RCTs characteristics (registration: https://osf.io/8pj5e/).

RESULTS

185 RCTs were identified. 106 (57%) mentioned willingness to share data and 79 (43%) did not. 208 secondary analyses, 67 MIPD and no re-analyses were identified. No significant association was found between intent to share and re-use where the first, last and corresponding authors were not among the authors of the primary RCT (adjusted hazard ratio = 1.04 [0.47-2.30]).

CONCLUSION

Over ten years, RCTs published in AIM expressing an intention to share data were not associated with more extensive re-use of the data.

Meta-research in pharmacy: Time for a look in the mirror

The volume of the biomedical literature continues to expand at a substantial rate. The research literature surrounding pharmaceutical services is no different. Due in part to events in the recent past, researchers, consumers, funders, and policymakers have raised concerns about the credibility, transparency, and potential waste in the global research enterprise. Meta-research, or research on research, provides a way to examine the efficiency, quality, and potential bias in the overall research ecosystem. The field of meta-research is a relatively new but rapidly growing field that has seen many applications in biomedical research. Applications in pharmacy research, however, are still developing. The goals of this commentary are to introduce pharmacy researchers to the concept of meta-research, discuss several examples of meta-research in pharmacy, and motivate the importance of sustained meta-research efforts in pharmacy.

Analysis of clinical and methodological characteristics of early COVID-19 treatment clinical trials: so much work, so many lost opportunities

BACKGROUND

The COVID-19 pandemic continues to rage on, and clinical research has been promoted worldwide. We aimed to assess the clinical and methodological characteristics of treatment clinical trials that have been set forth as an early response to the COVID-19 pandemic.

METHODS

First, we reviewed all registered clinical trials on COVID-19. The World Health Organization International Trials Registry Platform and national trial registries were searched for COVID-19 trials through April 19th, 2020. For each record, independent researchers extracted interventions, participants, and methodological characteristics. Second, on September 14th, 2020 we evaluated the recruitment status and availability of the results of COVID-19 treatment trials previously identified.

RESULTS

In April 2020, a total of 580 trials evaluating COVID-19 treatment were registered. Reporting quality was poor (core participant information was missing in 24.1 to 92.7%). Between 54.0 and 93.8% of the trials did not plan to include older people or those with a higher baseline risk. Most studies were randomised (67.9%), single-centre (58.3%), non-industry-funded (81.1%), to be conducted in China (47.6%), with a median duration of 184 days and a median sample size of 100 participants. Core endpoints (mortality, clinical status, and hospitalization length) were planned to be assessed in 5.2 to 13.1% of the trials. Five months later, 66 trials (11.4%) were reported as "Completed", and only 46 (7.9%) had public results available. One hundred forty-four of 580 trials (24.8%) either had the status "Not yet recruiting" or "Suspended", and 18 (3.1%) trials were prematurely stopped ("Terminated" or "Withdrawn") The number of completed trials and trials with results are much lower than anticipated, considering the planned follow-up.

CONCLUSIONS

Our results raise concerns about the success of the initial global research effort on COVID-19 treatment. The clinical and methodological characteristics of early COVID-19 treatment trials limit their capability to produce clear answers to critical questions in the shortest possible time.

Changes in evidence for studies assessing interventions for COVID-19 reported in preprints: meta-research study

BACKGROUND

The increasing use of preprints to disseminate evidence on the effect of interventions for the coronavirus disease 2019 (COVID-19) can lead to multiple evidence sources for a single study, which may differ in the reported evidence. We aim to describe the proportion of evidence on the effect of interventions for COVID-19 from preprints and journal articles and map changes in evidence between and within different sources reporting on the same study.

METHODS

Meta-research study. We screened the Cochrane living systematic review and network meta-analysis (COVID-NMA) database to identify all preprints and journal articles on all studies assessing interventions for COVID-19 published up to 15 August 2020. We compared all evidence sources (i.e., preprint and associated journal article) and the first and latest versions of preprints for each study to identify changes in two evidence components: study results (e.g., numeric change in hazard ratio, odds ratio, event rate, or change in p value > or < 0.05 in any outcome) and abstract conclusions (classified as positive, negative or neutral regarding the intervention effect, and as reporting uncertainty in the findings or not). Changes in study results were further classified as important changes if they (1) represented a change in any effect estimate by ≥ 10% and/or (2) led to a change in the p value crossing the threshold of 0.05.

RESULTS

We identified 556 studies. In total, 338 (61%) had been reported in a preprint: 66 (20%) of these had an associated journal article (median time to publication 76 days [interquartile range (IQR) 55-106]) and 91 (27%) had > 1 preprint version. A total of 139 studies (25% of the overall sample) were reported in multiple evidence sources or versions of the same source: for 63 (45%), there was a change in at least one evidence component between or within sources (42 [30%] had a change in study results, and in 29 [21%] the change was classified as important; 33 [24%] had a change in the abstract conclusion). For studies with both a preprint and an article, a median of 29% (IQR 14-50) of total citations were attributed to the preprint instead of the article.

CONCLUSIONS

Results on the effect of interventions for COVID-19 are often reported in multiple evidence sources or source versions for a single study. Evidence is not stable between and within evidence sources. Real-time linkage of all sources per study could help to keep systematic reviews up-to-date.

Declaration of use and appropriate use of reporting guidelines in high-impact rehabilitation journals is limited: a meta-research study

OBJECTIVES

The main aims of this metaresearch study conducted among high-impact rehabilitation journals were: 1) to evaluate if the use of reporting guidelines (RGs) was declared and 2) to categorize the declared use as appropriate or inappropriate.

STUDY DESIGN AND SETTING

Cross-sectional analysis of a random sample of 200 studies published in the period 2010-2019 in five generic rehabilitation journals with the highest 5-year impact factor. Randomized controlled trials, systematic reviews, observational studies, and diagnostic studies were included. Prevalence with 95% confidence intervals (CIs) was estimated for the main outcomes.

RESULTS

Among the 200 selected studies, 17.5% (95% CI: 12.2-22.8%) declared using RGs. Among these studies, 48.6% (95% CI: 32-65.1%) declared an appropriate use. There was an increasing trend over time for authors to report the use of RGs (OR: 1.31; 95% CI: 1.13-1.53). Systematic reviews (n = 54) reported more frequently the use of RGs than other study designs (35.2%).

CONCLUSION

In high-impact rehabilitation journals, a small minority of article authors declared the use of RGs. In approximately half of these studies, RGs were used inappropriately. There is an urgent need to improve the use of RGs in this field of research.

Opening Pandora's Box: Peeking inside Psychology's data sharing practices, and seven recommendations for change

Open data-sharing is a valuable practice that ought to enhance the impact, reach, and transparency of a research project. While widely advocated by many researchers and mandated by some journals and funding agencies, little is known about detailed practices across psychological science. In a pre-registered study, we show that overall, few research papers directly link to available data in many, though not all, journals. Most importantly, even where open data can be identified, the majority of these lacked completeness and reusability—conclusions that closely mirror those reported outside of Psychology. Exploring the reasons behind these findings, we offer seven specific recommendations for engineering and incentivizing improved practices, so that the potential of open data can be better realized across psychology and social science more generally.

A randomized trial of an editorial intervention to reduce spin in the abstract's conclusion of manuscripts showed no significant effect

OBJECTIVE

To estimate the effect of an intervention compared to the usual peer-review process on reducing spin in the abstract's conclusion of biomedical study reports.

STUDY DESIGN AND SETTING

We conducted a two-arm, parallel-group RCT in a sample of primary research manuscripts submitted to BMJ Open. The authors received short instructions alongside the peer reviewers' comments in the intervention group. We assessed the presence of spin (primary outcome), types of spin, and wording change in the revised abstract's conclusion. Outcome assessors were blinded to the intervention assignment.

RESULTS

Of the 184 manuscripts randomized, 108 (54 intervention, 54 control) were selected for revision and could be evaluated for the presence of spin. The proportion of manuscripts with spin was 6% lower (95% CI: 24% lower to 13% higher) in the intervention group (57%, 31/54) than in the control group (63%, 34/54). The wording of the revised abstract's conclusion was changed in 34/54 (63%) manuscripts in the intervention group and 26/54 (48%) in the control group. The four prespecified types of spin involved (i) selective reporting (12 in the intervention group vs. 8 in the control group), (ii) including information not supported by evidence (9 vs. 9), and (iii) interpretation not consistent with the study results (14 vs. 18), and (iv) unjustified recommendations for practice (5 vs. 11).

CONCLUSION

These short instructions to authors did not have a statistically significant effect on reducing spin in revised abstract conclusions, and based on the confidence interval, the existence of a large effect can be excluded. Other interventions to reduce spin in reports of original research should be evaluated.

STUDY REGISTRATION

osf.io/xnuyt.

A Bayesian analysis of non-significant rehabilitation findings: Evaluating the evidence in favour of truly absent treatment effects

BACKGROUND

Relying solely on null hypothesis significance testing (NHST) to investigate rehabilitation interventions may result in researchers erroneously concluding the absence of a treatment effect.

OBJECTIVE

We aimed to distinguish between truly null treatment effects and data that are insensitive to detecting treatment effects by calculating Bayes factors (BF₀₁s) for non-significant findings in the rehabilitation literature. Additionally, to examine associations between BF₀₁, sample size, and observed P-values.

METHOD

We searched the Cochrane Database of Systematic Reviews for meta-analyses with "rehabilitation" as a keyword that clearly evaluated a rehabilitation intervention. We extracted means, standard deviations, and sample sizes for treatment and comparison groups for individual findings within 175 meta-analyses. Two independent investigators classified the interventions into 4 categories using the Rehabilitation Treatment Specification System. We calculated t-statistics and associated P-values for each finding in order to extract non-significant results (P>0.05). We calculated BF₀₁s for 5790 non-significant results and classified BF₀₁s based on the strength of evidence in favour of the null hypothesis (i.e., anecdotal, moderate, and strong) across and within intervention types. We examined correlations between BF₀₁, sample size, and P-values across and within intervention types.

RESULTS

Across all intervention types, most (71.9%) findings were deemed anecdotal, and this pattern remained within distinct intervention types (58.4-76.0%). Larger sample sizes tended to be associated with greater strength in favour of the null hypothesis, both across and within intervention types. Larger P-values were not associated with greater strength in favour of the null hypothesis; this finding was present both across and within intervention types.

CONCLUSION

Our findings indicate that most non-significant rehabilitation findings are unable to distinguish between the true absence of a treatment effect and data that are merely insensitive to detecting a treatment effect. Findings also suggest that rehabilitation researchers may improve the strength of their statistical conclusions by increasing sample size and that Bayes factors may offer unique benefits relative to P-values.

Best-worst scaling identified adequate statistical methods and literature search as the most important items of AMSTAR2 (A measurement tool to assess systematic reviews)

OBJECTIVE

To assess the relative importance of A MeaSurement Tool to Assess systematic Reviews 2 (AMSTAR2) items.

STUDY DESIGN AND SETTING

A best-worst scaling object case was conducted among a sample of experts in the field of systematic reviews (SRs) and meta-analyses (MAs). Respondents were asked in a series of 15 choice tasks to choose the most and the least important item from a set of four items from the master list, which included the 16 AMSTAR2 items. Hierarchical Bayes analysis was used to generate the relative importance score for each item.

RESULTS

The most important items highlighted by our 242 experts to conduct overview of reviews and critically assess SRs/MAs were the appropriateness of statistical analyses and adequacy of the literature search, followed by items regarding the assessment of risk of bias, the research protocol, and the assessment of heterogeneity (relative importance score >6.5). Items related to funding sources and the assessment of study selection and data extraction in duplicate were rated as least important.

CONCLUSION

Although all AMSTAR2 items can be considered as important, our results highlighted the importance of keeping the two items (the appropriateness of statistical analyses and the adequacy of the literature search) among the critical items proposed by AMSTAR2 to critically appraise SRs/MAs.

A Bayesian Analysis of Evidence in Support of the Null Hypothesis in Gerontological Psychology (or Lack Thereof)

OBJECTIVES

Nonsignificant p values derived from null hypothesis significance testing do not distinguish between true null effects or cases where the data are insensitive in distinguishing the hypotheses. This study aimed to investigate the prevalence of Bayesian analyses in gerontological psychology, a statistical technique that can distinguish between conclusive and inconclusive nonsignificant results, by using Bayes factors (BFs) to reanalyze nonsignificant results from published gerontological research.

METHODS

Nonsignificant results mentioned in abstracts of articles published in 2017 volumes of 10 top gerontological psychology journals were extracted (N = 409) and categorized based on whether Bayesian analyses were conducted. BFs were calculated from nonsignificant t-tests within this sample to determine how frequently the null hypothesis was strongly supported.

RESULTS

Nonsignificant results were directly tested with BFs in 1.22% of studies. Bayesian reanalyses of 195 nonsignificant t-tests found that only 7.69% of the findings provided strong evidence in support of the null hypothesis.

CONCLUSIONS

Bayesian analyses are rarely used in gerontological research, and a large proportion of null findings were deemed inconclusive when reanalyzed with BFs. Researchers are encouraged to use BFs to test the validity of nonsignificant results and ensure that sufficient sample sizes are used so that the meaningfulness of null findings can be evaluated.

What Research Institutions Can Do to Foster Research Integrity

In many countries attention for fostering research integrity started with a misconduct case that got a lot of media exposure. But there is an emerging consensus that questionable research practices are more harmful due to their high prevalence. QRPs have in common that they can help to make study results more exciting, more positive and more statistically significant. That makes them tempting to engage in. Research institutions have the duty to empower their research staff to steer away from QRPs and to explain how they realize that in a Research Integrity Promotion Plan. Avoiding perverse incentives in assessing researchers for career advancement is an important element in that plan. Research institutions, funding agencies and journals should make their research integrity policies as evidence-based as possible. The dilemmas and distractions researchers face are real and universal. We owe it to society to collaborate and to do our utmost best to prevent QRPs and to foster research integrity.

Using the Cochrane Central Register of Controlled Trials to identify clinical trial registration is insufficient: a cross-sectional study

BACKGROUND

While conducting systemic reviews, searching for ongoing or unpublished trials is critical to address publication bias. As of April 2019, records of ongoing or unpublished randomized and/or quasi-randomized controlled trials registered in the International Clinical Trials Registry Platform (ICTRP) and ClinicalTrials.gov are available in the Cochrane Central Register of Controlled Trials (CENTRAL). These records registered in CENTRAL include studies published since the inception of ICTRP and ClinicalTrials.gov . Whether systematic reviewers can search CENTRAL to identify ongoing or unpublished trials instead of ICTRP and ClinicalTrials.gov is unknown.

METHODS

This was a cross-sectional study. A consecutive sample of ongoing or unpublished studies published from June 1, 2019 to December 27, 2019 was selected from the Cochrane Reviews. The sensitivity and the number needed to read (NNR) were assessed from among the studies selected from CENTRAL instead of ICTRP and ClinicalTrials.gov and also assessed the characteristics of studies not identified by searching CENTRAL.

RESULTS

In total, 247 records from 50 Cochrane reviews were included; of these, 200 were identified by searching CENTRAL, whereas the remaining 47 records were not. The sensitivity of searching CENTRAL was 0.81 (95% confidence interval [CI]: 0.76, 0.85). The NNR was 115 (95% CI: 101, 133). The 47 unidentified studies were registered through ClinicalTrials.gov or ICTRP. Sixteen unidentified studies were not indexed in CENTRAL.

CONCLUSIONS

For systematic reviewers, searching CENTRAL could not substitute for searching ClinicalTrials.gov and/or ICTRP. Systematic reviewers should not only search CENTRAL but also ICTRP and ClinicalTrials.gov to identify unpublished trials.

TRIAL REGISTRATION

A pre-specified protocol was applied to conduct this study. The study was registered in the University Hospital Medical Information Network Clinical Trials Registry (UMIN-CTR).

TRIAL REGISTRATION NUMBER

UMIN000038981 .

Systematic review supports the role of DNA methylation in the pathophysiology of preeclampsia: a call for analytical and methodological standardization

BACKGROUND

Studies have recently examined the role of epigenetic mechanisms in preeclampsia pathophysiology. One commonly examined epigenetic process is DNA methylation. This heritable epigenetic marker is involved in many important cellular functions. The aim of this study was to establish the association between DNA methylation and preeclampsia and to critically appraise the roles of major study characteristics that can significantly impact the association between DNA methylation and preeclampsia.

MAIN BODY

A systematic review was performed by searching PubMed, Web of Science, and EMBASE for original research articles published over time, until May 31, 2019 in English. Eligible studies compared DNA methylation levels in pregnant women with vs. without preeclampsia. Ninety articles were included. Epigenome-wide studies identified hundreds of differentially methylated places/regions in preeclamptic patients. Hypomethylation was the predominant finding in studies analyzing placental tissue (14/19), while hypermethylation was detected in three studies that analyzed maternal white blood cells (3/3). In candidate gene studies, methylation alterations for a number of genes were found to be associated with preeclampsia. A greater number of differentially methylated genes was found when analyzing more severe preeclampsia (70/82), compared to studies analyzing less severe preeclampsia vs. controls (13/27). A high degree of heterogeneity existed among the studies in terms of methodological study characteristics including design (study design, definition of preeclampsia, control group, sample size, confounders), implementation (biological sample, DNA methylation method, purification of DNA extraction, and validation of methylation), analysis (analytical method, batch effect, genotyping, and gene expression), and data presentation (methylation quantification measure, measure of variability, reporting). Based on the results of this review, we provide recommendations for study design and analytical approach for further studies.

CONCLUSIONS

The findings from this review support the role of DNA methylation in the pathophysiology of preeclampsia. Establishing field-wide methodological and analytical standards may increase value and reduce waste, allowing researchers to gain additional insights into the role of DNA methylation in the pathophysiology of preeclampsia.

Individual participant data meta-analyses (IPDMA): data contribution was associated with trial corresponding author country, publication year, and journal impact factor

OBJECTIVES

The objectives were to determine the proportion of eligible randomized controlled trials (RCTs) that contributed data to individual participant data meta-analyses (IPDMAs) and explore associated factors.

STUDY DESIGN AND SETTING

IPDMAs with ≥10 eligible RCTs were identified by searching MEDLINE, EMBASE, CINAHL, and Cochrane May 1, 2015 to February 13, 2017. Mixed-effect logistic regression was used to identify factors associated with data contribution.

RESULTS

Of 774 eligible RCTs from 35 included IPDMAs, 517 (67%, 95% confidence interval [CI]: 63%-70%) contributed data. Compared to RCTs from journals with low-impact factors (0-2.4), RCTs from journals with higher impact factors were more likely to contribute data: impact factor 5.0-9.9, odds ratio [OR] 2.6, 95% CI: 1.37-4.86; impact factor: 10.0-19.9, OR: 5.7, 95% CI: 3.0-10.8; impact factor >20.0, OR: 4.6, 95% CI: 1.9-11.4. RCTs from the United Kingdom were more likely to contribute data than those from the United States (reference; OR: 2.4, 95% CI, 1.3-4.6). There was an increase in OR per publication year (OR: 1.05, 95% CI: 1.02-1.09).

CONCLUSION

The country where RCTs are conducted, impact factor of the journal where RCTs are published, and RCT publication year were associated with data contribution in IPDMAs with ≥10 eligible RCTs.

Using a rapid environmental scan methodology to map country-level global health research expertise in Canada

Background

Many countries are currently rethinking their global health research funding priorities. When resources are limited, it is important to understand and use information about existing research strengths to inform research strategies and investments and to drive impact. This study describes a method to rapidly assess a country’s global health research expertise and applies this method in the Canadian context.

Methods

We developed a three-pronged rapid environmental scan to evaluate Canadian global health research expertise that focused on research funding inputs, research activities and research outputs. We assessed research funding inputs from Canada’s national health research funding agency and identified the 30 Canadian universities that received the most global health research funding. We systematically searched university websites and secondary databases to identify research activities, including research centres, research chairs and research training programmes. To evaluate research outputs, we searched PubMed to identify global health research publications by Canadian university-affiliated researchers. We used these three perspectives to develop a more nuanced understanding of Canadian strengths in global health research from different perspectives.

Results

Canada’s main global health research funder, the Canadian Institutes of Health Research, invested a total of $314 M from 2000 to 2016 on global health research grants. This investment has contributed to Canada’s wealth of global health research expertise, including 12 training programmes, 27 Canada Research Chairs, 6 research centres and 30 WHO Collaborating Centres across 27 universities. Research activities were concentrated in Canada’s biggest cities and most commonly focused on health equity and globalisation issues. Canadian-affiliated researchers have contributed to a research output of 822 unique publications on PubMed. There is an opportunity to build global health expertise in regions not already concentrated with research activity, focusing on transnational risks and neglected conditions research.

Conclusions

Our three-pronged approach allowed us to rapidly identify clear geographic and substantive areas of strength in Canadian global health research, including urban regions and research focused on health equity and globalisation topics. This information can be used to support research policy directives, including to inform a Canadian global health research strategy, and to allow relevant academic institutions and funding organisations to make more strategic decisions regarding their future investments.

Glucocorticoid-trials in rheumatoid arthritis mostly study representative real-world patients: A systematic review and meta-analysis

OBJECTIVE

Randomized controlled trials (RCTs) are considered the gold standard in clinical research due to credible causality. Their results, however, may not be generalizable to real-world populations. While glucocorticoids (GCs) remain a mainstay of rheumatoid arthritis (RA) treatment, it is unclear whether the results of GC-RCTs are generalizable to current real-world RA patients.

METHODS

MEDLINE was searched for RCTs and, as comparators, cohort studies (CSs) in RA evaluating systemic GCs. Random-effects meta-analyses were performed for descriptive baseline characteristics (including general demographics, comorbidities, and disease activity) that have been shown to be able to modify the benefit-risk-ratio of various RA therapeutics. These meta-analyses were stratified by study type (RCT and CS). Stratified estimates were subsequently compared. Further sensitivity analyses were performed stratifying by disease duration.

RESULTS

56 RCTs (7053 participants) and 10 CSs (14,688 participants) were included. 12 characteristics were reported frequently enough to allow for comparative analysis. In 10/12 characteristics (83%), RCT estimates did not appear to differ from CS estimates. However, RCT participants were younger (-4.7 years [95% CI -7.2 to -2.1]; p < 0.001) and had higher erythrocyte sedimentation rates (11.8 mm/h [5.7 to 17.8]; p < 0.001) than CS participants. Comorbidities could not be assessed due to insufficient reporting.

CONCLUSION

Our findings suggest that evidence from GC trials in RA is of acceptable generalizability to current real-world patients - especially compared to findings from biologic agents in RA. However, RCT participants were younger than real-world patients, potentially limiting the generalizability of trial results to elderly patients.

SYSTEMATIC REVIEW REGISTRATION

PROSPERO (CRD42019134675).

No evidence found for an association between trial characteristics and treatment effects in randomized trials of testosterone therapy in men: a meta-epidemiological study

OBJECTIVE

The objective of this study was to identify potential trial characteristics associated with reported treatment effect estimates in randomized trials of testosterone therapy in adult men.

STUDY DESIGN AND SETTING

This is a meta-epidemiological study. MEDLINE was searched for meta-analyses of randomized trials of testosterone therapy in men published between 2008 and 2018. Data on trial characteristics were extracted independently by two reviewers. The impact of trial characteristics on reported treatment effects was investigated using a two-step meta-analytic approach.

RESULTS

We identified 132 randomized trials, included in 19 meta-analyses, comprising data from 10,725 participants. None of the investigated design characteristics, including year of publication, sample size, trial registration status, center status, regionality, funding source, and conflict of interest were statistically significantly associated with reported treatment effects of testosterone therapy in men. Although trials rated at high risk of bias overall reported treatment effects that were 21% larger compared with trials rated at low risk of bias overall, the 95% confidence interval included the null (ratio of odds ratio: 0.79, 95% confidence interval: 0.60 to 1.03).

CONCLUSION

The present study found no clear evidence that trial characteristics are associated with treatment effects in randomized trials of testosterone therapy in men. To establish stronger evidence about the treatment effects of testosterone therapy in men, future randomized trials should not only be adequately designed but also transparently reported.

STUDY REGISTRATION

osf.io/x9g6m.