Determinants of selective reporting: A taxonomy based on content analysis of a random selection of the literature

Indicators of questionable research practices were identified in 163,129 randomized controlled trials

OBJECTIVES

To explore indicators of the following questionable research practices (QRPs) in randomized controlled trials (RCTs): (1) risk of bias in four domains (random sequence generation, allocation concealment, blinding of participants and personnel, and blinding of outcome assessment); (2) modifications in primary outcomes that were registered in trial registration records (proxy for selective reporting bias); (3) ratio of the achieved to planned sample sizes; and (4) statistical discrepancy.

STUDY DESIGN AND SETTING

Full texts of all human RCTs published in PubMed in 1996-2017 were automatically identified and information was collected automatically. Potential indicators of QRPs included author-specific, publication-specific, and journal-specific characteristics. Beta, logistic, and linear regression models were used to identify associations between these potential indicators and QRPs.

RESULTS

We included 163,129 RCT publications. The median probability of bias assessed using Robot Reviewer software ranged between 43% and 63% for the four risk of bias domains. A more recent publication year, trial registration, mentioning of CONsolidated Standards Of Reporting Trials-checklist, and a higher journal impact factor were consistently associated with a lower risk of QRPs.

CONCLUSION

This comprehensive analysis provides an insight into indicators of QRPs. Researchers should be aware that certain characteristics of the author team and publication are associated with a higher risk of QRPs.

Measurement challenges and causes of incomplete results reporting of biomedical animal studies: Results from an interview study

Background

Existing evidence indicates that a significant amount of biomedical research involving animals remains unpublished. At the same time, we lack standards for measuring the extent of results reporting in animal research. Publication rates may vary significantly depending on the level of measurement such as an entire animal study, individual experiments within a study, or the number of animals used.

Methods

Drawing on semi-structured interviews with 18 experts and qualitative content analysis, we investigated challenges and opportunities for the measurement of incomplete reporting of biomedical animal research with specific reference to the German situation. We further investigate causes of incomplete reporting.

Results

The in-depth expert interviews revealed several reasons for why incomplete reporting in animal research is difficult to measure at all levels under the current circumstances. While precise quantification based on regulatory approval documentation is feasible at the level of entire studies, measuring incomplete reporting at the more individual experiment and animal levels presents formidable challenges. Expert-interviews further identified six drivers of incomplete reporting of results in animal research. Four of these are well documented in other fields of research: a lack of incentives to report non-positive results, pressures to ‘deliver’ positive results, perceptions that some data do not add value, and commercial pressures. The fifth driver, reputational concerns, appears to be far more salient in animal research than in human clinical trials. The final driver, socio-political pressures, may be unique to the field.

Discussion

Stakeholders in animal research should collaborate to develop a clear conceptualisation of complete reporting in animal research, facilitate valid measurements of the phenomenon, and develop incentives and rewards to overcome the causes for incomplete reporting.

Differences Between the 2016 and 2022 Editions of the Enhanced Recovery After Bariatric Surgery (ERABS) Guidelines: Call to Action of FAIR Data and the Creation of a Global Consortium of Bariatric Care and Research

In 2016, the Enhanced Recovery After Bariatric Surgery guidelines (G16) was published, and in 2022, an update to it was released (G22). Grading of recommendations, assessment, development, and evaluations (GRADE), emphasizing the level of evidence (LoE) of both the guidelines, was performed. An overview of methodology was also performed, considering the following questions: how can research be improved, what can be done in the future using data, and how to collaborate more? Both guidelines did not explain how the LoE conclusions were derived regarding the risk of bias. There is also potential for forming a global consortium that deals with bariatric research, which can serve as a repository for all relevant data. Ensuring that this data is FAIR (findability, accessibility, interoperability, reusability) compliant and using this data to formulate future guidelines will benefit clinicians and patients alike.

Trace amine-associated receptor 1 (TAAR1): Potential application in mood disorders: A systematic review

There is a need for innovation with respect to therapeutics in psychiatry. Available evidence indicates that the trace amine-associated receptor 1 (TAAR1) agonist SEP-363856 is promising, as it improves measures of cognitive and reward function in schizophrenia. Hedonic and cognitive impairments are transdiagnostic and constitute major burdens in mood disorders. Herein, we systematically review the behavioural and genetic literature documenting the role of TAAR1 in reward and cognitive function, and propose a mechanistic model of TAAR1's functions in the brain. Notably, TAAR1 activity confers antidepressant-like effects, enhances attention and response inhibition, and reduces compulsive reward seeking without impairing normal function. Further characterization of the responsible mechanisms suggests ion-homeostatic, metabolic, neurotrophic, and anti-inflammatory enhancements in the limbic system. Multiple lines of evidence establish the viability of TAAR1 as a biological target for the treatment of mood disorders. Furthermore, the evidence suggests a role for TAAR1 in reward and cognitive function, which is attributed to a cascade of events that are relevant to the cellular integrity and function of the central nervous system.

Study Registration for the Field of Prevention Science: Considering Options and Paths Forward

The practice of prospectively registering the details of intervention studies in a public database or registry is gaining momentum across disciplines as a strategy for increasing the transparency, credibility, and accessibility of study findings. In this article, we consider five registries that may be relevant for registration of intervention studies in the field of prevention science: ClinicalTrials.gov, the American Economic Association Registry of Randomized Controlled Trials (AEA RCT Registry), the Open Science Framework Preregistration (OSF Preregistration), the Registry for International Development Impact Evaluations (RIDIE), and the Registry of Efficacy and Effectiveness Studies (REES). We examine the five registries in terms of substantive focus, study designs, and contents of registry entries. We consider two paths forward for prospective registration of intervention studies in the field of prevention science: Path A: register all studies in ClinicalTrials.gov and Path B: allow individual researchers to select the registry with the "best fit." Lastly, we consider how the field might begin to establish norms around registration.

An exploratory randomised controlled trial evaluating text prompts in Lebanon to encourage health-seeking behaviour for hypertension

AIMS OF THE STUDY

The current study evaluates the effectiveness of an opportunistic mobile screening on the percentage of people who are aware of whether they may be hypertensive (in an observational study) and the effectiveness of reminder prompts on the percentage of people who seek further medical attention (in a randomised controlled trial).

METHODS USED TO CONDUCT THE STUDY

The screening of 1227 participants (529 female) was conducted during the registration period of the 2018 Beirut International Marathon in Lebanon. Next, 266 participants whose screening indicated hypertension (64 Female) were randomly allocated to a treatment group or a control group in a 1:1 fashion. The treatment group received a reminder prompt to seek further medical attention for their potential hypertension and the control group did not. The overt nature of the text message meant that participants in the treatment group could not be blinded to their group allocation. The primary outcome is participants' self-reports of whether they sought further medical attention.

RESULTS OF THE STUDY

For the opportunistic screening, a 25% prevalence rate and a 24% awareness rate of hypertension was indicated. A McNemar analysis suggested that the screening increased participant awareness (X² (N = 1227) = 72.16, P < .001). For the randomised controlled trial, 219 participants provided follow-up data via a phone call (82% retention). A Chi-squared analysis suggested that the reminder prompt successfully encouraged more participants to seek further medical attention, 45.5% treatment group vs 28.0% control group (X² (1, N = 219) = 7.19, P = .007, φ = 0.18).

CONCLUSIONS DRAWN AND CLINICAL IMPLICATIONS

Extra support in the form of a brief reminder message can increase the percentage of people who seek further medical attention after attending an opportunistic screening at a marathon event. The discussion reviews how the results align with previous research, strengths and limitations of the current study, and implications for future research and practice.

Ensuring the quality and specificity of preregistrations

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of "researcher degrees of freedom" aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called "OSF Preregistration," http://osf.io/prereg/). The Prereg Challenge format was a "structured" workflow with detailed instructions and an independent review to confirm completeness; the "Standard" format was "unstructured" with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the "structured" format restricted the opportunistic use of researcher degrees of freedom better (Cliff's Delta = 0.49) than the "unstructured" format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.

fiddle: a tool to combat publication bias by getting research out of the file drawer and into the scientific community

Statistically significant findings are more likely to be published than non-significant or null findings, leaving scientists and healthcare personnel to make decisions based on distorted scientific evidence. Continuously expanding ´file drawers' of unpublished data from well-designed experiments waste resources creates problems for researchers, the scientific community and the public. There is limited awareness of the negative impact that publication bias and selective reporting have on the scientific literature. Alternative publication formats have recently been introduced that make it easier to publish research that is difficult to publish in traditional peer reviewed journals. These include micropublications, data repositories, data journals, preprints, publishing platforms, and journals focusing on null or neutral results. While these alternative formats have the potential to reduce publication bias, many scientists are unaware that these formats exist and don't know how to use them. Our open source file drawer data liberation effort (fiddle) tool (RRID:SCR_017327 available at: http://s-quest.bihealth.org/fiddle/) is a match-making Shiny app designed to help biomedical researchers to identify the most appropriate publication format for their data. Users can search for a publication format that meets their needs, compare and contrast different publication formats, and find links to publishing platforms. This tool will assist scientists in getting otherwise inaccessible, hidden data out of the file drawer into the scientific community and literature. We briefly highlight essential details that should be included to ensure reporting quality, which will allow others to use and benefit from research published in these new formats.

Publication rates and reported results in a cohort of gene- and cell-based therapy trials

Aim: We investigated publication rates and reported results for gene- and cell-based therapy trials. Materials & methods: In a cohort of Institutional Review Board (IRB)-authorized trials during 2007-2017 in the Netherlands (n = 105), we examine publication rates and reported results in scientific papers and conference abstracts as well as associations with the occurrence of trial characteristics. Results: The publication rate for scientific papers was 27% and 17% for conference abstracts (median survival time: 1050 days). Academic hospitals published more in scientific papers whereas private sponsors published more in conference abstracts. Manufacturing protocols were underreported compared with clinical outcomes. Most publications reported positive results (78%). Conclusion: Publication rates are currently suboptimal indicating a need for enhanced knowledge sharing to stimulate gene- and cell-based therapy development.

Impact of SARS-CoV-2 Infection on Cognitive Function: A Systematic Review

The prevalence and etiology of COVID-19's impact on brain health and cognitive function is poorly characterized. With mounting reports of delirium, systemic inflammation, and evidence of neurotropism, a statement on cognitive impairment among COVID-19 cases is needed. A substantial literature has demonstrated that inflammation can severely disrupt brain function, suggesting an immune response, a cytokine storm, as a possible cause of neurocognitive impairments. In this light, the aim of the present study was to summarize the available knowledge of the impact of COVID-19 on cognition (i.e., herein, we broadly define cognition reflecting the reporting on this topic in the literature) during the acute and recovery phases of the disease, in hospitalized patients and outpatients with confirmed COVID-19 status. A systematic review of the literature identified six studies which document the prevalence of cognitive impairment, and one which quantifies deficits after recovery. Pooling the samples of the included studies (total sample n = 644) at three standards of quality produced conservative estimates of cognitive impairment ranging from 43.0 to 66.8% prevalence in hospitalized COVID-19 patients only, as no studies which report on outpatients met criteria for inclusion in the main synthesis. The most common impairment reported was delirium and frequent reports of elevated inflammatory markers suggest etiology. Other studies have demonstrated that the disease involves marked increases in IL-6, TNFα, and IL-1β; cytokines known to have a profound impact on working memory and attention. Impairment of these cognitive functions is a characteristic aspect of delirium, which suggests these cytokines as key mediators in the etiology of COVID-19 induced cognitive impairments. Researchers are encouraged to assay inflammatory markers to determine the potential role of inflammation in mediating the disturbance of cognitive function in individuals affected by COVID-19.

Causes of reporting bias: a theoretical framework

Reporting of research findings is often selective. This threatens the validity of the published body of knowledge if the decision to report depends on the nature of the results. The evidence derived from studies on causes and mechanisms underlying selective reporting may help to avoid or reduce reporting bias. Such research should be guided by a theoretical framework of possible causal pathways that lead to reporting bias. We build upon a classification of determinants of selective reporting that we recently developed in a systematic review of the topic. The resulting theoretical framework features four clusters of causes. There are two clusters of necessary causes: (A) motivations (e.g. a preference for particular findings) and (B) means (e.g. a flexible study design). These two combined represent a sufficient cause for reporting bias to occur. The framework also features two clusters of component causes: (C) conflicts and balancing of interests referring to the individual or the team, and (D) pressures from science and society. The component causes may modify the effect of the necessary causes or may lead to reporting bias mediated through the necessary causes. Our theoretical framework is meant to inspire further research and to create awareness among researchers and end-users of research about reporting bias and its causes.

Reporting and analysis of repeated measurements in preclinical animals experiments

A common feature of preclinical animal experiments is repeated measurement of the outcome, e.g., body weight measured in mice pups weekly for 20 weeks. Separate time point analysis or repeated measures analysis approaches can be used to analyze such data. Each approach requires assumptions about the underlying data and violations of these assumptions have implications for estimation of precision, and type I and type II error rates. Given the ethical responsibilities to maximize valid results obtained from animals used in research, our objective was to evaluate approaches to reporting repeated measures design used by investigators and to assess how assumptions about variation in the outcome over time impact type I and II error rates and precision of estimates. We assessed the reporting of repeated measures designs of 58 studies in preclinical animal experiments. We used simulation modelling to evaluate three approaches to statistical analysis of repeated measurement data. In particular, we assessed the impact of (a) repeated measure analysis assuming that the outcome had non-constant variation at all time points (heterogeneous variance) (b) repeated measure analysis assuming constant variation in the outcome (homogeneous variance), (c) separate ANOVA at individual time point in repeated measures designs. The evaluation of the three model fitting was based on comparing the p-values distributions, the type I and type II error rates and by implication, the shrinkage or inflation of standard error estimates from 1000 simulated dataset. Of 58 studies with repeated measures design, three provided a rationale for repeated measurement and 23 studies reported using a repeated-measures analysis approach. Of the 35 studies that did not use repeated-measures analysis, fourteen studies used only two time points to calculate weight change which potentially means collected data was not fully utilized. Other studies reported only select time points (n = 12) raising the issue of selective reporting. Simulation studies showed that an incorrect assumption about the variance structure resulted in modified error rates and precision estimates. The reporting of the validity of assumptions for repeated measurement data is very poor. The homogeneous variation assumption, which is often invalid for body weight measurements, should be confirmed prior to conducting the repeated-measures analysis using homogeneous covariance structure and adjusting the analysis using corrections or model specifications if this is not met.

The ENCePP Code of Conduct: A best practise for scientific independence and transparency in noninterventional postauthorisation studies

PURPOSE

The ENCePP Code of Conduct provides a framework for scientifically independent and transparent pharmacoepidemiological research. Despite becoming a landmark reference, practical implementation of key provisions was still limited. The fourth revision defines scientific independence and clarifies uncertainties on the applicability to postauthorisation safety studies requested by regulators. To separate the influence of the funder from the investigator's scientific responsibility, the Code now requires that the lead investigator is not employed by the funding institution.

METHOD

To assess how the revised Code fits the ecosystem of noninterventional pharmacoepidemiology research in Europe, we first mapped key recommendations of the revised Code against ISPE Good Pharmacoepidemiology Practices and the ADVANCE Code of Conduct. We surveyed stakeholders to understand perceptions on its value and practical applicability. Representatives from the different stakeholders' groups described their experience and expectations.

RESULTS

Unmet needs in pharmacoepidemiological research are fulfilled by providing unique guidance on roles and responsibilities to support scientific independence. The principles of scientific independence and transparency are well understood and reinforce trust in study results; however, around 70% of survey respondents still found some provisions difficult to apply. Representatives from stakeholders' groups found the new version promising, although limitations still exist.

CONCLUSION

By clarifying definitions and roles, the latest revision of the Code sets a new standard in the relationship between investigators and funders to support scientific independence of pharmacoepidemiological research. Disseminating and training on the provisions of the Code would help stakeholders to better understand its advantages and promote its adoption in noninterventional research.