The Weak Spots in Contemporary Science (and How to Fix Them)

Footprint of publication selection bias on meta-analyses in medicine, environmental sciences, psychology, and economics

Publication selection bias undermines the systematic accumulation of evidence. To assess the extent of this problem, we survey over 68,000 meta-analyses containing over 700,000 effect size estimates from medicine (67,386/597,699), environmental sciences (199/12,707), psychology (605/23,563), and economics (327/91,421). Our results indicate that meta-analyses in economics are the most severely contaminated by publication selection bias, closely followed by meta-analyses in environmental sciences and psychology, whereas meta-analyses in medicine are contaminated the least. After adjusting for publication selection bias, the median probability of the presence of an effect decreased from 99.9% to 29.7% in economics, from 98.9% to 55.7% in psychology, from 99.8% to 70.7% in environmental sciences, and from 38.0% to 29.7% in medicine. The median absolute effect sizes (in terms of standardized mean differences) decreased from d = 0.20 to d = 0.07 in economics, from d = 0.37 to d = 0.26 in psychology, from d = 0.62 to d = 0.43 in environmental sciences, and from d = 0.24 to d = 0.13 in medicine.

Preregistration in practice: A comparison of preregistered and non-preregistered studies in psychology

Preregistration has gained traction as one of the most promising solutions to improve the replicability of scientific effects. In this project, we compared 193 psychology studies that earned a Preregistration Challenge prize or preregistration badge to 193 related studies that were not preregistered. In contrast to our theoretical expectations and prior research, we did not find that preregistered studies had a lower proportion of positive results (Hypothesis 1), smaller effect sizes (Hypothesis 2), or fewer statistical errors (Hypothesis 3) than non-preregistered studies. Supporting our Hypotheses 4 and 5, we found that preregistered studies more often contained power analyses and typically had larger sample sizes than non-preregistered studies. Finally, concerns about the publishability and impact of preregistered studies seem unwarranted, as preregistered studies did not take longer to publish and scored better on several impact measures. Overall, our data indicate that preregistration has beneficial effects in the realm of statistical power and impact, but we did not find robust evidence that preregistration prevents p-hacking and HARKing (Hypothesizing After the Results are Known).

Artificial Intelligence in Scientific Writing: A Deuteragonistic Role?

In this article, we reflect on the pros and cons of artificial intelligence (AI)-augmented scientific writing for more comprehensible research towards society to gain trust for science-led policy. For this purpose, we integrated our thoughts into the Factors of Perceived Trustworthiness from Mayer, Davis, and Schoorman's Model of Trust and made propositions to define AI's role in trustful scholarly communication.

How do psychology researchers interpret the results of multiple replication studies?

Employing two vignette studies, we examined how psychology researchers interpret the results of a set of four experiments that all test a given theory. In both studies, we found that participants' belief in the theory increased with the number of statistically significant results, and that the result of a direct replication had a stronger effect on belief in the theory than the result of a conceptual replication. In Study 2, we additionally found that participants' belief in the theory was lower when they assumed the presence of p-hacking, but that belief in the theory did not differ between preregistered and non-preregistered replication studies. In analyses of individual participant data from both studies, we examined the heuristics academics use to interpret the results of four experiments. Only a small proportion (Study 1: 1.6%; Study 2: 2.2%) of participants used the normative method of Bayesian inference, whereas many of the participants' responses were in line with generally dismissed and problematic vote-counting approaches. Our studies demonstrate that many psychology researchers overestimate the evidence in favor of a theory if one or more results from a set of replication studies are statistically significant, highlighting the need for better statistical education.

Robust Bayesian meta-analysis: Model-averaging across complementary publication bias adjustment methods

Publication bias is a ubiquitous threat to the validity of meta-analysis and the accumulation of scientific evidence. In order to estimate and counteract the impact of publication bias, multiple methods have been developed; however, recent simulation studies have shown the methods' performance to depend on the true data generating process, and no method consistently outperforms the others across a wide range of conditions. Unfortunately, when different methods lead to contradicting conclusions, researchers can choose those methods that lead to a desired outcome. To avoid the condition-dependent, all-or-none choice between competing methods and conflicting results, we extend robust Bayesian meta-analysis and model-average across two prominent approaches of adjusting for publication bias: (1) selection models of p-values and (2) models adjusting for small-study effects. The resulting model ensemble weights the estimates and the evidence for the absence/presence of the effect from the competing approaches with the support they receive from the data. Applications, simulations, and comparisons to preregistered, multi-lab replications demonstrate the benefits of Bayesian model-averaging of complementary publication bias adjustment methods.

Using selection models to assess sensitivity to publication bias: A tutorial and call for more routine use

In meta-analyses, it is critical to assess the extent to which publication bias might have compromised the results. Classical methods based on the funnel plot, including Egger's test and Trim-and-Fill, have become the de facto default methods to do so, with a large majority of recent meta-analyses in top medical journals (85%) assessing for publication bias exclusively using these methods. However, these classical funnel plot methods have important limitations when used as the sole means of assessing publication bias: they essentially assume that the publication process favors large point estimates for small studies and does not affect the largest studies, and they can perform poorly when effects are heterogeneous. In light of these limitations, we recommend that meta-analyses routinely apply other publication bias methods in addition to or instead of classical funnel plot methods. To this end, we describe how to use and interpret selection models. These methods make the often more realistic assumption that publication bias favors "statistically significant" results, and the methods also directly accommodate effect heterogeneity. Selection models have been established for decades in the statistics literature and are supported by user-friendly software, yet remain rarely reported in many disciplines. We use a previously published meta-analysis to demonstrate that selection models can yield insights that extend beyond those provided by funnel plot methods, suggesting the importance of establishing more comprehensive reporting practices for publication bias assessment.

Numbers of publications and citations for researchers in fields pertinent to the social services: a comparison of peer-reviewed journal publications across six disciplines

Timely dissemination of knowledge is essential and fosters development of knowledge both within academe and the wider society, not least for knowledge that serves practises outside of academia. Here, we compare six disciplines which provide health-related knowledge that serve the health and social services. Most previous research compares the size and impact of the body of publications belonging to each discipline, which ignores the distribution of seniority, productivity, and impact amongst researchers. Instead, we consider the whole population of academics in Sweden employed or active within each discipline, including those who have nil publications. The disciplines form three clusters, where researchers in Public Health and Nursing and Caring science claim about 15 articles per author, Psychology about 10, and Education, Sociology and Social Work less than four. Their numbers of citations follow the same pattern, and are substantially correlated with the number of articles. Tenured or full professors had about 50% more publications and citations per publication than had associate professors. The distributions indicate clear modes at 0, 4, and 16 publications for each cluster, and provide the proportions of researchers within each discipline who have no such publications at all. We discuss the implications of these results for policy, practice, and knowledge quality in the social services and the welfare sector.

A qualitative analysis of stakeholder experiences with Registered Reports Funding Partnerships

Background: Registered Reports (RRs) could be a way to increase the quality of scientific research and literature, such as by reducing publication bias and increasing the rigour of study designs. These potential benefits have led to Registered Report funding partnerships (RRFPs or partnerships for short) between research funders and academic journals who collaborate to encourage researchers to publish RRs. In this study we investigated the research question: "What are the experiences of the stakeholders (authors, reviewers, journal editors, funders) in the various partnership models?". Our companion paper addresses a related, but separate, research question. Methods: We conducted a thematic analysis of 32 semi-structured interviews with stakeholders (funders, editors, authors, reviewers, matchmakers) from six partnerships. Results: Interviewees had highly variable perceptions and experiences, reflecting the complex and nuanced impacts of partnerships. We identified 6 themes: "Importance of communication with authors and reviewers", "Influence on study design", "Appropriateness of partners", "Potential to reduce publication bias", "Impact on reviewer workload", and "Insufficient evidence". Conclusions: This was the first investigation into these novel initiatives. We hope that our findings can benefit and shape current and future partnerships.

Evaluation of a Meta-Analysis of Ambient Air Quality as a Risk Factor for Asthma Exacerbation

Background: An irreproducibility crisis currently afflicts a wide range of scientific disciplines, including public health and biomedical science. A study was undertaken to assess the reliability of a meta-analysis examining whether air quality components (carbon monoxide, particulate matter 10 µm and 2.5 µm (PM10 and PM2.5), sulfur dioxide, nitrogen dioxide and ozone) are risk factors for asthma exacerbation. Methods: The number of statistical tests and models were counted in 17 randomly selected base papers from 87 used in the meta-analysis. Confidence intervals from all 87 base papers were converted to p-values. p-value plots for each air component were constructed to evaluate the effect heterogeneity of the p-values. Results: The number of statistical tests possible in the 17 selected base papers was large, median = 15,360 (interquartile range = 1536–40,960), in comparison to results presented. Each p-value plot showed a two-component mixture with small p-values < 0.001 while other p-values appeared random (p-values > 0.05). Given potentially large numbers of statistical tests conducted in the 17 selected base papers, p-hacking cannot be ruled out as explanations for small p-values. Conclusions: Our interpretation of the meta-analysis is that random p-values indicating null associations are more plausible and the meta-analysis is unlikely to replicate in the absence of bias.

Towards a Research Agenda for Promoting Responsible Research Practices

This opinion piece aims to inform future research funding programs on responsible research practices (RRP) based on three specific objectives: (1) to give a sketch of the current international discussion on responsible research practices (RRPs); (2) to give an overview of current initiatives and already obtained results regarding RRP; and (3) to give an overview of potential future needs for research on RRP. In this opinion piece, we have used seven iterative methodological steps (including literature review, ranking, and sorting exercises) to create the proposed research agenda. We identified six main themes that we believe need attention in future research: (1) responsible evaluation of research and researchers, (2) the influence of open science and transparency on RRP, (3) research on responsible mentoring, supervision, and role modeling, (4) the effect of education and training on RRP, (5) checking for reproducibility, and (6) responsible and fair peer review. These themes have in common that they address aspects of research that are mostly on the level of the scientific system, more than on the level of the individual researcher. Some current initiatives are already gathering substantial empirical evidence to start filling these gaps. We believe that with sufficient support from all relevant stakeholders, more progress can be made.

Meta-research publications in dentistry: a review

The present scoping review has the objective of providing an overview of meta-research in dentistry. A search of the PubMed database was performed for the period 11 October 2014 to 10 October 2019. Study selection and data extraction were performed independently by one author; prior to this, a random sample of 10% of the retrieved titles and abstracts were independently screened by two authors, achieving agreement of >80% on eligibility for initial inclusion, corresponding to good agreement. The following information was extracted from the full text of each article: meta-research area of interest; study design; type of studies evaluated in the meta-research; type of methodology used in assessment of the primary research; conflicts of interest reported; sponsorships reported; dental discipline; journal of publication; country of the first author; number of citations; and impact factor. A total of 7800 documents were initially retrieved. After analysis of the title/abstract and the full text of each article, and a snowballing procedure, 155 meta-research studies were identified and included. The 'methods' and 'reporting' meta-research areas were the most prevalent, with 73 (47%) and 61 (40%) studies, respectively. General dentistry, and orthodontics and dentofacial orthopaedics were the dental specialties with the greatest number/proportion of included studies with 45 (29%) and 28 (18%) studies, respectively. These findings may help to prioritize future meta-research in dentistry, consequently avoiding unnessecary investigations, and increasing the value of oral and dental research.

An Increase in Vigorous but Not Moderate Physical Activity Makes People Feel They Have Changed Their Behavior

Objective: While behavioral recommendations regarding physical activity commonly focus on reaching demanding goals by proposing “thresholds,” little attention has been paid to the question of how much of a behavioral change is needed to make people feel that they have changed. The present research investigated this relation between actual and felt behavior change.

Design: Using data from two longitudinal community samples, Study 1 and Study 2 comprised 614 (63% women) and 398 participants (61% women) with a mean age of 40.9 years (SD = 13.6) and 42.5 years (SD = 13.4), respectively. Using a stage-approach, participants were classified into four groups by asking them at the respective second measurement to indicate whether they had become more physically active since their last participation 6 months ago (“Changers”), they had tried but did not succeed in becoming more physically active (“Attempters”), they were already physically active on a regular basis (“Regular Actives”), or they had not tried to become more physically active (“Non-Attempters”). Physical activity was measured using the International Physical Activity Questionnaire (IPAQ), and fitness level was assessed as physical working capacity (PWC) via bicycle ergometry. Mixed ANOVAs including Time and Perceived Change as within and between factors were conducted, followed up by simple effect analyses.

Results: Participants stating to have become more active in the past 6 months (Changers) showed a significant increase in vigorous physical activity but not in moderate physical activity, with an average of 6.8 (Study 1) and 10.6 (Study 2) metabolic equivalent value-hours (MET-hours) per week in vigorous activity. Corroborating these findings, objective fitness also significantly increased in the group of Changers. No systematic change in moderate or vigorous physical activity was observed for the three other “non-changer” groups (Regular actives, Attempters, Non-Attempters).

Conclusion: The intensity of physical activity is the crucial variable for people’s perception of change in physical activity. Moderate physical activity seems not to be perceived as an effective means for behavior change. It thus might fail to unfold sufficient motivational impact, despite its known positive effects on health.

Compared to an active control condition, in persons with multiple sclerosis two different types of exercise training improved sleep and depression, but not fatigue, paresthesia, and intolerance of uncertainty

BACKGROUND

In persons with multiple sclerosis (MS), physical activity favorably impacts on psychological well-being. The aims of the present study were to investigate the influence of physical activity on depression, fatigue, sleep, paresthesia, and personality traits (intolerance of uncertainty), and to explore, if endurance training or coordinative training are superior to an active control condition.

METHODS

92 female individuals with MS (mean age: 37.36 years; mean EDSS: 2.43) took part in this intervention study. Participants were randomly assigned either to endurance training, coordinative training, or to an active control condition. At baseline, 4 weeks, and 8 weeks later at the end of the study, participants completed questionnaires on sleep, depression, fatigue, paresthesia and intolerance of uncertainty. Exercise training interventions took place three times/week for 45 min/session. Participants in the active control condition also met with the same duration and frequency.

RESULTS

Sleep complaints and symptoms of depression decreased over time, but more so in the exercising groups, compared to the active control group. No changes over time and between groups were observed for fatigue, paresthesia, and intolerance of uncertainty.

CONCLUSIONS

Both endurance and coordinative exercising had the potential to favorably impact on some aspects of cognitive-emotional processing, while also an active control condition appeared to have a positive impact.

Editors Should Declare Conflicts of Interest

Editors have increasing pressure as scholarly publishing tries to shore up trust and reassure academics and the public that traditional peer review is robust, fail-safe, and corrective. Hidden conflicts of interest (COIs) may skew the fairness of the publishing process because they could allow the status of personal or professional relationships to positively influence the outcome of peer review or reduce the processing period of this process. Not all authors have such privileged relationships. In academic journals, editors usually have very specialized skills and are selected as agents of trust, entrusted with the responsibility of serving as quality control gate-keepers during peer review. In many cases, editors form extensive networks, either with other professionals, industry, academic bodies, journals, or publishers. Such networks and relationships may influence their decisions or even their subjectivity towards a set of submitting authors, paper, or institute, ultimately influencing the peer review process. These positions and relationships are not simply aspects of a curriculum, they are potential COIs. Thus, on the editorial board of all academic journals, editors should carry a COI statement that reflects their past history, as well as actual relationships and positions that they have, as these may influence their editorial functions.

How scientific literature analysis yields innovative therapeutic hypothesis through integrative iterations

It is becoming generally accepted that the current diagnostic system often guarantees, rather than diminishes, disease heterogeneity. In effects, syndrome-dominated conceptual thinking has become a barrier to understanding the biological causes of complex, multifactorial diseases characterized by clinical and therapeutic heterogeneity. Furthermore, not only is the flood of currently available medical and biological information highly heterogeneous, it is also often conflicting. Together with the entire absence of functional models of pathogenesis and pathological evolution of complex diseases, this leads to a situation where illness activity cannot be coherently approached and where therapeutic developments become highly problematic. Acquisition of the necessary knowledge can be obtained, in parts, using in silico models produced through analytical approaches and processes collectively known as `Systems Biology'. However, without analytical approaches that specifically incorporate the facts that all that is called `information' is not necessarily useful nor utilisable and that all information should be considered as a priori suspect, modelling attempts will fail because of the much too numerous conflicting and, although correct in molecular terms, physiologically invalid reports. In the present essay, we suggest means whereby this body of problems could be functionally attacked and describe new analytical approaches that have demonstrated their efficacy in alleviating these difficulties.