Evidence synthesis for tackling research waste

Heterogeneity in effect size estimates

A typical empirical study involves choosing a sample, a research design, and an analysis path. Variation in such choices across studies leads to heterogeneity in results that introduce an additional layer of uncertainty, limiting the generalizability of published scientific findings. We provide a framework for studying heterogeneity in the social sciences and divide heterogeneity into population, design, and analytical heterogeneity. Our framework suggests that after accounting for heterogeneity, the probability that the tested hypothesis is true for the average population, design, and analysis path can be much lower than implied by nominal error rates of statistically significant individual studies. We estimate each type's heterogeneity from 70 multilab replication studies, 11 prospective meta-analyses of studies employing different experimental designs, and 5 multianalyst studies. In our data, population heterogeneity tends to be relatively small, whereas design and analytical heterogeneity are large. Our results should, however, be interpreted cautiously due to the limited number of studies and the large uncertainty in the heterogeneity estimates. We discuss several ways to parse and account for heterogeneity in the context of different methodologies.

Supporting study registration to reduce research waste

An estimated 82-89% of ecological research and 85% of medical research has limited or no value to the end user because of various inefficiencies. We argue that registration and registered reports can enhance the quality and impact of ecological research. Drawing on evidence from other fields, chiefly medicine, we support our claim that registration can reduce research waste. However, increasing registration rates, quality and impact will be very slow without coordinated effort of funders, publishers and research institutions. We therefore call on them to facilitate the adoption of registration by providing adequate support. We outline several aspects to be considered when designing a registration system that would best serve the field of ecology. To further inform the development of such a system, we call for more research to identify the causes of low registration rates in ecology. We suggest short- and long-term actions to bolster registration and reduce research waste.

Global mismatches between threat mapping research effort and the potential of threat abatement actions to reduce extinction risk

Threat mapping is a necessary tool for identifying and abating direct threats to species in the ongoing extinction crisis. There are known gaps in the threat mapping literature for particular threats and geographic locations, and it remains unclear if the distribution of research effort is appropriately targeted relative to conservation need. We aimed to determine the drivers of threat mapping research effort and to quantify gaps that, if filled, could inform actions with the highest potential to reduce species' extinction risk. We used a negative binomial generalized linear model to analyze research effort as a function of threat abatement potential (quantified as the potential reduction in species extinction risk from abating threats), species richness, land area, and human pressure. The model showed that threat mapping research effort increased by 1.1 to 1.2 times per standardized unit change in threat abatement potential. However, species richness and land area were stronger predictors of research effort overall. The greatest areas of mismatch between research effort and threat abatement potential, receiving disproportionately low research effort, were related to the threats to species of agriculture, aquaculture, and biological resource use across the tropical regions of the Americas, Asia, and Madagascar. Conversely, the threat of linear infrastructure (e.g., roads and rails) across regions, the threat of biological resource use (e.g., hunting or collection) in sub-Saharan Africa, and overall threats in North America and Europe all received disproportionately high research effort. We discuss the range of methodological and sociopolitical factors that may be behind the overall trends and specific areas of mismatch we found. We urge a stronger emphasis on targeting research effort toward those threats and geographic locations where threat abatement activities could make the greatest contribution to reducing global species extinction risk.

Comparative analysis of Cochrane and non-Cochrane reviews over three decades

BACKGROUND

Systematic reviews are viewed as the best study design to guide clinical decision-making as they are the least biased publications assuming they are well-conducted and include well-designed studies. Cochrane was initiated in 1993 with an aim of conducting high-quality systematic reviews. We aimed to examine the publication rates of non-Cochrane systematic reviews (henceforth referred to simply as "systematic reviews") and Cochrane reviews produced throughout Cochrane's existence and characterize changes throughout the period.

METHODS

This observational study collected data on systematic reviews published between 1993 and 2022 in PubMed. Identified Cochrane reviews were linked to data from the Cochrane Database of Systematic Reviews via their Digital Object Identifier. Systematic reviews and Cochrane reviews were analyzed separately. Two authors screened a random sample of records to validate the overall sample, providing a precision of 98%.

RESULTS

We identified 231,602 (94%) systematic reviews and 15,038 (6%) Cochrane reviews. Publication of systematic reviews has continuously increased with a median yearly increase rate of 26%, while publication of Cochrane reviews has decreased since 2015. From 1993 to 2002, Cochrane reviews constituted 35% of all systematic reviews in PubMed compared with 3.5% in 2013-2022. Systematic reviews consistently had fewer authors than Cochrane reviews, but the number of authors increased over time for both. Chinese first authors conducted 15% and 4% of systematic reviews published from 2013-2022 and 2003-2012, respectively. Most Cochrane reviews had first authors from the UK (36%). The native English-speaking countries the USA, the UK, Canada, and Australia produced a large share of systematic reviews (42%) and Cochrane reviews (62%). The largest publishers of systematic reviews in the last 10 years were gold open access journals.

CONCLUSIONS

Publication of systematic reviews is increasing rapidly, while fewer Cochrane reviews have been published through the last decade. Native English-speaking countries produced a large proportion of both types of systematic reviews. Gold open access journals and Chinese first authors dominated the publication of systematic reviews for the past 10 years. More research is warranted examining why fewer Cochrane reviews are being published. Additionally, examining these systematic reviews for research waste metrics may provide a clearer picture of their utility.

Exploring trial publication and research waste in COVID-19 randomised trials of hydroxychloroquine, corticosteroids, and vitamin D: a meta-epidemiological cohort study

BACKGROUND

The global research response to the COVID-19 pandemic was impressive, but also led to an infodemic and considerable research waste. Registered, but unpublished trials added to this noise. We aimed to determine the proportion of registered randomised trials of common COVID-19 treatments that were published and to describe the characteristics of these trials to examine the association between trial characteristics, publication status and research waste.

METHODS

This meta-epidemiological cohort study used a sample of randomised trials of corticosteroids, hydroxychloroquine or vitamin D as treatments for COVID-19, registered between 1 November 2019 and 31 December 2021 and available via the WHO ICTRP portal. We searched for the trials' published results up to 20 October 2022. We extracted the trial characteristics, analysing with descriptive statistics. We performed univariate logistic regression to examine the association between trials' characteristics and publication status, followed by multiple logistic regression using significant characteristics to assess the association between trial characteristics and publication status.

RESULTS

We identified 357 eligible trials on ICTRP. Of these, 107 (30%) had published or made their results available publicly by 20 October 2022, while 250 (70%) had not been published or shared their results publicly. Multiple logistic regression analysis showed that a larger target sample size was a significant positive predictor of publication with target sample sizes above 300 almost tripling the odds of publication (aOR: 2.75, 95% CI: 1.35 to 5.62).

CONCLUSIONS

Less than one third of registered trials made their results public and our findings identified that many trialists had not updated their trial registry entry with the trial status, results or both. Failure to share trial results publicly is a disservice to patients, clinicians and policy makers and adds to research waste.

Advanced methods and implementations for the meta-analyses of animal models: Current practices and future recommendations

Meta-analytic techniques have been widely used to synthesize data from animal models of human diseases and conditions, but these analyses often face two statistical challenges due to complex nature of animal data (e.g., multiple effect sizes and multiple species): statistical dependency and confounding heterogeneity. These challenges can lead to unreliable and less informative evidence, which hinders the translation of findings from animal to human studies. We present a literature survey of meta-analysis using animal models (animal meta-analysis), showing that these issues are not adequately addressed in current practice. To address these challenges, we propose a meta-analytic framework based on multilevel (linear mixed-effects) models. Through conceptualization, formulations, and worked examples, we illustrate how this framework can appropriately address these issues while allowing for testing new questions. Additionally, we introduce other advanced techniques such as multivariate models, robust variance estimation, and meta-analysis of emergent effect sizes, which can deliver robust inferences and novel biological insights. We also provide a tutorial with annotated R code to demonstrate the implementation of these techniques.

We need to talk about nonprobability samples

In most circumstances, probability sampling is the only way to ensure unbiased inference about population quantities where a complete census is not possible. As we enter the era of 'big data', however, nonprobability samples, whose sampling mechanisms are unknown, are undergoing a renaissance. We explain why the use of nonprobability samples can lead to spurious conclusions, and why seemingly large nonprobability samples can be (effectively) very small. We also review some recent controversies surrounding the use of nonprobability samples in biodiversity monitoring. These points notwithstanding, we argue that nonprobability samples can be useful, provided that their limitations are assessed, mitigated where possible and clearly communicated. Ecologists can learn much from other disciplines on each of these fronts.

Reproducibility and Scientific Integrity of Big Data Research in Urban Public Health and Digital Epidemiology: A Call to Action

The emergence of big data science presents a unique opportunity to improve public-health research practices. Because working with big data is inherently complex, big data research must be clear and transparent to avoid reproducibility issues and positively impact population health. Timely implementation of solution-focused approaches is critical as new data sources and methods take root in public-health research, including urban public health and digital epidemiology. This commentary highlights methodological and analytic approaches that can reduce research waste and improve the reproducibility and replicability of big data research in public health. The recommendations described in this commentary, including a focus on practices, publication norms, and education, are neither exhaustive nor unique to big data, but, nonetheless, implementing them can broadly improve public-health research. Clearly defined and openly shared guidelines will not only improve the quality of current research practices but also initiate change at multiple levels: the individual level, the institutional level, and the international level.

The role of scoping reviews in reducing research waste

OBJECTIVES

Scoping reviews and evidence map methodologies are increasingly being used by researchers. The objective of this article is to examine how scoping reviews can reduce research waste.

STUDY DESIGN AND SETTING

This article summarizes the key issues facing the research community regarding research waste and how scoping reviews can make an important contribution to the reduction of research waste in both primary and secondary research.

RESULTS

The problem of research waste is an enduring challenge for global health, leading to a waste of human and financial resources and producing research outputs that do not provide answers to the most pressing research questions. Research waste occurs within primary research but also in secondary research such as evidence syntheses. The focus of scoping reviews on characterizing the nature of existing evidence on a topic and including all types of evidence, potentially reduces research waste in five ways: (1) identifying key research gaps on a topic, (2) determining appropriate outcome measures, (3) mapping existing methodological approaches, (4) developing a consistent understanding of terms and concepts used in existing evidence, and (5) ensuring scoping reviews do not exacerbate the issue of research waste.

CONCLUSION

To ensure that scoping reviews do not themselves end up contributing to research waste, it is important to register the scoping review and to ensure that international reporting standards and methodological guidance are followed.

Summarizing the effects of different exercise types in chronic low back pain - a systematic review of systematic reviews

Background

In chronic LBP (CLBP), guideline-endorsed treatment is to stay active, return to normal activity, and to exercise. Several reviews on various exercise types used in CLBP have been published. We aimed to identify systematic reviews of common exercise types used in CLBP, to appraise their quality, and to summarize and compare their effect on pain and disability.

Methods

We searched the databases OVID MEDLINE, EMBASE, COCHRANE LIBRARY, and WEB OF SCIENCE (Core collection) for systematic reviews and meta-analyses on adults between 18 and 70 years of age suffering from chronic or recurrent LBP for a period of at least 12 weeks, which investigated the effects of exercises on pain and disability. All searches were conducted without language restriction. The search was performed up until 2022–01-26. The included reviews were grouped into nine exercise types: aerobic training, aquatic exercises, motor control exercises (MCE), resistance training, Pilates, sling exercises, traditional Chinese exercises (TCE), walking, and yoga. The study quality was assessed with AMSTAR-2. For each exercise type, a narrative analysis was performed, and the level of evidence for the effects of exercise was assessed through GRADE.

Results

Our database search resulted in 3,475 systematic reviews. Out of the 253 full texts that were screened, we included 45 systematic reviews and meta-analyses. The quality of the included reviews ranged from high to critically low. Due to large heterogeneity, no meta-analyses were performed. We found low-to-moderate evidence of mainly short-term and small beneficial effects on pain and disability for MCE, Pilates, resistance training, TCE, and yoga compared to no or minimal intervention.

Conclusions

Our findings show that the effect of various exercise types used in CLBP on pain and disability varies with no major difference between exercise types. Many of the included systematic reviews were of low-to-moderate quality and based on randomized controlled trials with high risk of bias. The conflicting results seen, undermine the certainty of the results leading to very-low-to-moderate quality of evidence for our results. Future systematic reviews should be of higher quality to minimize waste of resources.

Trial registration

PROSPERO: Reg no 190409 Registration date 01AUG 2020.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12891-022-05722-x.

Quantifying research waste in ecology

Research inefficiencies can generate huge waste: evidence from biomedical research has shown that most research is avoidably wasted and steps have been taken to tackle this costly problem. Although other scientific fields could also benefit from identifying and quantifying waste and acting to reduce it, no other estimates of research waste are available. Given that ecological issues interweave most of the United Nations Sustainable Development Goals, we argue that tackling research waste in ecology should be prioritized. Our study leads the way. We estimate components of waste in ecological research based on a literature review and a meta-analysis. Shockingly, our results suggest only 11-18% of conducted ecological research reaches its full informative value. All actors within the research system-including academic institutions, policymakers, funders and publishers-have a duty towards science, the environment, study organisms and the public, to urgently act and reduce this considerable yet preventable loss. We discuss potential ways forward and call for two major actions: (1) further research into waste in ecology (and beyond); (2) focused development and implementation of solutions to reduce unused potential of ecological research.

Open Data Practices among Users of Primary Biodiversity Data

Presence-only biodiversity data are increasingly relied on in biodiversity, ecology, and conservation research, driven by growing digital infrastructures that support open data sharing and reuse. Recent reviews of open biodiversity data have clearly documented the value of data sharing, but the extent to which the biodiversity research community has adopted open data practices remains unclear. We address this question by reviewing applications of presence-only primary biodiversity data, drawn from a variety of sources beyond open databases, in the indexed literature. We characterize how frequently researchers access open data relative to data from other sources, how often they share newly generated or collated data, and trends in metadata documentation and data citation. Our results indicate that biodiversity research commonly relies on presence-only data that are not openly available and neglects to make such data available. Improved data sharing and documentation will increase the value, reusability, and reproducibility of biodiversity research.

The data-index: An author-level metric that values impactful data and incentivizes data sharing

Author-level metrics are a widely used measure of scientific success. The h-index and its variants measure publication output (number of publications) and research impact (number of citations). They are often used to influence decisions, such as allocating funding or jobs. Here, we argue that the emphasis on publication output and impact hinders scientific progress in the fields of ecology and evolution because it disincentivizes two fundamental practices: generating impactful (and therefore often long-term) datasets and sharing data. We describe a new author-level metric, the data-index, which values both dataset output (number of datasets) and impact (number of data-index citations), so promotes generating and sharing data as a result. We discuss how it could be implemented and provide user guidelines. The data-index is designed to complement other metrics of scientific success, as scientific contributions are diverse and our value system should reflect that both for the benefit of scientific progress and to create a value system that is more equitable, diverse, and inclusive. Future work should focus on promoting other scientific contributions, such as communicating science, informing policy, mentoring other scientists, and providing open-access code and tools.

European agroforestry has no unequivocal effect on biodiversity: a time-cumulative meta-analysis

Background

Agroforestry is a production system combining trees with crops or livestock. It has the potential to increase biodiversity in relation to single-use systems, such as pastures or cropland, by providing a higher habitat heterogeneity. In a literature review and subsequent meta-analysis, we investigated the relationship between biodiversity and agroforestry and critically appraised the underlying evidence of the results.

Results

Overall, there was no benefit of agroforestry to biodiversity. A time-cumulative meta-analysis demonstrated the robustness of this result between 1991 and 2019. In a more nuanced view silvopastoral systems were not more diverse in relation to forests, pastures or abandoned silvopastures. However, silvoarable systems increased biodiversity compared to cropland by 60%. A subgroup analysis showed that bird and arthropod diversity increased in agroforestry systems, while bats, plants and fungi did not.

Conclusion

Agroforestry increases biodiversity only in silvoarable systems in relation to cropland. But even this result is of small magnitude, and single-study effect sizes were heterogeneous with sometimes opposing conclusions. The heterogeneity suggests the importance of other, usually unmeasured variables, such as landscape parameters or land-use history, influencing biodiversity in agroforestry systems.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12862-021-01911-9.

Nearly 80 systematic reviews were published each day: Observational study on trends in epidemiology and reporting over the years 2000-2019

BACKGROUND

Systematic reviews (SRs) are useful tools in synthesising the available evidence, but high numbers of overlapping SRs are also discussed in the context of research waste. Although it is often claimed that the number of SRs being published is increasing steadily, there are no precise data on that. We aimed to assess trends in the epidemiology and reporting of published SRs over the last 20 years.

METHODS

A retrospective observational study was conducted to identify potentially eligible SRs indexed in PubMed from 2000 to 2019. From all 572,871 records retrieved, we drew a simple random sample of 4,000. The PRISMA-P definition of SRs was applied to full texts and only SRs published in English were included. Characteristics were extracted by one reviewer, with a 20% sample verified by a second person.

RESULTS

A total of 1,132 SRs published in 710 different journals were included. The estimated number of SRs indexed in 2000 was 1,432 (95% CI: 547-2,317), 5,013 (95% CI: 3,375-6,650) in 2010 and 29,073 (95% CI: 25,445-32,702) in 2019. Transparent reporting of key items increased over the years. About 7 out of 10 named their article a SR (2000-2004: 41.9% and 2015-2019: 74.4%). In 2000-2004, 32.3% of SRs were based in the UK (0% in China), in 2015-2019 24.0% were from China and 10.8% from the UK. Nearly all articles from China (94.9%) conducted a meta-analysis (overall: 58.9%). Cochrane reviews (n = 84; 7.4%) less often imposed language restrictions, but often did not report the number of records and full texts screened and did not name their article a SR (22.6% vs. 73.4%).

CONCLUSIONS

We observed a more than 20-fold increase in the number of SRs indexed over the last 20 years. In 2019, this is equivalent to 80 SRs per day. Over time, SRs got more diverse in respect to journals, type of review, and country of corresponding authors. The high proportion of meta-analyses from China needs further investigation.

STUDY REGISTRATION

Open Science Framework (https://osf.io/pxjrv/).

From a crisis to an opportunity: Eight insights for doing science in the COVID-19 era and beyond

The COVID-19 crisis has forced researchers in Ecology to change the way we work almost overnight. Nonetheless, the pandemic has provided us with several novel components for a new way of conducting science. In this perspective piece, we summarize eight central insights that are helping us, as early career researchers, navigate the uncertainties, fears, and challenges of advancing science during the COVID-19 pandemic. We highlight how innovative, collaborative, and often Open Science-driven developments that have arisen from this crisis can form a blueprint for a community reinvention in academia. Our insights include personal approaches to managing our new reality, maintaining capacity to focus and resilience in our projects, and a variety of tools that facilitate remote collaboration. We also highlight how, at a community level, we can take advantage of online communication platforms for gaining accessibility to conferences and meetings, and for maintaining research networks and community engagement while promoting a more diverse and inclusive community. Overall, we are confident that these practices can support a more inclusive and kinder scientific culture for the longer term.

Next-generation field courses: Integrating Open Science and online learning

As Open Science practices become more commonplace, there is a need for the next generation of scientists to be well versed in these aspects of scientific research. Yet, many training opportunities for early career researchers (ECRs) could better emphasize or integrate Open Science elements. Field courses provide opportunities for ECRs to apply theoretical knowledge, practice new methodological approaches, and gain an appreciation for the challenges of real-life research, and could provide an excellent platform for integrating training in Open Science practices. Our recent experience, as primarily ECRs engaged in a field course interrupted by COVID-19, led us to reflect on the potential to enhance learning outcomes in field courses by integrating Open Science practices and online learning components. Specifically, we highlight the opportunity for field courses to align teaching activities with the recent developments and trends in how we conduct research, including training in: publishing registered reports, collecting data using standardized methods, adopting high-quality data documentation, managing data through reproducible workflows, and sharing and publishing data through appropriate channels. We also discuss how field courses can use online tools to optimize time in the field, develop open access resources, and cultivate collaborations. By integrating these elements, we suggest that the next generation of field courses will offer excellent arenas for participants to adopt Open Science practices.

Safety and efficacy of stem cell therapy: an overview protocol on published meta-analyses and evidence mapping

Background

Stem cell therapy (SCT) is an emerging and promising treatment measure for many conditions (e.g., chronic liver disease, diabetes mellitus, and knee osteoarthritis). Although there are numerous meta-analyses (MAs) concerning SCT, the quality of these MAs and the efficacy and safety data for SCT reported in these MAs remain unknown. Therefore, it is of utmost importance to conduct an overview of existing MAs concerning SCT for evaluating these parameters.

Methods

We will systematically search PubMed and EMBASE databases from inception to October 2020 for identifying MAs of SCT published in English. Two independent reviewers will select appropriate MAs against the predefined eligibility criteria. The efficacy and safety data of SCT reported in MAs will be descriptively summarized. Following this, the reporting quality and methodological quality of included MAs will be appraised using Preferred Reporting Items for Systematic reviews and Meta-analyses (PRISMA) and A Measurement Tool to Assess Systematic Reviews 2 (AMSTAR-2) tools by two reviewers, respectively. Further, the evidence mapping method will be used to present assessment results. The key information will also be extracted by two independent reviewers. The Spearman’s correlation coefficient will be used to explore the association between reporting quality and methodological quality. The factors influencing the quality will be assessed through linear regression analyses. The sensitivity analysis will also be conducted. Data analyses will be performed using Stata 16.0 and Excel 2016. P<0.05 will be considered statistically significant.

Discussion

This overview of MAs concerning SCTs will provide comprehensive evidence on the quality of MAs and data of interest reported in MAs. Further, these data can be used to guide clinical practice and future research.

Overview registration

International Prospective Register of Systematic Reviews (PROSPERO): CRD42020206642.