Does citation matter? Research citation in policy documents as an indicator of research impact - an Australian obesity policy case-study

Improving academic and public health impact of Cochrane public health reviews: what can we learn from bibliographic metrics and author dissemination strategies? A cross-sectional study

BACKGROUND

To facilitate the development of impactful research dissemination strategies, this study aimed to: (i) survey authors of trials included in a sample of Cochrane reviews to describe strategies to disseminate trial findings, and examine their association with academic and policy impacts and (ii) audit academic and policy impact of CPH reviews.

METHODS

Authors of 104 trials within identified Cochrane reviews completed survey items assessing the dissemination strategies. Field weighted citation (FWCI) data extracted from bibliographic databases served as a measure of academic impact of trials and CPH reviews. Policy and practice impacts of trials were assessed during the survey of trial authors using items based on the Payback Framework, and for CPH reviews using 'policy mention' data collected via Altmetric Explorer.

RESULTS

Among the included trials, univariate (but not multivariable) regression models revealed significant associations between the use of dissemination strategies (i.e. posts on social media; workshops with end-users; media-releases) and policy or practice impacts. No significant associations were reported between dissemination strategies and trial FWCI. The mean FWCI of CPH reviews suggest that they are cited 220% more than other reviews in their field.

CONCLUSIONS

Comprehensive dissemination strategies are likely required to maximize the potential the potential impacts of public health research.

Influence of research on open science in the public policy sphere

This paper analyses the scientific activity related to open science in Spain and its influence on public policy from a bibliometric perspective. For this purpose, Spanish centres' projects and publications on open science from 2010 to 2020 are studied. Subsequently, policy documents using papers related to open science are analysed to study their influence on policymaking. A total of 142 projects and 1491 publications are analysed, 15% of which are mentioned in policy documents.The publications cited in policy documents display high proportions of international collaboration, open access publication and publication in first-quartile journals. The findings underline governments’ leading role in the implementation of open science policies and the funding of open science research. The same government agencies that promote and fund open science research are shown to use that research in their institutional reports, a process known as knowledge flow feedback. Other non-academic actors are also observed to make use of the knowledge produced by open science research, showing how the open science movement has crossed the boundaries of academia.

A research intelligence approach to assess the research impact of the Dutch university medical centres

BACKGROUND

The way in which research impact is evaluated and assessed has long been under debate. In recent years the focus is moving away from the use of numerical indicators, towards an emphasis on narratives. The Dutch university medical centres (UMCs) have a long-standing tradition of using bibliometric indicators. Because of the declining interest in indicators alone, this study was designed to repurpose bibliometrics to answer specific strategic questions. In this article we discuss the strategic and policy-based questions, the methodology we used in uncovering relevant information and conclusions we draw from the analyses we performed. The aim of this article is to inform a broader audience about the potential applications of bibliometric information to support a new form of research intelligence.

METHODS

In this study we used a curated set of publications from the UMCs. We performed different bibliometric analyses and used bibliometric visualization tools to shed light on research focus, open science practices, collaboration, societal impact and scientific impact.

RESULTS

The analyses allowed us to visualize and contextualize the research focus of the UMCs as a whole, but also to show specific focus areas of each UMC. The UMCs are active in the full spectrum of biomedical research, and at the same time are very complementary to each other. Furthermore, we were able to show the development of open access of UMC publications over time, to support the national mission. Visualizing collaboration is a powerful way of showing both the international orientation and the regional and national engine function of UMCs in research. We were able to assess societal impact by looking at the different channels in which publications find their way to societally relevant sources such as news media, policy documents and guidelines. Finally, we assessed scientific impact and put this into an international perspective.

CONCLUSIONS

Research intelligence is able to transform bibliometric information by interpretation and annotation into highly relevant insights that can be used for several different strategic purposes and for research impact assessment in general.

Pharmacologic therapies for neuropathic pain: an assessment of reporting biases in randomized controlled trials

ABSTRACT

Several different reporting biases cited in scientific literature have raised concerns about the overestimation of effects and the subsequent potential impact on the practice of evidence-based medicine and human health. Up to 7% to 8% of the population experiences neuropathic pain (NP), and established treatment guidelines are based predominantly on published clinical trial results. Therefore, we examined published randomized controlled trials (RCTs) of first-line drugs for NP and assessed the relative proportions with statistically significant (ie, positive) and nonsignificant (ie, negative) results and their rates of citation. We determined the relationships between reported study outcome and the frequency of their citations with journal impact factor, sample size, time to publication after study completion, and study quality metrics. We also examined the association of study outcome with maximum study drug dosage and conflict of interest. We found that of 107 published RCTs, 68.2% reported a statistically significant outcome regarding drug efficacy for chronic peripheral and central NP. Positive studies were cited nearly twice as often as negative studies in the literature (P = 0.01), despite similar study sample size, quality metrics, and publication in journals with similar impact factors. The time to publication, journal impact factor, and conflict of interest did not differ statistically between positive and negative studies. Our observations that negative and positive RCTs were published in journals with similar impact at comparable time-lags after study completion are encouraging. However, the citation bias for positive studies could affect the validity and generalization of conclusions in literature and potentially influence clinical practice.

What informs oral health and chronic disease policy development in Australia: a citation analysis

Improvement of health services and patient outcomes depends on the translation of health research into health policy. Oral health research can inform policies to manage chronic diseases and improve quality of life of affected individuals. To determine if translation of this research into policy is occurring, we identify where policymakers obtain evidence to inform the development of Australian health policy. We conducted a citation analysis of oral health, cardiovascular disease, diabetes and cognitive impairment National policies to determine if current oral health research has informed oral health and chronic disease policy. We analysed five National policies with 268 individual references to policy or organizational documents (n = 179), peer-reviewed research (n = 74), grey literature (n = 12), or unidentifiable (n = 3). Although we found oral health references listed in the National policies (92), we did not find this information to have been translated into the oral health and chronic disease policies we analysed.

Looking for evidence of research impact and use: A qualitative study of an Australian research-policy system

Current assessments of research impact have been criticized for capturing what can be easily counted not what actually counts. To empirically examine this issue, we approached measuring research impact from two directions, tracing forwards from research and backwards from policy, within a defined research-policy system (childhood obesity prevention research and policy in New South Wales, Australia from 2000 to 2015). The forward tracing research impact assessment component traced a sample of 148 local research projects forward to examine their policy impacts. Of the projects considered, 16% had an impact on local policy and for a further 19%, decision-makers were aware of the research, but there was no evidence it influenced policy decisions. The backward tracing component of the study included an analysis of research use across three policy initiatives. It provided a more nuanced understanding of the relative influence of research on policy. Both direct uses of specific research and indirect uses of research incorporated as broader bodies of knowledge were evident. Measuring research impact from both directions captured the diverse ways that research was used in decision-making. Our findings illustrate complexities in the assessment process and in real-life policymaking trajectories. They highlight the role that timing of assessment plays in perception of impacts and difficulties attributing longer-term impacts to specific research. This study supports the use of models where politics and complex system dynamics shape knowledge and its influence on decision-making, rather than research being the primary driver for policy change.

Research and Policy Impact of Trials Published by the UK National Institute of Health Research (2006-2015)

OBJECTIVES

Health technology assessment aims to inform and support healthcare decision making, and trials are part of that process. The purpose of this study was to measure the impact of a sample of trials in a meaningful but robust fashion.

METHODS

All randomized controlled trials funded and published by the UK National Institute of Health Research in the Health Technology Assessment journal series and other peer-reviewed journals were identified for 2006 to 2015. Citation analysis was performed for all trials, and quantitative content analysis was undertaken on a purposive sample to determine whether impact could be categorized as "instrumental" (ie, having a clear influence on key research and policy publications).

RESULTS

The search identified 133 relevant trials. The citation rate per trial was 102.97. Of the 133 trials, 129 (98%) were cited in 1 or more systematic reviews or meta-analyses (mean per trial = 7.18, range = 0-44). Where they were cited, the trials were used in some form of synthesis 63% of the time. Ninety-one of the 133 (68%) trials were found to be cited in 1 or more guidance or policy document (mean per trial = 2.75, range = 0-26) and had an instrumental influence 41% of the time. The publication of these trials' results in journals other than the Health Technology Assessment journal appears to enhance the discoverability of the trial data. Altmetric.com proved to be very useful in identifying unique policy and guidance documents.

CONCLUSION

These trials have impressive citation rates, and a sizeable proportion are certainly being used in key publications in a genuinely instrumental manner.

Evidence map of knowledge translation strategies, outcomes, facilitators and barriers in African health systems

BACKGROUND

The need for research-based knowledge to inform health policy formulation and implementation is a chronic global concern impacting health systems functioning and impeding the provision of quality healthcare for all. This paper provides a systematic overview of the literature on knowledge translation (KT) strategies employed by health system researchers and policy-makers in African countries.

METHODS

Evidence mapping methodology was adapted from the social and health sciences literature and used to generate a schema of KT strategies, outcomes, facilitators and barriers. Four reference databases were searched using defined criteria. Studies were screened and a searchable database containing 62 eligible studies was compiled using Microsoft Access. Frequency and thematic analysis were used to report study characteristics and to establish the final evidence map. Focus was placed on KT in policy formulation processes in order to better manage the diversity of available literature.

RESULTS

The KT literature in African countries is widely distributed, problematically diverse and growing. Significant disparities exist between reports on KT in different countries, and there are many settings without published evidence of local KT characteristics. Commonly reported KT strategies include policy briefs, capacity-building workshops and policy dialogues. Barriers affecting researchers and policy-makers include insufficient skills and capacity to conduct KT activities, time constraints and a lack of resources. Availability of quality locally relevant research was the most reported facilitator. Limited KT outcomes reflect persisting difficulties in outcome identification and reporting.

CONCLUSION

This study has identified substantial geographical gaps in knowledge and evidenced the need to boost local research capacities on KT practices in low- and middle-income countries. Evidence mapping is also shown to be a useful approach that can assist local decision-making to enhance KT in policy and practice.

Updated Editorial Guidance for Quality and Reliability of Research Output

Over the past few years, updated editorial policy statements of several associations have provided a platform for improving the quality of scientific research and publishing. The updates have particularly pointed to the need for following research reporting standards, authorship and contributorship regulations, implementing digital tools for the identification and crediting academic contributors, and moving towards optimal ethical open-access models. This article overviews some of the recent editorial policy statements of global editorial associations and reflects on the role of the regional counterparts in advancing scholarly publishing. One of the globally promoted documents is the Recommendations of the International Committee of Medical Journal Editors (ICMJE). Its latest versions contain statements on proper research reporting, reviewing, editing, and publishing. Points on ethical target journals and 'predatory' sources are also available. This year, in a move to update its editorial policy, the Committee on Publication Ethics (COPE) released the Core Practices, comprehensively reflecting on the major issues in publication ethics. Updated joint statements of medical writers associations are also available to implement transparent policy on contributorship in sponsor-supported research projects and related reports. Several suggestions are put forward to improve global editorial statements on online profiling, crediting, and referencing. It is also highlighted that knowledge and implementation of updated editorial guidance is essential for editors' good standing.

A 14-year literature survey on spine-related clinical research output by orthopedic surgeons from mainland China

BACKGROUND

In recent years, China is increasingly playing an active role in various fields of biomedical research. Many bibliometric studies have provided valuable insights to different fields of clinical studies. However, similar evaluation on spine surgery-related clinical research is still limited. We herein aimed to examine the scientific publications by orthopedic spine surgeons from mainland China within a 14-year period.

METHODS

Articles were identified in PubMed using predetermined query terms. Descriptive statistics were calculated, and T tests, Chi-squared tests, and regression analysis were conducted on the number of publications, impact factors (IFs), citations, region of the study, and associated medical subject headings (MeSHs).

RESULTS

A total of 1498 articles were identified and the annual number of publications, citations, and IFs all increased exponentially. The average IF was significantly higher in 2007 to 2013 than 2000 to 2006. Most publications were from Shanghai and Beijing and the 5 most productive administrative regions generated 70% of all publications. Analysis of associated MeSHs suggested research topics became more heterogeneous over the study period.

CONCLUSION

This was the first comprehensive evaluation on the clinical research output by orthopedic spine surgeons from mainland China. The annual number of publications and citations both increased significantly; however, research was highly concentrated in a handful of administrative regions.