Multidimensional assessment of scholarly research impact

Incidence of periprosthetic joint infection after primary total knee arthroplasty shows significant variation : a synthesis of meta-analysis and bibliometric analysis

Total Knee Arthroplasty (TKA) is a surgery that is commonly performed on older adults to improve their quality of life. However, the increasing use of knee joint prostheses has led to a rise in the incidence of Prosthetic Joint Infections (PJI) in patients after TKA. Different clinical studies have looked at the occurrence of PJI after TKA in different regions, but they have drawn varying conclusions. To better understand this topic, we conducted a meta-analysis and bibliometric study using data from multiple databases. Our research found that the estimated prevalence of PJI after TKA is approximately 1.08% across different regions, but there is still considerable variation. Additionally, our regression analysis of sub-groups shows significant differences in follow-up periods. Furthermore, our comprehensive bibliometric analysis identifies current research trends, "hotspots" related to TKA-related PJI, influential nations, organizations, and noteworthy publications. Our analysis provides valuable insights to guide future research in this area.

The role of microRNAs in axon regeneration after peripheral nerve injury: a bibliometric analysis

Objective

This study analyzed the current research hotspots and future development trends of the therapeutic effects of microRNA on PNI axonal regeneration through bibliometric methods. Moreover, the current advantages and disadvantages of this field as well as future development prospects are discussed in depth.

Methods

CiteSpace V and VOSviewer were used as bibliometric tools to complete the analysis of the research focus and direction of the published articles. To supplement, sort out, and summarize, we analyzed the research status of the study on the application of microRNAs for axonal regeneration after peripheral nerve injury from 2013 to 2023.

Results

A total of 207 publications were retrieved from the Web of Science database. After exclusion and screening, a final selection of 174 articles that met the research criteria. These 174 articles were authored by a total of 846 individuals, representing 24 countries and 199 institutions. Additionally, this study presents information on the annual publication output, country distribution, top 5 contributing authors, top 5 most cited articles, and top 10 contributing institutions.

Conclusion

As one of the hottest topics today, microRNAs have become the current research hotspot in neural inflammation, neural cell repair and regeneration, neural protection, and functional recovery. With more investment in research in this field, more high-quality articles will be published in both domestic and international outstanding journals, which will bring a new era for the treatment of peripheral nerve injury.

The inverted U-shaped relationship between knowledge diversity of researchers and societal impact

With the increasing importance of interdisciplinary research, some studies have focused on the role of reference diversity by analysing reference lists of published papers. However, the relationship between the knowledge diversity of collaborating team members and research performance has been overlooked. In this study, we measured knowledge diversity through the disciplinary attributes of collaborating authors and research performance (understood as societal impact) through altmetric data. The major findings are: (1) The relationship between interdisciplinary collaboration diversity and societal impact is not a simple linear one, showing an inverted U-shaped pattern; and (2) As the number of collaborative disciplines increases, the marginal effects diminish or even become outweighed by the costs, showing a predominance of negative influences. Hence, diversity in interdisciplinary collaboration does not always have a positive impact. Research collaborations need to take into account the cost issues associated with the diversity of member disciplines.

A heuristic approach based on Leiden rankings to identify outliers: evidence from Italian universities in the European landscape

We propose an innovative use of the Leiden Rankings (LR) in institutional management. Although LR only consider research output of major universities reported in Web of Science (WOS) and share the limitations of other existing rankings, we show that they can be used as a base of a heuristic approach to identify “outlying” institutions that perform significantly below or above expectations. Our approach is a non-rigorous intuitive method (“heuristic”) because is affected by all the biases due to the technical choices and incompleteness that affect the LR but offers the possibility to discover interesting findings to be systematically verified later. We propose to use LR as a departure base on which to apply statistical analysis and network mapping to identify “outlier” institutions to be analyzed in detail as case studies. Outliers can inform and guide science policies about alternative options. Analyzing the publications of the Politecnico di Bari in more detail, we observe that “small teams” led by young and promising scholars can push the performance of a university up to the top of the LR. As argued by Moed (Applied evaluative informetrics. Springer International Publishing, Berlin, 2017a), supporting “emerging teams”, can provide an alternative to research support policies, adopted to encourage virtuous behaviours and best practices in research. The results obtained by this heuristic approach need further verification and systematic analysis but may stimulate further studies and insights on the topics of university rankings policy, institutional management, dynamics of teams, good research practice and alternative funding methods.

Mapping the impact of papers on various status groups in excellencemapping.net: a new release of the excellence mapping tool based on citation and reader scores

In over five years, Bornmann, Stefaner, de Moya Anegon, and Mutz (2014b) and Bornmann, Stefaner, de Moya Anegón, and Mutz (2014c, 2015) have published several releases of the www.excellencemapping.net tool revealing (clusters of) excellent institutions worldwide based on citation data. With the new release, a completely revised tool has been published. It is not only based on citation data (bibliometrics), but also Mendeley data (altmetrics). Thus, the institutional impact measurement of the tool has been expanded by focusing on additional status groups besides researchers such as students and librarians. Furthermore, the visualization of the data has been completely updated by improving the operability for the user and including new features such as institutional profile pages. In this paper, we describe the datasets for the current excellencemapping.net tool and the indicators applied. Furthermore, the underlying statistics for the tool and the use of the web application are explained.

Societal impact of research: a text mining study of impact types

In addition to academic impact, researchers are increasingly concerned with understanding and demonstrating the practical impact of research outside academia. Several frameworks capturing key impact types have been developed based on project experiences, expert opinions, and surveys. This empirical study seeks to contribute to this development by identifying impact types documented in 6,882 case studies submitted to impact evaluation groups in Australia (Engagement and Impact Assessment) and the United Kingdom (Research Excellence Framework). The results of text mining indicate three emerging impact types that extend existing frameworks in terms of the recognition of new opportunities, the length of use, and experience improvement, thereby allowing a variety of researchers, not just those who address popular, short-term, and instrumental issues, to understand and demonstrate their practice impact.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.

Evaluating cancer research impact: lessons and examples from existing reviews on approaches to research impact assessment

BACKGROUND

Performing cancer research relies on substantial financial investment, and contributions in time and effort from patients. It is therefore important that this research has real life impacts which are properly evaluated. The optimal approach to cancer research impact evaluation is not clear. The aim of this study was to undertake a systematic review of review articles that describe approaches to impact assessment, and to identify examples of cancer research impact evaluation within these reviews.

METHODS

In total, 11 publication databases and the grey literature were searched to identify review articles addressing the topic of approaches to research impact assessment. Information was extracted on methods for data collection and analysis, impact categories and frameworks used for the purposes of evaluation. Empirical examples of impact assessments of cancer research were identified from these literature reviews. Approaches used in these examples were appraised, with a reflection on which methods would be suited to cancer research  impact evaluation going forward.

RESULTS

In total, 40 literature reviews were identified. Important methods to collect and analyse data for impact assessments were surveys, interviews and documentary analysis. Key categories of impact spanning the reviews were summarised, and a list of frameworks commonly used for impact assessment was generated. The Payback Framework was most often described. Fourteen examples of impact evaluation for cancer research were identified. They ranged from those assessing the impact of a national, charity-funded portfolio of cancer research to the clinical practice impact of a single trial. A set of recommendations for approaching cancer research impact assessment was generated.

CONCLUSIONS

Impact evaluation can demonstrate if and why conducting cancer research  is worthwhile. Using a mixed methods, multi-category assessment organised within a framework, will provide a robust evaluation, but the ability to perform this type of assessment may be constrained by time and resources. Whichever approach is used, easily measured, but inappropriate metrics should be avoided. Going forward, dissemination of the results of cancer research impact assessments will allow the cancer research community to learn how to conduct these evaluations.

Reliability of researcher capacity estimates and count data dispersion: a comparison of Poisson, negative binomial, and Conway-Maxwell-Poisson models

Item-response models from the psychometric literature have been proposed for the estimation of researcher capacity. Canonical items that can be incorporated in such models to reflect researcher performance are count data (e.g., number of publications, number of citations). Count data can be modeled by Rasch’s Poisson counts model that assumes equidispersion (i.e., mean and variance must coincide). However, the mean can be larger as compared to the variance (i.e., underdispersion), or b) smaller as compared to the variance (i.e., overdispersion). Ignoring the presence of overdispersion (underdispersion) can cause standard errors to be liberal (conservative), when the Poisson model is used. Indeed, number of publications or number of citations are known to display overdispersion. Underdispersion, however, is far less acknowledged in the literature. In the current investigation the flexible Conway-Maxwell-Poisson count model is used to examine reliability estimates of capacity in relation to various dispersion patterns. It is shown, that reliability of capacity estimates of inventors drops from .84 (Poisson) to .68 (Conway-Maxwell-Poisson) or .69 (negative binomial). Moreover, with some items displaying overdispersion and some items displaying underdispersion, the dispersion pattern in a reanalysis of Mutz and Daniel’s (2018b) researcher data was found to be more complex as compared to previous results. To conclude, a careful examination of competing models including the Conway-Maxwell-Poisson count model should be undertaken prior to any evaluation and interpretation of capacity reliability. Moreover, this work shows that count data psychometric models are well suited for decisions with a focus on top researchers, because conditional reliability estimates (i.e., reliability depending on the level of capacity) were highest for the best researchers.

Gênese, Impacto e Identidade da Revista de Administração Contemporânea

RESUMO Contexto: como as pessoas e as organizações, os periódicos também apresentam uma identidade. Assim, pensar na identidade de um periódico científico remete, em primeiro lugar, a compreender como sua trajetória moldou suas preferências acerca do que seus membros entendem sobre ciência e academia. Em segundo lugar, remete a como ele se projeta na comunidade científica, tanto em termos de regras de julgamento sobre o que é considerado válido como pesquisa quanto da intensidade e do modo como ele impacta o conhecimento científico e a realidade social. Objetivo: diante desse contexto, buscamos, neste artigo, recuperar elementos distintivos da Revista de Administração Contemporânea (RAC) em sua gênese, ressaltando como tal período deixou uma impressão duradoura em sua identidade. Método: utilizamos textos históricos e depoimentos para embasar nossos argumentos, triangulando o material qualitativo com padrões de citação, de referência e de escrita científica para evidenciar a identidade e o impacto da RAC. Resultados: as análises apontam que a RAC, desde sua gênese, diferenciou-se pelo rigor teórico, metodológico e empírico. Isso refletiu, ao longo do tempo, em seu impacto e centralidade na comunidade acadêmica. Conclusão: defendemos que a eficácia na construção da identidade da RAC a levou a ser o periódico mais influente da área de administração.

Measuring researchers' potential scholarly impact with structural variations: Four types of researchers in information science (1979-2018)

We propose a method to measure the potential scholarly impact of researchers based on network structural variations they introduced to the underlying author co-citation network of their field. We applied the method to the information science field based on 91,978 papers published between 1979 and 2018 from the Web of Science. We divided the entire period into eight consecutive intervals and measured structural variation change rates (ΔM) of individual authors in corresponding author co-citation networks. Four types of researchers are identified in terms of temporal dynamics of their potential scholarly impact—1) Increasing, 2) Decreasing, 3) Sustained, and 4) Transient. The study contributes to the understanding of how researchers’ scholarly impact might evolve in a broad context of the corresponding research community. Specifically, this study illustrated a crucial role played by structural variation metrics in measuring and explaining the potential scholarly impact of a researcher. This method based on the structural variation analysis offers a theoretical framework and a practical platform to analyze the potential scholarly impact of researchers and their specific contributions.

Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005-2019

Despite growing demand for practicable methods of research evaluation, the use of bibliometric indicators remains controversial. This paper examines performance assessment practice in Europe-first, identifying the most commonly used bibliometric methods and, second, identifying the actors who have defined wide-spread practices. The framework of this investigation is Abbott's theory of professions, and I argue that indicator-based research assessment constitutes a potential jurisdiction for both individual experts and expert organizations. This investigation was conducted using a search methodology that yielded 138 evaluation studies from 21 EU countries, covering the period 2005 to 2019. Structured content analysis revealed the following findings: (1) Bibliometric research assessment is most frequently performed in the Nordic countries, the Netherlands, Italy, and the United Kingdom. (2) The Web of Science (WoS) is the dominant database used for public research assessment in Europe. (3) Expert organizations invest in the improvement of WoS citation data, and set technical standards with regards to data quality. (4) Citation impact is most frequently assessed with reference to international scientific fields. (5) The WoS classification of science fields retained its function as a de facto reference standard for research performance assessment. A detailed comparison of assessment practices between five dedicated organizations and other individual bibliometric experts suggests that corporate ownership and limited access to the most widely used citation databases have had a restraining effect on the development and diffusion of professional bibliometric methods during this period.

Envisioning the scientific paper of the future

Consider for a moment the rate of advancement in the scientific understanding of DNA. It is formidable; from Fredrich Miescher’s nuclein extraction in the 1860s to Rosalind Franklin’s double helix X-ray in the 1950s to revolutionary next-generation sequencing in the late 2000s. Now consider the scientific paper, the medium used to describe and publish these advances. How is the scientific paper advancing to meet the needs of those who generate and use scientific information? We review four essential qualities for the scientific paper of the future: ( i) a robust source of trustworthy information that remains peer reviewed and is ( ii) communicated to diverse users in diverse ways, ( iii) open access, and ( iv) has a measurable impact beyond Impact Factor. Since its inception, scientific literature has proliferated. We discuss the continuation and expansion of practices already in place including: freely accessible data and analytical code, living research and reviews, changes to peer review to improve representation of under-represented groups, plain language summaries, preprint servers, evidence-informed decision-making, and altmetrics.

Are Contributions from Chinese Physicists Undercited?

Purpose

In this work, we want to examine whether or not there are some scientific fields to which contributions from Chinese scholars have been under or over cited.

Design/methodology/approach

We do so by comparing the number of received citations and the IOF of publications in each scientific field from each country. The IOF is calculated from applying the modified closed system input–output analysis (MCSIOA) to the citation network. MCSIOA is a PageRank-like algorithm which means here that citations from the more influential subfields are weighted more towards the IOF.

Findings

About 40% of subfields in physics in China are undercited, meaning that their net influence ranks are higher (better) than the direct rank, while about 75% of subfields in the USA and German are undercited.

Research limitations

Only APS data is analyzed in this work. The expected citation influence is assumed to be represented by the IOF, and this can be wrong.

Practical implications

MCSIOA provides a measure of net influences and according to that measure. Overall, Chinese physicists’ publications are more likely overcited rather than being undercited.

Originality/value

The issue of under or over cited has been analyzed in this work using MCSIOA.

Research on impact evaluation of academic journals from multidimensional perspective

Purpose

The purpose of this paper is to introduce altmetric indicators and combine with traditional citation indicators to comprehensively evaluate the impact of academic journals from the perspective of multidimensional and multi-indicator fusion.

Design/methodology/approach

The authors take international multidisciplinary journals as an example, combining 14 traditional citation indicators of academic journals and introducing 14 altmetric indicators to build a comprehensive evaluation model of the impact of academic journals (academic impact and societal impact). At the same time, the authors systematically construct a journal evaluation indicator system from three dimensions. Then, the indicators data of three dimensions are evaluated by normalized processing, correlation analysis, reliability and validity analysis, PCA and factor analysis.

Findings

Two-dimensional and three-dimensional analyses can exactly provide some useful information for academic journals’ location in the respective coordinate systems. There are strong positive correlations among the measured indicators in the three dimensions, and each indicator has a significant consistency between whole and internal. The correlation coefficient between FD1 and FD2 is 0.888 with a strong positive correlation. It shows that the traditional citation indicators provided by WoS and Scopus database are highly consistent, and they are comparable and alternative in evaluating the academic impact of journals. The correlation coefficients of FD1, FD2 with FD3 are 0.831 and 0.798. There are strong positive correlations among them, which indicate that the evaluation of journals’ societal impact based on altmetrics indicator can be considered as a potential supplement to academic impact evaluation based on citation and to reflect the multidimensional nature of journals impact in an immediate way.

Originality/value

Multidimensional and multi-indicator perspective evaluation can provide references for the selection of impact evaluation indicators and model optimization of academic journals, and also provide new ideas for improving the status of the impact evaluation of academic journals.

Comprehensive Researcher Achievement Model (CRAM): a framework for measuring researcher achievement, impact and influence derived from a systematic literature review of metrics and models

OBJECTIVES

Effective researcher assessment is key to decisions about funding allocations, promotion and tenure. We aimed to identify what is known about methods for assessing researcher achievements, leading to a new composite assessment model.

DESIGN

We systematically reviewed the literature via the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols framework.

DATA SOURCES

All Web of Science databases (including Core Collection, MEDLINE and BIOSIS Citation Index) to the end of 2017. ELIGIBILITY CRITERIA: (1) English language, (2) published in the last 10 years (2007-2017), (3) full text was available and (4) the article discussed an approach to the assessment of an individual researcher's achievements.

DATA EXTRACTION AND SYNTHESIS

Articles were allocated among four pairs of reviewers for screening, with each pair randomly assigned 5% of their allocation to review concurrently against inclusion criteria. Inter-rater reliability was assessed using Cohen's Kappa (ĸ). The ĸ statistic showed agreement ranging from moderate to almost perfect (0.4848-0.9039). Following screening, selected articles underwent full-text review and bias was assessed.

RESULTS

Four hundred and seventy-eight articles were included in the final review. Established approaches developed prior to our inclusion period (eg, citations and outputs, h-index and journal impact factor) remained dominant in the literature and in practice. New bibliometric methods and models emerged in the last 10 years including: measures based on PageRank algorithms or 'altmetric' data, methods to apply peer judgement and techniques to assign values to publication quantity and quality. Each assessment method tended to prioritise certain aspects of achievement over others.

CONCLUSIONS

All metrics and models focus on an element or elements at the expense of others. A new composite design, the Comprehensive Researcher Achievement Model (CRAM), is presented, which supersedes past anachronistic models. The CRAM is modifiable to a range of applications.

Bibliometric differences – a case study in bibliometric evaluation across SSH and STEM

Purpose

The purpose of this paper is to acknowledge that there are bibliometric differences between Social Sciences and Humanities (SSH) vs Science, Technology, Engineering and Mathematics (STEM). It is not so that either SSH or STEM has the right way of doing research or working as a scholarly community. Accordingly, research evaluation is not done properly in one framework based on either a method from SSH or STEM. However, performing research evaluation in two separate frameworks also has disadvantages. One way of scholarly practice may be favored unintentionally in evaluations and in research profiling, which is necessary for job and grant applications.

Design/methodology/approach

In the case study, the authors propose a tool where it may be possible, on one hand, to evaluate across disciplines and on the other hand to keep the multifaceted perspective on the disciplines. Case data describe professors at an SSH and a STEM department at Aalborg University. Ten partial indicators are compiled to build a performance web – a multidimensional description – and a one-dimensional ranking of professors at the two departments. The partial indicators are selected in a way that they should cover a broad variety of scholarly practice and differences in data availability.

Findings

A tool which can be used both for a one-dimensional ranking of researchers and for a multidimensional description is described in the paper.

Research limitations/implications

Limitations of the study are that panel-based evaluation is left out and that the number of partial indicators is set to 10.

Originality/value

The paper describes a new tool that may be an inspiration for practitioners in research analytics.

How Efficiently Do Elite US Universities Produce Highly Cited Papers?

While output and impact assessments were initially at the forefront of institutional research evaluations, efficiency measurements have become popular in recent years. Research efficiency is measured by indicators that relate research output to input. The additional consideration of research input in research evaluation is obvious, since the output depends on the input. The present study is based on a comprehensive dataset with input and output data for 50 US universities. As input, we used research expenses, and as output the number of highly-cited papers. We employed Data Efficiency Analysis (DEA), Free Disposal Hull (FDH) and two more robust models: the order-m and order-α approaches. The results of the DEA and FDH analysis show that Harvard University and Boston College can be called especially efficient compared to the other universities. While the strength of Harvard University lies in its high output of highly-cited papers, the strength of Boston College is its small input. In the order-α and order-m frameworks, Harvard University remains efficient, but Boston College becomes super-efficient. We produced university rankings based on adjusted efficiency scores (subsequent to regression analyses), in which single covariates (e.g., the disciplinary profile) are held constant.

[Determinants of patents in the life sciences and health sciences at federal universities in Minas Gerais State, Brazil: an analysis of panel data for 1995-2016]

In the 1990s, the development of microelectronics and communication technologies spawned the information technology paradigm, characterized by an intensity of information which, transformed into knowledge and innovation, plays a crucial role for development. In order for this to occur, countries need to build appropriate institutional arrangements. This led to the Triple Helix model for systems innovation, which views innovation and technical progress in a Knowledge Economy as the result of a complex set of relations between universities, companies, and government. In this context, the current study aims to identify the impact of knowledge production and public policies for the production of patent applications by life sciences and health sciences researchers affiliated with federal universities in the State of Minas Gerais, Brazil. The analysis covered the years 1995 to 2016 and used econometric techniques in panel data. According to the findings, educational factors (thesis supervision), economic factors (technological products and the provision of technological services), and political factors (legal framework, financial resources, and incentive policies) impact the production of patent applications by the selected researchers. The results reveal efforts by government, although incipient, in the attempt to promote a technological leap.

Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016

The use of bibliometric measures in the evaluation of research has increased considerably based on expertise from the growing research field of evaluative citation analysis (ECA). However, mounting criticism of such metrics suggests that the professionalization of bibliometric expertise remains contested. This paper investigates why impact metrics, such as the journal impact factor and the h-index, proliferate even though their legitimacy as a means of professional research assessment is questioned. Our analysis is informed by two relevant sociological theories: Andrew Abbott's theory of professions and Richard Whitley's theory of scientific work. These complementary concepts are connected in order to demonstrate that ECA has failed so far to provide scientific authority for professional research assessment. This argument is based on an empirical investigation of the extent of reputational control in the relevant research area. Using three measures of reputational control that are computed from longitudinal inter-organizational networks in ECA (1972-2016), we show that peripheral and isolated actors contribute the same number of novel bibliometric indicators as central actors. In addition, the share of newcomers to the academic sector has remained high. These findings demonstrate that recent methodological debates in ECA have not been accompanied by the formation of an intellectual field in the sociological sense of a reputational organization. Therefore, we conclude that a growing gap exists between an academic sector with little capacity for collective action and increasing demand for routine performance assessment by research organizations and funding agencies. This gap has been filled by database providers. By selecting and distributing research metrics, these commercial providers have gained a powerful role in defining de-facto standards of research excellence without being challenged by expert authority.

Researcher and Author Impact Metrics: Variety, Value, and Context

Numerous quantitative indicators are currently available for evaluating research productivity. No single metric is suitable for comprehensive evaluation of the author-level impact. The choice of particular metrics depends on the purpose and context of the evaluation. The aim of this article is to overview some of the widely employed author impact metrics and highlight perspectives of their optimal use. The h-index is one of the most popular metrics for research evaluation, which is easy to calculate and understandable for non-experts. It is automatically displayed on researcher and author profiles on citation databases such as Scopus and Web of Science. Its main advantage relates to the combined approach to the quantification of publication and citation counts. This index is increasingly cited globally. Being an appropriate indicator of publication and citation activity of highly productive and successfully promoted authors, the h-index has been criticized primarily for disadvantaging early career researchers and authors with a few indexed publications. Numerous variants of the index have been proposed to overcome its limitations. Alternative metrics have also emerged to highlight 'societal impact.' However, each of these traditional and alternative metrics has its own drawbacks, necessitating careful analyses of the context of social attention and value of publication and citation sets. Perspectives of the optimal use of researcher and author metrics is dependent on evaluation purposes and compounded by information sourced from various global, national, and specialist bibliographic databases.

Identifying landmark publications in the long run using field-normalized citation data

Purpose

The purpose of this paper is to propose an approach for identifying landmark papers in the long run. These publications reach a very high level of citation impact and are able to remain on this level across many citing years. In recent years, several studies have been published which deal with the citation history of publications and try to identify landmark publications.

Design/methodology/approach

In contrast to other studies published hitherto, this study is based on a broad data set with papers published between 1980 and 1990 for identifying the landmark papers. The authors analyzed the citation histories of about five million papers across 25 years.

Findings

The results of this study reveal that 1,013 papers (less than 0.02 percent) are “outstandingly cited” in the long run. The cluster analyses of the papers show that they received the high impact level very soon after publication and remained on this level over decades. Only a slight impact decline is visible over the years.

Originality/value

For practical reasons, approaches for identifying landmark papers should be as simple as possible. The approach proposed in this study is based on standard methods in bibliometrics.

Making visible the invisible through the analysis of acknowledgements in the humanities

Purpose

Science is subject to a normative structure that includes how the contributions and interactions between scientists are rewarded. Authorship and citations have been the key elements within the reward system of science, whereas acknowledgements, despite being a well-established element in scholarly communication, have not received the same attention. The purpose of this paper is to put forward the bearing of acknowledgements in the humanities to bring to the foreground contributions and interactions that, otherwise, would remain invisible through traditional indicators of research performance.

Design/methodology/approach

The study provides a comprehensive framework to understanding acknowledgements as part of the reward system with a special focus on their value in the humanities as a reflection of intellectual indebtedness. The distinctive features of research in the humanities are outlined and the role of acknowledgements as a source of contributorship information is reviewed to support these assumptions.

Findings

“Peer interactive communication” is the prevailing support thanked in the acknowledgements of humanities, so the notion of acknowledgements as “super-citations” can make special sense in this area. Since single-authored papers still predominate as publishing pattern in this domain, the study of acknowledgements might help to understand social interactions and intellectual influences that lie behind a piece of research and are not visible through authorship.

Originality/value

Previous works have proposed and explored the prevailing acknowledgement types by domain. This paper focusses on the humanities to show the role of acknowledgements within the reward system and highlight publication patterns and inherent research features which make acknowledgements particularly interesting in the area as a reflection of the socio-cognitive structure of research.

Metric-based vs peer-reviewed evaluation of a research output: Lesson learnt from UK's national research assessment exercise

Purpose

There is a general inquisition regarding the monetary value of a research output, as a substantial amount of funding in modern academia is essentially awarded to good research presented in the form of journal articles, conferences papers, performances, compositions, exhibitions, books and book chapters etc., which, eventually leads to another question if the value varies across different disciplines. Answers to these questions will not only assist academics and researchers, but will also help higher education institutions (HEIs) make informed decisions in their administrative and research policies.

Design and methodology

To examine both the questions, we applied the United Kingdom’s recently concluded national research assessment exercise known as the Research Excellence Framework (REF) 2014 as a case study. All the data for this study is sourced from the openly available publications which arose from the digital repositories of REF’s results and HEFCE’s funding allocations.

Findings

A world leading output earns between £7504 and £14,639 per year within the REF cycle, whereas an internationally excellent output earns between £1876 and £3659, varying according to their area of research. Secondly, an investigation into the impact rating of 25315 journal articles submitted in five areas of research by UK HEIs and their awarded funding revealed a linear relationship between the percentage of quartile-one journal publications and percentage of 4* outputs in Clinical Medicine, Physics and Psychology/Psychiatry/Neuroscience UoAs, and no relationship was found in the Classics and Anthropology/Development Studies UoAs, due to the fact that most publications in the latter two disciplines are not journal articles.

Practical implications

The findings provide an indication of the monetary value of a research output, from the perspectives of government funding for research, and also what makes a good output, i.e. whether a relationship exists between good quality output and the source of its publication. The findings may also influence future REF submission strategies in HEIs and ascertain that the impact rating of the journals is not necessarily a reflection of the quality of research in every discipline, and this may have a significant influence on the future of scholarly communications in general.

Originality

According to the author’s knowledge, this is the first time an investigation has estimated the monetary value of a good research output.

Impacto y calidad de la productividad académica de los investigadores en Colombia en neurociencia comportamental utilizando modelos animales

La productividad científica colombiana es limitada considerando indicadores como el número de artículos publicados en revistas de alto impacto y el índice h. La ausencia de una línea de base y de metodologías de comparación de productividad dificulta trazar las expectativas de publicación provenientes de las instancias nacionales. El objetivo del presente estudio es el de evaluar de manera comparativa aspectos relacionados con el impacto y la calidad de la productividad académica de los investigadores en neurociencia comportamental utilizando modelos animales en Colombia. Se encontró que aunque los investigadores en Colombia publican de manera significativa (0.9 artículos por año), están rezagados en todos los aspectos de productividad con relación a investigadores internacionales. Se espera que la metodología propuesta pueda extenderse a la evaluación del impacto y calidad de la productividad en temáticas específicas de otras áreas de investigación.

How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data

In this short communication, we provide an overview of a relatively newly provided source of altmetrics data which could possibly be used for societal impact measurements in scientometrics. Recently, Altmetric—a start-up providing publication level metrics—started to make data for publications available which have been mentioned in policy-related documents. Using data from Altmetric, we study how many papers indexed in the Web of Science (WoS) are mentioned in policy-related documents. We find that less than 0.5% of the papers published in different subject categories are mentioned at least once in policy-related documents. Based on our results, we recommend that the analysis of (WoS) publications with at least one policy-related mention is repeated regularly (annually) in order to check the usefulness of the data. Mentions in policy-related documents should not be used for impact measurement until new policy-related sites are tracked.