Ranking games

Evaluation and Comparison of Academic Impact and Disruptive Innovation Level of Medical Journals: Bibliometric Analysis and Disruptive Evaluation

BACKGROUND

As an important platform for researchers to present their academic findings, medical journals have a close relationship between their evaluation orientation and the value orientation of their published research results. However, the differences between the academic impact and level of disruptive innovation of medical journals have not been examined by any study yet.

OBJECTIVE

This study aims to compare the relationships and differences between the academic impact, disruptive innovation levels, and peer review results of medical journals and published research papers. We also analyzed the similarities and differences in the impact evaluations, disruptive innovations, and peer reviews for different types of medical research papers and the underlying reasons.

METHODS

The general and internal medicine Science Citation Index Expanded (SCIE) journals in 2018 were chosen as the study object to explore the differences in the academic impact and level of disruptive innovation of medical journals based on the OpenCitations Index of PubMed open PMID-to-PMID citations (POCI) and H1Connect databases, respectively, and we compared them with the results of peer review.

RESULTS

First, the correlation coefficients of the Journal Disruption Index (JDI) with the Journal Cumulative Citation for 5 years (JCC5), Journal Impact Factor (JIF), and Journal Citation Indicator (JCI) were 0.677, 0.585, and 0.621, respectively. The correlation coefficient of the absolute disruption index (Dz) with the Cumulative Citation for 5 years (CC5) was 0.635. However, the average difference in the disruptive innovation and academic influence rankings of journals reached 20 places (about 17.5%). The average difference in the disruptive innovation and influence rankings of research papers reached about 2700 places (about 17.7%). The differences reflect the essential difference between the two evaluation systems. Second, the top 7 journals selected based on JDI, JCC5, JIF, and JCI were the same, and all of them were H-journals. Although 8 (8/15, 53%), 96 (96/150, 64%), and 880 (880/1500, 58.67%) of the top 0.1%, top 1%, and top 10% papers selected based on Dz and CC5, respectively, were the same. Third, research papers with the "changes clinical practice" tag showed only moderate innovation (4.96) and impact (241.67) levels but had high levels of peer-reviewed recognition (6.00) and attention (2.83).

CONCLUSIONS

The results of the study show that research evaluation based on innovative indicators is detached from the traditional impact evaluation system. The 3 evaluation systems (impact evaluation, disruptive innovation evaluation, and peer review) only have high consistency for authoritative journals and top papers. Neither a single impact indicator nor an innovative indicator can directly reflect the impact of medical research for clinical practice. How to establish an integrated, comprehensive, scientific, and reasonable journal evaluation system to improve the existing evaluation system of medical journals still needs further research.

Emerging Scholars in Academia: An Analysis of the Impact of IAPSS Politikon in the Academic Careers of Its Authors

This paper analyzes students, recent graduates, and emerging scholars’ involvement in academic publishing, specifically by studying the impact of their publications in Politikon, the International Association for Political Science Students’ Journal, and their academic careers. The results of a standardized self-administered survey serve to assess authors’ motivations and impact of publish ing with IAPSS Politikon. The data show that publishing with Politikon has helped emerging researchers get more experience concerning the publication process and has improved their research, writing, and analyzing skills. Additionally, an essential part of them said they increased their educational background or obtained a new higher position after publishing with Politikon. In fact, 50 per cent of these scholars said they were promoted after publishing with the journal. Furthermore, Politikon, establishes a platform for Global South scholarship to be at the forefront of Political Science/International Relations debates and knowledge production. This implies the importance of Politikon in the early career of scholars by giving them the right tools to develop professionally and by reaching scholars from all around the world, especially from the Global South in an effort to contribute to global international relations and global governance reform.

China's Research Evaluation Reform: What are the Consequences for Global Science?

In the 1990s, China created a research evaluation system based on publications indexed in the Science Citation Index (SCI) and on the Journal Impact Factor. Such system helped the country become the largest contributor to the scientific literature and increased the position of Chinese universities in international rankings. Although the system had been criticized by many because of its adverse effects, the policy reform for research evaluation crawled until the breakout of the COVID-19 pandemic, which accidently accelerates the process of policy reform. This paper highlights the background and principles of this reform, provides evidence of its effects, and discusses the implications for global science.

Cumulative advantage and citation performance of repeat authors in scholarly journals

Cumulative advantage–commonly known as the Matthew Effect–influences academic output and careers. Given the challenge and uncertainty of gauging the quality of academic research, gatekeepers often possess incentives to prefer the work of established academics. Such preferences breach scientific norms of universalism and can stifle innovation. This article analyzes repeat authors within academic journals as a possible exemplar of the Matthew Effect. Using publication data for 347 economics journals from 1980–2017, as well as from three major generalist science journals, we analyze how articles written by repeat authors fare vis-à-vis less-experienced authors. Results show that articles written by repeat authors steadily decline in citation impact with each additional repeat authorship. Despite these declines, repeat authors also tend to garner more citations than debut authors. These contrasting results suggest both benefits and drawbacks associated with repeat authorships. Journals appear to respond to feedback from previous publications, as more-cited authors in a journal are more likely to be selected for repeat authorships. Institutional characteristics of journals also affect the likelihood of repeat authorship, as well as citation outcomes. Repeat authorships–particularly in leading academic journals–reflect innovative incentives and professional reward structures, while also influencing the intellectual content of science.

Trust or attention? Medialization of science revisited

The article traces the intensifying media orientation of universities and research organizations first by referring to early diagnoses of the spread of mutual observation and attention seeking as defining societies after WWII. This development provides the background for the unlikely, yet massive turn of scientific organizations to the general public, the media and more recently social media. Details are analyzed on the interactional, organizational and systems levels, and are followed with a focus on the reasons motivating universities. A closer look reveals the self-referentiality of institutional communication deriving its rationale from 'imagined publics'. The politically sponsored 'engagement of the public' has been derailed to become marketing, branding and public relations exercises. The unintended consequences of the establishment of communication units and the blurring of science communication and persuasion are conflicts between faculty and management and possibly a loss of trust in science.

Measuring the excellence contribution at the journal level: an alternative to Garfield’s impact factor

The aim of this study is to analyze to which extent the Journal Impact Factor (JIF) reflects the amount of excellent publications contained in a journal in the corresponding subject category. We are introducing two percentile-based indicators in order to measure the excellence contribution at journal level. Calculations of these indicators have been carried out for twenty different Journal Citation Reports (JCR) subject categories to investigate the correlation with Garfield’s Journal Impact Factor. Differences in the ranking according to all three indicators especially in First Quartile (Q1) of each category are shown and discussed. We have also studied the effect of multidisciplinary journals to the excellence contribution at category level and observed considerable differences between the twenty considered categories under analysis. Their omission would lead to neglect a large part of excellent publications, especially in the hard sciences. The introduced excellence indicators are very robust considering the types of documents considered for their calculation. The results of this study show that the journal excellence content is not completely reflected in the JIF measure and affects both Science and Social Science Edition categories. Therefore, the introduction of journal excellence indicators will help to give a more complete picture of the citation impact of a journal, because they are informing directly about the total and normalized excellence contribution of each journal to the corresponding subject category. They are a valuable enrichment and complement of the JIF, and more suitable than the Journal Citation Indicator, recently added to the JCR, because they reflect the inherent skewness of the citation distribution.

Publishing patterns and citation performance of manuscripts relating to paediatric cardiology and congenital heart disease: comparison of paediatric and adult cardiology journals

BACKGROUND

Manuscripts pertaining to paediatric cardiology and CHD have been published in a variety of different journals. Some of these journals are journals dedicated to paediatric cardiology, while others are focused on adult cardiology. Historically, it has been considered that manuscripts published in journals devoted to adult cardiology have greater citation potential. Our objective was to compare citation performance between manuscripts related to paediatric cardiology and CHD published in paediatric as opposed to adult cardiology journals.

METHODS

We identified manuscripts related to paediatric cardiology and CHD published in five journals of interest during 2014. Of these journals, two were primarily concerned with adult cardiology, while the other three focused on paediatric cardiology. The number of citations for these identified manuscripts was gathered from Google Scholar. We compared the number of citations (median, mean, and 25th, 75th, 90th, and 95th percentiles), the potential for citation, and the h-index for the identified manuscripts.

RESULTS

We identified a total of 828 manuscripts related to paediatric cardiology and congenital heart as published in the 5 journals during 2014. Of these, 783 (95%) were published in journals focused on paediatric cardiology, and the remaining 45 (5%) were published in journals focused on adult cardiology. The median number of citations was 41 in the manuscripts published in the journals focused on adult cardiology, as opposed to 7 in journals focused on paediatric cardiology (p < 0.001). The h-index, however, was greater for the journals dedicated to paediatric cardiology (36 versus 27).

CONCLUSION

Approximately one-twentieth of the work relating to paediatric cardiology and CHD is published in journals that focus predominantly on adult cardiology. The median number of citations is greater when manuscripts concerning paediatric cardiology and CHD are published in these journals focused on adult cardiology. The h-index, however, is higher when the manuscripts are published in journals dedicated to paediatric cardiology. While such publications in journals that focus on adult cardiology tend to generate a greater number of citations than those achieved for works published in specialised paediatric cardiology journals, the potential for citation is no different between the journals. Due to the drastically lower number of manuscripts published in journals dedicated to adult cardiology, however, median performance is different.

Is Bauman's "liquid modernity" influencing the way we are doing science?

This commentary analyzes the possible effects of lightness—a typical attribute of modern (liquid) society, according to Bauman—on the way we are doing science. We share our opinion in an attempt to discern whether some unwanted practices that may affect our scientific results (such as technology misuse, bonus rewards, publishing under pressure, or indolence for getting accurate results) can be attributed, at least partially, to the liquid characteristic of modern society. We also examine whether the different systems that support science favor these actions, conspiring against what should be the primary goal of science: the search for truth. We finally consider several aspects that should be taken into account to rescue science from the intrusion of weightless actions.

Investigating the division of scientific labor using the Contributor Roles Taxonomy (CRediT)

Contributorship statements were introduced by scholarly journals in the late 1990s to provide more details on the specific contributions made by authors to research papers. After more than a decade of idiosyncratic taxonomies by journals, a partnership between medical journals and standards organizations has led to the establishment, in 2015, of the Contributor Roles Taxonomy (CRediT), which provides a standardized set of 14 research contributions. Using the data from Public Library of Science (PLOS) journals over the 2017–2018 period (N = 30,054 papers), this paper analyzes how research contributions are divided across research teams, focusing on the association between division of labor and number of authors, and authors’ position and specific contributions. It also assesses whether some contributions are more likely to be performed in conjunction with others and examines how the new taxonomy provides greater insight into the gendered nature of labor division. The paper concludes with a discussion of results with respect to current issues in research evaluation, science policy, and responsible research practices.

How the publish-or-perish principle divides a science: the case of economists

The publish-or-perish principle has become a fact of academic life in gaining a position or being promoted. Evidence is mounting that benefits of this pressure is being countered by the downsides, like forms of goal displacement by scientists or unethical practices. In this paper we evaluate whether perceived work pressure (publishing, acquisition funds, teaching, administration) is associated with different attitudes towards science and the workplace among economists working at Dutch universities. Publication pressure is high and is related to faculty position and university ranking position. Based on a latent class analysis we can detect a clear divide among economists. Around two third of the economists perceives that this pressure has upsides as well as serious downsides and one third only perceives upsides and no downsides. Full professors see more than other faculty members the positive sides of the publish-or-perish principle and virtually no downsides. These different perceptions are also reflected in their appreciation of the academic work environment.

Scholars as government-appointed research evaluators: Do they create congruence between their professional quality standards and political demands?

All across the globe politically initiated research evaluations are based on “informed peer review” procedures. Scholars are appointed as evaluators and can apply self-defined quality standards in order to overcome shortcomings of standardized measures. Even though there are no binding criteria in these procedures and the quality standards of the scholars' disciplines vary, studies suggest that scholars, in their role as government-appointed research evaluators, assess research uniformly.By drawing on a small-N investigation, this study compares the quality standards scholars apply as government-appointed research evaluators with quality standards they follow as researchers. The study points to a paradox: Criteria scholars refer to while describing the excellence of their own research and criteria they use as evaluators differ and contradict each other. The results are discussed from different angles.

Proposed methodology for measuring the effectiveness of policies designed to further research

The effects of economic incentives on research have been widely debated in the literature. Some authors deem them to have no or even an adverse impact, particularly, if they are perceived as irrelevant to or an attempt to control researcher activity, whilst others believe they enhance research productivity by inducing new habits such as international collaboration or publication in high-impact journals. In 2007, the Carlos III University of Madrid introduced bonuses associated with research merits to reward research and educational excellence. The policy aims to enhance the quantity and quality of the institution’s scholarly publications. This study analyses whether and to what extent the new policy has had a potential effect on scientific output, impact, and visibility. Scientific activity indicators between 1991 and 2018 were analysed and a state-space model was used to establish possible scenarios (pre- and post-bonus periods) and 3 year predictions. Further to the findings, despite weak growth in researcher staff size, the number of papers rose during the period slightly more than in the pre-bonus simulation. The number of first-quartile papers also rose substantially, attesting to higher impact and visibility of the university’s research. Greater internationalization was also observed. The incentives were found to be highly suggestive of a change in university researchers’ publication habits.

Should Google Scholar be used for benchmarking against the professoriate in education?

In the neoliberal environment of contemporary academia, an individual’s research rankings and outputs can shape their career security and progression. When applying for ongoing employment and promotional opportunities, academics may benchmark their performance against that of superior colleagues to demonstrate their performance in relation to their discipline. The H-index and citation rates are commonly used to quantify the value of an academic’s work, and they can be used comparatively for benchmarking purposes. The focus of this paper is to critically consider if Google Scholar be used for benchmarking against the professoriate in education, by weighting up issues of data reliability and participation. The Google Scholar profiles of full professors at top ranked universities in Australia, the United Kingdom and the United States of America are analysed to explore how widespread Google Scholar use is in the education professoriate. Quartiles of impact are established in relation to H-index, with exploration of how gender is distributed across these quartiles. Limitations of using Google Scholar data are highlighted through a taxonomy of quality confounders, and the utility of Google Scholar as a legitimate tool for benchmarking against the professoriate in education is strongly challenged. As metrics continue to rise in their importance for academics’ job security and promotional prospects, reliance on metrics of dubious quality and uneven participation must be questioned.

The Hong Kong Principles for assessing researchers: Fostering research integrity

For knowledge to benefit research and society, it must be trustworthy. Trustworthy research is robust, rigorous, and transparent at all stages of design, execution, and reporting. Assessment of researchers still rarely includes considerations related to trustworthiness, rigor, and transparency. We have developed the Hong Kong Principles (HKPs) as part of the 6th World Conference on Research Integrity with a specific focus on the need to drive research improvement through ensuring that researchers are explicitly recognized and rewarded for behaviors that strengthen research integrity. We present five principles: responsible research practices; transparent reporting; open science (open research); valuing a diversity of types of research; and recognizing all contributions to research and scholarly activity. For each principle, we provide a rationale for its inclusion and provide examples where these principles are already being adopted.

The Hong Kong Principles for assessing researchers: Fostering research integrity

For knowledge to benefit research and society, it must be trustworthy. Trustworthy research is robust, rigorous, and transparent at all stages of design, execution, and reporting. Assessment of researchers still rarely includes considerations related to trustworthiness, rigor, and transparency. We have developed the Hong Kong Principles (HKPs) as part of the 6th World Conference on Research Integrity with a specific focus on the need to drive research improvement through ensuring that researchers are explicitly recognized and rewarded for behaviors that strengthen research integrity. We present five principles: responsible research practices; transparent reporting; open science (open research); valuing a diversity of types of research; and recognizing all contributions to research and scholarly activity. For each principle, we provide a rationale for its inclusion and provide examples where these principles are already being adopted.

Unbundling practice: the unbundling of big deal journal packages as an information practice

Purpose

The purpose of this article is to introduce a theoretical framework and approach for studying the evaluation and decision-making practices through which academic librarians attempt to reduce the cost of electronic journal subscriptions – an organizational practice known as the unbundling of big deal journal packages.

Design/methodology/approach

The article presents a literature-based conceptual analysis of several fields to delineate the elements of the practice of unbundling of big deal journal packages. Beyond analysing the prior literature, the discussion is supported by empirical findings from a pilot study on the topic conducted by two of the article's authors.

Findings

The main finding of the article is that the unbundling of big deal packages is a case of what sociologist refers to as decision-making in a social context. By reviewing previous studies, the article identifies the social and material elements constitutive of this practice. This, in turn, allows to develop questions and concepts for future research on the topic and to position it as an area of inquiry within the field of information behaviour/practices.

Originality/value

The article is the first attempt to conceptualize the unbundling of big deal journal packages by highlighting its phenomenological status as a type of information practice. In addition, the article proposes a research approach for studying this type of information practice by drawing on insights from the information behaviour/practice literature and enriching them through practice theory contributions in organizational studies and sociology.

[Scientific impact of the TraumaRegister DGU®]

BACKGROUND

The trauma registry of the German Trauma Society (TraumaRegister DGU®) is not only a tool for quality management but also for research purposes.

OBJECTIVE

Evaluation of the impact of the TraumaRegister DGU® on scientific output and patient treatment.

MATERIAL AND METHODS

Analysis of publications from the TraumaRegister DGU® with respect to numbers, impact factors, journals, citations and presentations.

RESULTS AND CONCLUSION

The number and impact factors of publications from the TraumaRegister DGU® rose steeply during the last 10 years and in the last 3 years consisted of 25 publications per year. More than two thirds of them were published in high quality international journals and reflect the great scientific importance. For the German speaking readership and the specific aspects of treatment of the severely injured relevant to Germany, the large number of German language articles are just as important. Independent of the impact factor publications in Deutsches Ärzteblatt, the journal with the highest circulation and Der Unfallchirurg play the most important role. A large amount of scientific information gained from the TraumaRegister DGU® has been included in treatment guidelines and structures. The register is a basic prerequisite for the TraumaNetzwerk DGU®. Since almost all severely injured patients in Germany are now included in the registry, it is possible to obtain epidemiologically reliable data of treatment and outcomes for these patient groups.

Edge Factors: Scientific Frontier Positions of Nations

A key decision in scientific work is whether to build on novel or well-established ideas. Because exploiting new ideas is often harder than more conventional science, novel work can be especially dependent on interactions with colleagues, the training environment, and ready access to potential collaborators. Location may thus influence the tendency to pursue work that is close to the edge of the scientific frontier in the sense that it builds on recent ideas. We calculate for each nation its position relative to the edge of the scientific frontier by measuring its propensity to build on relatively new ideas in biomedical research. Text analysis of 20+ million publications shows that the United States and South Korea have the highest tendencies for novel science. China has become a leader in favoring newer ideas when working with basic science ideas and research tools, but is still slow to adopt new clinical ideas. Many locations remain far behind the leaders in terms of their tendency to work with novel ideas, indicating that the world is far from flat in this regard.

Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016

The use of bibliometric measures in the evaluation of research has increased considerably based on expertise from the growing research field of evaluative citation analysis (ECA). However, mounting criticism of such metrics suggests that the professionalization of bibliometric expertise remains contested. This paper investigates why impact metrics, such as the journal impact factor and the h-index, proliferate even though their legitimacy as a means of professional research assessment is questioned. Our analysis is informed by two relevant sociological theories: Andrew Abbott's theory of professions and Richard Whitley's theory of scientific work. These complementary concepts are connected in order to demonstrate that ECA has failed so far to provide scientific authority for professional research assessment. This argument is based on an empirical investigation of the extent of reputational control in the relevant research area. Using three measures of reputational control that are computed from longitudinal inter-organizational networks in ECA (1972-2016), we show that peripheral and isolated actors contribute the same number of novel bibliometric indicators as central actors. In addition, the share of newcomers to the academic sector has remained high. These findings demonstrate that recent methodological debates in ECA have not been accompanied by the formation of an intellectual field in the sociological sense of a reputational organization. Therefore, we conclude that a growing gap exists between an academic sector with little capacity for collective action and increasing demand for routine performance assessment by research organizations and funding agencies. This gap has been filled by database providers. By selecting and distributing research metrics, these commercial providers have gained a powerful role in defining de-facto standards of research excellence without being challenged by expert authority.

Neophilia Ranking of Scientific Journals

The ranking of scientific journals is important because of the signal it sends to scientists about what is considered most vital for scientific progress. Existing ranking systems focus on measuring the influence of a scientific paper (citations)-these rankings do not reward journals for publishing innovative work that builds on new ideas. We propose an alternative ranking based on the proclivity of journals to publish papers that build on new ideas, and we implement this ranking via a text-based analysis of all published biomedical papers dating back to 1946. In addition, we compare our neophilia ranking to citation-based (impact factor) rankings; this comparison shows that the two ranking approaches are distinct. Prior theoretical work suggests an active role for our neophilia index in science policy. Absent an explicit incentive to pursue novel science, scientists underinvest in innovative work because of a coordination problem: for work on a new idea to flourish, many scientists must decide to adopt it in their work. Rankings that are based purely on influence thus do not provide sufficient incentives for publishing innovative work. By contrast, adoption of the neophilia index as part of journal-ranking procedures by funding agencies and university administrators would provide an explicit incentive for journals to publish innovative work and thus help solve the coordination problem by increasing scientists' incentives to pursue innovative work.

Academic inequality through the lens of community ecology: a meta-analysis

Ecological assemblages are generally characterized by a few dominant species and numerous others. Such unequal distributions of dominance also emerge in human society, including in scientific communities. Here, based on formal community ecological analyses, we show the temporal trends in the number of scientific publication in the discipline of "ecology." Based on this, we infer possible factors causing the imbalance of reputation and dominance among countries. We relied on 454 ecological meta-analysis papers published from 1998 to 2014, which sourced over 29,000 original publications. Formal meta-analyses are essential for synthesizing findings from individual studies and are critical for assessing issues and informing policy. We found that, despite the rapid expansion of outlets for ecology papers (analogous to an increase in carrying capacity, in ecological systems), country diversity as determined from first author affiliations (analogous to species diversity) did not increase. Furthermore, a country identity was more powerful than the popularity of the scientific topic and affected the chance of publication in high-profile journals, independent of the potential novelty of findings and arguments of the papers, suggesting possible academic injustice. Consequently, a rank order and hierarchy has been gradually formed among countries. Notably, this country-dominance rank is not only specific to this scientific domain but also universal across different societal situations including sports and economics, further emphasizing that inequality and hierarchical structure exist even in modern human society. Our study demonstrates a need for having robust frameworks to facilitate equality and diversity in the scientific domain in order to better inform society and policy.