On the stability of citation-based journal rankings

Assessing the publication output in the field of forensic science and legal medicine using Web of Science database from 2011 to 2020

The aim of this study was threefold. First, it analyzed the characteristics of the publication outputs for the Legal Medicine (LM) category using the Web of Science (WoS) database during 2011-2020. Second, it discussed the distribution of the papers for the 25 most productive countries/regions in terms of quality and quantity, such as the h-index and GDP per capita. Finally, it investigated the trend and temporal stability of the journal impact factor (JIF) and determined the percentage of the journal self-citations. The findings suggested that the number of papers, the average number of pages of the papers, the average number of cited references in the papers, the average number of authors per paper, the percentage of open access papers, as well as international and domestic collaboration tended to increase regularly. However, the productivity was limited when compared to the whole WoS database, since there was no significant change in the number of the journals. The countries/regions with the highest number of publications were not those that made the most impact in terms of the widespread impact of the publications. The level of international cooperation and the funding for the research had dramatic impact on the visibility of papers. The average JIF has increased significantly while the journal self-citation rates have decreased in a similar way. The journals have had very stable (have not fluctuated) impact factors over time. During the period studied, the journals with the higher impact factors (Q1, Q2) published many more papers than journals with the lower impact factors (Q3, Q4).

Improved PageRank and New Indices for Academic Impact Evaluation Using AI Papers as Case Studies

Evaluating academic papers and groups is important in scholar evaluation and literature retrieval. However, current evaluation indices, which pay excessive attention to the citation number rather than the citation importance and unidirectionality, are relatively simple. This study proposes new evaluation indices for papers and groups. First, an improved PageRank (PR) algorithm introducing citation importance is proposed to obtain a new citation-based paper index (CPI) via a pre-ranking and fine-tuning strategy. Second, to evaluate the paper’s influence inside and outside its research field, the focus citation-based paper index (FCPI) and diversity citation-based paper index (DCPI) are proposed based on topic similarity and diversity, respectively. Third, aside from the statistical indices for academic papers, we propose a foreign academic degree of dependence (FAD) to characterise the dependence between two academic groups. Finally, artificial intelligence (AI) papers from 2005 to 2019 are utilised for a case study.

Ranking of addiction journals in eight widely used impact metrics

Background and aims

Journal metrics assess impact upon the research literature, and are now used to assess individual researchers in hiring and promotion decisions. This study compared the ranking of addiction journals according to eight widely used metrics; assessed the correlations between journal rankings; and assessed changes over time in metric scores.

Methods

Data pertaining to the 2020 scores on eight metrics for 43 journals were obtained and the top 20 ranking in each compared and the correlations between rankings assessed. The Impact Factor was employed to assess changes over time.

Results

Ignoring the two categorization systems used by some metrics, 31 journals appeared in at least one metric top 20 and 11 in all eight. The top rank in each was occupied by one of three journals. Three-quarters of the correlations between rankings were above 6.0. The number of journals with an Impact Factor rose from 23 in 1997 to 38 in 2020, and the journals added tended to focus on addictions other than alcohol and drugs or have a specific focus.

Conclusions and discussion

The results indicate a concentration of journals at the top of the metrics and moderate to strong agreement between them, but almost three-quarters of journals appeared in at least one metric. The longitudinal data reflect both a broadening and specialization of the addiction field. The study limitations include exclusion of some journals and metrics.

Publishing international library and information science journals: The changing landscape

This study aimed to examine the publication trends of Scimago Journal & Country Rank (SJR)-indexed library and information science (LIS) journals by examining publication share growth by country and region. For this study, we used LIS journals indexed in the SJR in 2000, 2010, and 2020. The results showed that the most frequent publishers of SJR-indexed LIS journals are large commercial publishers. The top three publishers since 2000 were Taylor & Francis, Emerald, and Springer Nature, despite their publication share among SJR-indexed journals declining since this period. The top three countries in journal publishing were the United States, the United Kingdom, and the Netherlands. As the large commercial publishers have established themselves in these countries, the publication shares of these countries have also declined slightly because of the slowing growth rate of their publishers. Similarly, publication shares in both Northern America and Western Europe have declined slightly since 2000. However, the large commercial publishers, based mainly in the United Kingdom and the United States, are likely to take the lead in publishing SJR-indexed international LIS journals over the next decade. The results suggest that publishers from non-Western countries will need to publish significantly more international LIS journals indexed in the major databases to achieve a significant publication share. In conclusion, more innovative ways to support journals published in non-Western countries are needed in order to meet the essential selection criteria of the Scopus and Web of Science journal indexes.

Analysis of the journal impact factor and related bibliometric indicators in education and educational research category

This study analyzes the journal impact factor and related bibliometric indicators in Education and Educational Research (E&ER) category, highlighting the main differences among journal quartiles, using Web of Science (Social Sciences Citation Index, SSCI) as the data source. High impact journals (Q1) publish only slightly more papers than expected, which is different to other areas. The papers published in Q1 journal have greater average citations and lower uncitedness rates compared to other quartiles, although the differences among quartiles are lower than in other areas. The impact factor is only weakly negative correlated (r=-0.184) with the journal self-citation but strongly correlated with the citedness of the median journal paper (r= 0.864). Although this strong correlation exists, the impact factor is still far to be the perfect indicator for expected citations of a paper due to the high skewness of the citations distribution. This skewness was moderately correlated with the citations received by the most cited paper of the journal (r= 0.649) and the number of papers published by the journal (r= 0.484), but no important differences by journal quartiles were observed. In the period 2013–2018, the average journal impact factor in the E&ER has increased largely from 0.908 to 1.638, which is justified by the field growth but also by the increase in international collaboration and the share of papers published in open access. Despite their inherent limitations, the use of impact factors and related indicators is a starting point for introducing the use of bibliometric tools for objective and consistent assessment of researcher.

Measuring academic entities’ impact by content-based citation analysis in a heterogeneous academic network

Evaluating the impact of papers, researchers and venues objectively is of great significance to academia and beyond. This may help researchers, research organizations, and government agencies in various ways, such as helping researchers find valuable papers and authoritative venues and helping research organizations identify good researchers. A few studies find that rather than treating citations equally, differentiating them is a promising way for impact evaluation of academic entities. However, most of those methods are metadata-based only and do not consider contents of cited and citing papers; while a few content-based methods are not sophisticated, and further improvement is possible. In this paper, we study the citation relationships between entities by content-based approaches. Especially, an ensemble learning method is used to classify citations into different strength types, and a word-embedding based method is used to estimate topical similarity of the citing and cited papers. A heterogeneous network is constructed with the weighted citation links and several other features. Based on the heterogeneous network that consists of three types of entities, we apply an iterative PageRank-like method to rank the impact of papers, authors and venues at the same time through mutual reinforcement. Experiments are conducted on an ACL dataset, and the results demonstrate that our method greatly outperforms state-of-the art competitors in improving ranking effectiveness of papers, authors and venues, as well as in being robust against malicious manipulation of citations.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.

Journals that Rise from the Fourth Quartile to the First Quartile in Six Years or Less: Mechanisms of Change and the Role of Journal Self-Citations

Journal self-citations may be increased artificially to inflate a journal’s scientometric indicators. The aim of this study was to identify possible mechanisms of change in a cohort of journals that rose from the fourth (Q4) to the first quartile (Q1) over six years or less in Journal Citation Reports (JCR), and the role of journal self-citations in these changes. A total of 51 different journals sampled from all JCR Science Citation Index (SCI) subject categories improved their rank position from Q4 in 2009 to Q1 in any year from 2010 to 2015. I identified changes in the numerator or denominator of the Journal Impact Factor (JIF) that were involved in each year-to-year transition. The main mechanism of change was the increase in the number of citations used to compute the JIF. The effect of journal self-citations in the increase of the JIF was studied. The main conclusion is that there was no evidence of widespread JIF manipulation through the overuse of journal self-citations.

Does an expert-based evaluation allow us to go beyond the Impact Factor? Experiences from building a ranking of national journals in Poland

This article discusses the Polish Journal Ranking, which is used in the research evaluation system in Poland. In 2015, the ranking, which represents all disciplines, allocated 17,437 journals into three lists: A, B, and C. The B list constitutes a ranking of Polish journals that are indexed neither in the Web of Science nor the European Reference Index for the Humanities. This ranking was built by evaluating journals in three dimensions: formal, bibliometric, and expert-based. We have analysed data on 2035 Polish journals from the B list. Our study aims to determine how an expert-based evaluation influenced the results of final evaluation. In our study, we used structural equation modelling, which is regression based, and we designed three pairs of theoretical models for three fields of science: (1) humanities, (2) social sciences, and (3) engineering, natural sciences, and medical sciences. Each pair consisted of the full model and the reduced model (i.e., the model without the expert-based evaluation). Our analysis revealed that the multidimensional evaluation of local journals should not rely only on the bibliometric indicators, which are based on the Web of Science or Scopus. Moreover, we have shown that the expert-based evaluation plays a major role in all fields of science. We conclude with recommendations that the formal evaluation should be reduced to verifiable parameters and that the expert-based evaluation should be based on common guidelines for the experts.