Exploring the limitations of the h-index and h-type indexes in measuring the research performance of authors

The Relative Citation Ratio: An Impartial Assessment of Productivity in Academic Otolaryngologists

OBJECTIVES

Quantification of academic productivity relies on bibliometric measurements, such as the Hirsch index (h-index). The National Institutes of Health (NIH) recently developed the relative citation ratio (RCR), an article-level, citation-driven metric that compares researchers with others within their respective fields. Our study is the first to compare the usage of RCR in academic otolaryngology.

STUDY DESIGN

Retrospective Database Review.

METHODS

Academic otolaryngology residency programs were identified using the 2022 Fellowship and Residency Electronic Interactive Database. Demographic and training data were collected for surgeons using institutional websites. RCR was calculated using the NIH iCite tool, and h-index was calculated using Scopus. Mean RCR (m-RCR) is the average score of the author's articles. Weighted RCR (w-RCR) is the sum of all article scores. These derivatives are a measure of impact and output, respectively. The career duration of a physician was categorized into the following cohorts: 0-10, 11-20, 21-30, and 31+ years.

RESULTS

A total of 1949 academic otolaryngologists were identified. Men had higher h-indices and w-RCRs than women (both p less than 0.001). m-RCR was not different between genders (p = 0.083). There was a difference in h-index and w-RCR (both p less than 0.001) among the career duration cohorts, but there was no difference in m-RCR among the cohorts (p = 0.416). The faculty rank professor was the greatest for all metrics (p < 0.001).

CONCLUSION

Critics of the h-index argue that it is reflective of the time a researcher has spent in the field, instead of impact. The RCR may reduce historic bias against women and younger otolaryngologists.

LEVEL OF EVIDENCE

NA Laryngoscope, 134:592-599, 2024.

Does Research Training lead to Academic Success in Orthopedic Surgery? An Analysis of U.S Academic Orthopedic Surgeons

BACKGROUND

Academic surgeons are invaluable for scientific advancement and training the next generation of orthopedic surgeons.

OBJECTIVE

This study aimed to describe a cohort of academic orthopedic surgeons currently in practice with common academic metrics.

METHODS

ACGME-accredited orthopedic surgery programs with a university affiliation were identified. The primary independent variable in this study was formal research training as defined by a research fellowship or attainment of a PhD. Outcomes included academic rank, h-index attained, number of publications, and funding by the National Institutes of Health (NIH).

RESULTS

1641 orthopedic surgeons were identified across 73 programs. 116 surgeons (7.07%) received formal academic research training. The academic training group and non-academic training group had a similar completion rate of clinical fellowship programs (93.97% vs 93.77%, p=0.933), attainment of other advanced degrees (10.34% vs 8.46%, p=0.485), and years since completion of training (17.49-years vs 16.28-years, p=0.284). Surgeons completing academic research training had a significantly higher h-index (18.46 vs 10.88, p<0.001), higher publication number (67.98 vs 37.80, p<0.001), and more likely to be NIH funded (16.38% vs 3.15%, p<0.001). Surgeons completing academic training were more likely to be associate professors (34.48% vs 25.77%), professors (25.00% vs 22.82%), and endowed professors (10.34% vs 2.43%) (p<0.001). On regression analysis, formalized research training was independently associated with h-index and NIH funding (p<0.001 for both).

CONCLUSION

Formalized research training, either as a research fellowship or PhD, is associated with an increased h-index and likelihood of NIH funding, although this association was not found for academic rank after adjusted regression analysis.

Improved PageRank and New Indices for Academic Impact Evaluation Using AI Papers as Case Studies

Evaluating academic papers and groups is important in scholar evaluation and literature retrieval. However, current evaluation indices, which pay excessive attention to the citation number rather than the citation importance and unidirectionality, are relatively simple. This study proposes new evaluation indices for papers and groups. First, an improved PageRank (PR) algorithm introducing citation importance is proposed to obtain a new citation-based paper index (CPI) via a pre-ranking and fine-tuning strategy. Second, to evaluate the paper’s influence inside and outside its research field, the focus citation-based paper index (FCPI) and diversity citation-based paper index (DCPI) are proposed based on topic similarity and diversity, respectively. Third, aside from the statistical indices for academic papers, we propose a foreign academic degree of dependence (FAD) to characterise the dependence between two academic groups. Finally, artificial intelligence (AI) papers from 2005 to 2019 are utilised for a case study.

The H Index Myth: A Form of Fanaticism or a Simple Misconception?

Bibliometry represents a branch of library and information sciences that uses statistical methods to analyse publications [...].

Metrics and methods in the evaluation of prestige bias in peer review: A case study in computer systems conferences

The integrity of peer review is essential for modern science. Numerous studies have therefore focused on identifying, quantifying, and mitigating biases in peer review. One of these better-known biases is prestige bias, where the recognition of a famous author or affiliation leads reviewers to subconsciously treat their submissions preferentially. A common mitigation approach for prestige bias is double-blind reviewing, where the identify of authors is hidden from reviewers. However, studies on the effectivness of this mitigation are mixed and are rarely directly comparable to each other, leading to difficulty in generalization of their results. In this paper, we explore the design space for such studies in an attempt to reach common ground. Using an observational approach with a large dataset of peer-reviewed papers in computer systems, we systematically evaluate the effects of different prestige metrics, aggregation methods, control variables, and outlier treatments. We show that depending on these choices, the data can lead to contradictory conclusions with high statistical significance. For example, authors with higher h-index often preferred to publish in competitive conferences which are also typically double-blind, whereas authors with higher paper counts often preferred the single-blind conferences. The main practical implication of our analyses is that a narrow evaluation may lead to unreliable results. A thorough evaluation of prestige bias requires a careful inventory of assumptions, metrics, and methodology, often requiring a more detailed sensitivity analysis than is normally undertaken. Importantly, two of the most commonly used metrics for prestige evaluation, past publication count and h-index, are not independent from the choice of publishing venue, which must be accounted for when comparing authors prestige across conferences.

Researcher Performance in Scopus Articles (RPSA) as a New Scientometric Model of Scientific Output: Tested in Business Area of V4 Countries

The purpose of this study is to present a new scientometric model for measuring individual scientific performance in Scopus article publications in the field of Business, Management, and Accounting (BMA). With the help of this model, the study also compares the publication performance of the top 50 researchers according to SciVal in the field of BMA, in each of the Central European V4 countries (Czech Republic; Hungary; Poland; Slovakia). To analyze the scientific excellence of a total of top 200 researchers in the countries studied, we collected and analyzed the data of a total of 1844 partially redundant and a total of 1492 cleansed BMA publications. In the scope of the study, we determined the quality of the journals using SCImago, the individual contributions to the journal articles, and the number of citations using Scopus data. A comparison of individual performance, as shown by published journal articles, can be made based on the qualities of the journals, the determination of the aggregated co-authorship ratios, and the number of citations received. The performance of BMA researchers in Hungary lags behind the average of V4s in terms of quantity, but in terms of quality it reaches this average. As for BMA journal articles, the average number of co-authors is between two and three; concerning Q4 to Q2 publications, this number typically increases. In fact, in the case of these Q journals multiple co-authorship results in higher citations, but it is not the case concerning Q1 journals.

Do researchers know what the h-index is? And how do they estimate its importance?

The h-index is a widely used scientometric indicator on the researcher level working with a simple combination of publication and citation counts. In this article, we pursue two goals, namely the collection of empirical data about researchers’ personal estimations of the importance of the h-index for themselves as well as for their academic disciplines, and on the researchers’ concrete knowledge on the h-index and the way of its calculation. We worked with an online survey (including a knowledge test on the calculation of the h-index), which was finished by 1081 German university professors. We distinguished between the results for all participants, and, additionally, the results by gender, generation, and field of knowledge. We found a clear binary division between the academic knowledge fields: For the sciences and medicine the h-index is important for the researchers themselves and for their disciplines, while for the humanities and social sciences, economics, and law the h-index is considerably less important. Two fifths of the professors do not know details on the h-index or wrongly deem to know what the h-index is and failed our test. The researchers’ knowledge on the h-index is much smaller in the academic branches of the humanities and the social sciences. As the h-index is important for many researchers and as not all researchers are very knowledgeable about this author-specific indicator, it seems to be necessary to make researchers more aware of scholarly metrics literacy.

Web of Science (WoS) and Scopus: The Titans of Bibliographic Information in Today’s Academic World

Nowadays, the importance of bibliographic databases (DBs) has increased enormously, as they are the main providers of publication metadata and bibliometric indicators universally used both for research assessment practices and for performing daily tasks. Because the reliability of these tasks firstly depends on the data source, all users of the DBs should be able to choose the most suitable one. Web of Science (WoS) and Scopus are the two main bibliographic DBs. The comprehensive evaluation of the DBs’ coverage is practically impossible without extensive bibliometric analyses or literature reviews, but most DBs users do not have bibliometric competence and/or are not willing to invest additional time for such evaluations. Apart from that, the convenience of the DB’s interface, performance, provided impact indicators and additional tools may also influence the users’ choice. The main goal of this work is to provide all of the potential users with an all-inclusive description of the two main bibliographic DBs by gathering the findings that are presented in the most recent literature and information provided by the owners of the DBs at one place. This overview should aid all stakeholders employing publication and citation data in selecting the most suitable DB.

An indexing system for the relevance of academic production and research from digital repositories and metadata

Purpose

This paper aims to propose a set of quantitative statistical indicators for measuring the scientific relevance of research groups and researchers, based on high-impact open-access digital production repositories.

Design/methodology/approach

An action research (AR) methodology is proposed in which research is associated with the practice; research informs practice and practice is responsible for informing research in a cooperative way. AR is divided into five phases, beginning with the definition of the problematic scenario and an analysis of the state of the art and ending with conducting tests and publishing the results.

Findings

The proposed indicators were used to characterise group and individual output in a major public university in south-eastern Mexico. University campuses hosting a large number of high-impact research groups. These indicators were very useful in generating information that confirmed specific assumptions about the scientific production of the university.

Research limitations/implications

The data used here were retrieved from Scopus and open access national repository of Mexico. It would be possible to use other data sources to calculate these indicators.

Practical implications

The system used to implement the proposed indicators is independent of any particular technological tool and is based on standards for metadata description and exchange, thus facilitating the easy integration of new elements for evaluation.

Social implications

Many organisations evaluate researchers according to specific criteria, one of which is the prestige of journals. Although the guidelines differ between evaluation bodies, relevance is measured based on elements that can be adapted and where some have greater weight than others, including the prestige of the journal, the degree of collaboration with other researchers and individual production, etc. The proposed indicators can be used by various entities to evaluate researchers and research groups. Each country has its own organisations that are responsible for evaluation, using various criteria based on the impact of the publications.

Originality/value

The proposed indicators assess based on the importance of the types of publications and the degree of collaborations. However, they can be adapted to other similar scenarios.

Social Media Influence Does Not Reflect Scholarly or Clinical Activity in Real Life

BACKGROUND

Social media has become a major source of communication in medicine. We aimed to understand the relationship between physicians' social media influence and their scholarly and clinical activity.

METHODS

We identified attending US electrophysiologists on Twitter. We compared physician Twitter activity to (1) scholarly publication record (h-index) and (2) clinical volume according to Centers for Medicare and Medicaid Services. The ratio of observed versus expected (obs/exp) Twitter followers was calculated based on each scholarly (K-index) and clinical activity.

RESULTS

We identified 284 physicians, with mean Twitter age of 5.0 (SD, 3.1) years and median 568 followers (25th, 75th: 195, 1146). They had a median 34.5 peer-reviewed articles (25th, 75th: 14, 105), 401 citations (25th, 75th: 102, 1677), and h-index 9 (25th, 75th: 4, 19.8). The median K-index was 0.4 (25th, 75th: 0.15, 1.0), ranging from 0.0008 to 29.2. The median number of electrophysiology procedures was 77 (25th, 75th: 0, 160) and evaluation and management visits 264 (25th, 75th: 59, 516) in 2017. The top 1% electrophysiologists for followers accounted for 20% of all followers, 17% of status updates, had a mean h-index of 6 (versus 15 for others, P=0.3), and accounted for 1% of procedural and evaluation and management volumes. They had a mean K-index of 21 (versus 0.77 for others, P<0.0001) and clinical obs/exp follower ratio of 17.9 and 18.1 for procedures and evaluation and management (P<0.001 each, versus others [0.81 for each]).

CONCLUSIONS

Electrophysiologists are active on Twitter, with modest influence often representative of scholarly and clinical activity. However, the most influential physicians appear to have relatively modest scholarly and clinical activity.

Emotions and Sport Management: A Bibliometric Overview

Emotions are considered a fundamental aspect of sport scenarios, and within sports, consumer behavior is a very popular area of research in the sport management field. Thus, in recent years, there has been a growing interest for sport managers regarding the role that emotions play in sport consumer behavior. Thus, the aim of this paper is to provide an overview of the academic research on emotions in the sport management field using two techniques: a bibliometric performance analysis and a graphic mapping of the references in this field. This analysis focuses on authors, journals, papers, institutions and countries. Bibliometric indicators including the h-index measure, productivity and the number of citations were used to perform the performance analysis. Then, VOSviewer software was used to perform co-citation, bibliographic coupling and co-occurrence of keyword analysis (mapping analysis). The results of both types of analysis are consistent, with the United States being the most influential country in emotions in sport management research because the main authors and institutions in this research field belong to this country. The overall results indicate that the literature on this research topic has grown significantly in recent years in all scientific disciplines; however, the research topic is incipient, and therefore, the number of articles is still limited. Thus, this research presents the key aspects in the topic of emotions in sport management that could be helpful for researchers and policy makers in the field of sport management to make future decisions.

An Overview of the Dynamics of Relative Research Performance in Central-Eastern Europe Using a Ranking-Based Analysis Derived from SCImago Data

In recent times, rankings seem to play an increasingly important role, influencing the lives of individual researchers or academics and their institutions. Individual and institutional rankings used for promotion and research or academic funding seem to illustrate more and more the “publish or perish” mantra, relying sometimes almost exclusively on publications and their citations. Eastern Europe found itself part of this new world after a period of isolation, uneven for the countries within the area. The present study uses SCImago data to perform a regional analysis of individual and aggregated domains, for individual countries and the entire region, based on a novel “adjusted citation index”, in order to measure the performance and identify, using correlations with additional data and information, the mechanisms that can increase the research performance of a country. In a nutshell, the results indicate that the national research policies are responsible for performance. Adaptive research policies simulate a real performance, in comparison with more restrictive ones, which are more likely to stimulate unethical behaviors such as self-citations or citation stacking, especially when used for the assessment of researchers. The importance of the findings lies in the possibility of replicating the methodology, adapting it to different spatial scales.