Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005-2019

Towards equal representation - A bibliometric analysis of authorships in Laboratory Medicine and Clinical Chemistry from the United States, Canada, and Europe (2005-2022)

Objectives

Although diversity has been demonstrated to benefit research groups, women remain underrepresented in most scientific disciplines, including Laboratory Medicine and Clinical Chemistry. In order to promote diversity and equality in scientific communities, understanding the gender distribution of authorship is crucial.

Methods

This study included a total of 30,268 Web of Science-listed Clinical Chemistry and Laboratory Medicine publications from the United States of America, Canada, and the member countries of the European Federation of Clinical Chemistry and Laboratory Medicine from 2005 to 2022. In addition to the publication productivity of female and male authors over time, gender-specific publication characteristics and country-specific gender distributions of authorships were examined.

Results

Overall, publications with female first authors increased by 49 % between 2005 and 2022, averaging 42 % female first authors. Eastern Europe (60 %) and Southern Europe (51 %) had particularly high proportions of female first authors. While female last authorship was the most predictive of female first authorship, with an odds ratio of 2.01 (95 % CI: 1.91-2.12, p < 0.001), only 27 % of last authors were female. Moreover, citation rate was not predictive of female first or last authorship.

Conclusion

Authorship in Clinical Chemistry and Laboratory Medicine is moving towards gender parity. This trend is more pronounced for first authors than for last authors. Further research into the citations of female authors in this discipline could be a starting point for increasing the visibility of women researchers in science. Moreover, geographical differences may provide opportunities for future research on gender parity across disciplines.

AI and business management: Tracking future research agenda through bibliometric network analysis

This study has been designed to analyse the academic landscape of AI on the Scopus and Web of Science (WOS) indices and compare the findings. AI is one of the most prominent and preferred research areas, only a few studies are dedicated to the bibliometric aspect of it. There is a need to compare studies on AI over different databases to identify the impact and usefulness of those studies in decision-making in business management. To conduct this analysis, the authors have collected data from both Scopus and WOS. 'VOSviewer', 'R-Studio', and 'MS Excel' software have been used for performance analysis and science mapping. This is one of the exceptional studies which perform a comparative analysis between two indices and also identifies funding sponsors for support of research in AI. "Dwivedi, Y.K." is the most productive author and "Huang, Minghui" is the most impactful author. "National Natural Science Foundation of China" is the funding agency which has significantly supported AI research. Technical aspects like "Machine learning", "neural networks", and "blockchain" with 'Sustainability', 'sustainable development', 'accounting', and 'auditing' are trending themes for managerial decision-making.

Preliminary guideline for reporting bibliometric reviews of the biomedical literature (BIBLIO): a minimum requirements

BACKGROUND

A bibliometric review of the biomedical literature could be essential in synthesizing evidence if thoroughly conducted and documented. Although very similar to review papers in nature, it slightly differs in synthesizing the data when it comes to providing a pile of evidence from different studies into a single document. This paper provides a preliminary guideline for reporting bibliometric reviews of the biomedical literature (BIBLIO).

METHODS

The BIBLIO was developed through two major processes: literature review and the consensus process. The BIBLIO started with a comprehensive review of publications on the conduct and reporting of bibliometric studies. The databases searched included PubMed, Scopus, Web of Sciences, and Cochrane Library. The process followed the general recommendations of the EQUATOR Network on how to develop a reporting guideline, of which one fundamental part is a consensus process. A panel of experts was invited to identify additional items and was asked to choose preferred options or suggest another item that should be included in the checklist. Finally, the checklist was completed based on the comments and responses of the panel members in four rounds.

RESULTS

The BIBLIO includes 20 items as follows: title (2 items), abstract (1 item), introduction/background (2 items), methods (7 items), results (4 items), discussion (4 items). These should be described as a minimum requirements in reporting a bibliometric review.

CONCLUSIONS

The BIBLIO for the first time provides a preliminary guideline of its own kind. It is hoped that it could contribute to the transparent reporting of bibliometric reviews. The quality and utility of BIBILO remain to be investigated further.

Bibliometric Analysis of Research Articles on Virtual Reality in the Field of Pain Medicine Published from 1993 to 2022

Purpose

This study aims to analyze global and regional (China, Japan, and South Korea) research on virtual reality (VR) in the field of pain medicine over the past 30 years. Specifically, we quantify VR-related publications, examine the distribution of research topics on chronic and acute pain, and identify trends and future directions.

Methods

The Web of Science Core Collection (WoSCC) database was used for bibliometric analysis. This study included articles written in English between 1993 and 2022. The search strategy used predefined terms related to VR and pain. Based on the articles' titles and abstracts, two pain physicians independently reviewed and classified them as acute or chronic pain. Quantitative data on countries, institutions, journals, and research categories were analyzed. VOSviewer software was used for keyword mapping and clustering.

Results

We analyzed 808 VR-related articles on pain medicine. Over the past three decades, the number of publications in this field has increased steadily. The United States of America (n = 259) had the highest number of publications. Moreover, China (n = 42), Japan (n = 18), and South Korea (n = 24) also contributed continuously. Acute and chronic pain research accounted for 44.2% and 37.9% of the articles, respectively. The most common acute pain topic was procedure-related (n = 129, 16.0%), whereas the most common chronic pain topic was neuropathic (n = 104, 12.9%). Keywords clustered around neuroscience, pediatric pain management, and chronic pain management.

Conclusion

Our study revealed academic achievements and growing interest in VR-related research in pain medicine. Researchers worldwide have shown balanced interest in applying VR technology to acute and chronic pain, with specific contributions from China, Japan, and South Korea. Harnessing VR technology is promising for improving pain management and enhancing patients' quality of life in the field of pain medicine.

Prostate cancer, online health information and communication technology - Bibliometric analysis of field with research frontiers

OBJECTIVES

The purpose of this study is to conduct the first bibliometric analysis which examines eHealth communication technologies in prostate cancer care, and the utilization of internet-based health information and communication technology by men with prostate cancer.

METHODS

Original articles were extracted from the Science Citation Index Expanded (SCI-E) on Web of Science (WOS) and analyzed concerning their distributions. Quantitative guidance directed investigation of findings from previous studies and trending issues within the field. The WOS, VOSViewer and CiteSpace IV were used for information analysis.

RESULTS

302 articles were included in the final analysis. There has been a 165 % increase in productivity over the past decade. The leading country by publication was the USA (145 articles = 48.02 %). Journals which published the highest number of original articles were the Journal of Medical Internet Research (6.95 %), and Patient Education and Counseling (4.64 %).

DISCUSSION AND PRACTICE IMPLICATIONS

The field of research which examines utilization and impacts of internet-based health information on men with prostate cancer is growing and diverse. Research frontiers are 'Information quality and diversity', 'eHealth literacy', 'decision making', and 'survivorship and advanced disease'. Clinicians should be aware of several significant limitations which exist within the current field of research.

Quantifying progress in research topics across nations

A scientist's choice of research topic affects the impact of their work and future career. While the disparity between nations in scientific information, funding, and facilities has decreased, scientists on the cutting edge of their fields are not evenly distributed across nations. Here, we quantify relative progress in research topics of a nation from the time-series comparison of reference lists from papers, using 71 million published papers from Scopus. We discover a steady leading-following relationship in research topics between Western nations or Asian city-states and others. Furthermore, we find that a nation's share of information-rich scientists in co-authorship networks correlates highly with that nation's progress in research topics. These results indicate that scientists' relationships continue to dominate scientific evolution in the age of open access to information and explain the failure or success of nations' investments in science.

Thirty-year survey of bibliometrics used in the research literature of pain: Analysis, evolution, and pitfalls

During the last decades, the emergence of Bibliometrics and the progress in Pain research have led to a proliferation of bibliometric studies on the medical and scientific literature of pain (B/P). This study charts the evolution of the B/P literature published during the last 30 years. Using various searching techniques, 189 B/P studies published from 1993 to August 2022 were collected for analysis-half were published since 2018. Most of the selected B/P publications use classic bibliometric analysis of Pain in toto, while some focus on specific types of Pain with Headache/Migraine, Low Back Pain, Chronic Pain, and Cancer Pain dominating. Each study is characterized by the origin (geographical, economical, institutional, …) and the medical/scientific context over a specified time span to provide a detailed landscape of the Pain research literature. Some B/P studies have been developed to pinpoint difficulties in appropriately identifying the Pain literature or to highlight some general publishing pitfalls. Having observed that most of the recent B/P studies have integrated newly emergent software visualization tools (SVTs), we found an increase of anomalies and suggest that readers exercise caution when interpreting results in the B/P literature details.

Bibliometric analysis of research articles on post-herpetic neuralgia published from 1991 to 2020

Post-herpetic neuralgia (PHN) is one of the most painful diseases, which has made it a major concern for pain physicians. We aimed to quantitatively analyze the research outputs of studies on PHN published over the past 30 years using bibliometric analysis. We also aimed to analyze the research outputs of studies on interventional treatments for PHN and evaluate the academic achievements of Korean pain physicians. Bibliometric analysis was performed by searching the Web of Science database for PHN-related articles published between 1991 and 2020. Publication number, year, source, country, institution, and citation-related information were retrieved from the database. We also quantitatively analyzed publications related to interventional treatments for PHN. A total of 3285 publications were extracted from the database; 101 (3.1%) of the articles were published by South Korean authors, making South Korea the 11th in the order of countries that published the most articles. There were 185 articles on the effects of interventional treatments for PHN. South Korean authors published 30 (16.2%) articles out of these, making South Korea the 3rd in the order of countries that published the most articles on the effects of interventional treatment for PHN. Our results showed an increasing trend in the number of PHN-related publications and the academic achievements of Korean pain physicians in this field over the past 3 decades. However, the proportion of studies on interventional treatments is relatively small. Korean pain physicians need to establish academic evidence on interventional treatment to expand their role in this field and improve the outcomes of PHN patients.

From Research Evaluation to Research Analytics. The digitization of academic performance measurement

One could think that bibliometric measurement of academic performance has always been digital since the computer-assisted invention of the Science Citation Index. Yet, since the 2000s, the digitization of bibliometric infrastructure has accelerated at a rapid pace. Citation databases are indexing an increasing variety of publication types. Altmetric data aggregators are producing data on the reception of research outcomes. Machine-readable persistent identifiers are created to unambiguously identify researchers, research organizations, and research objects; and evaluative software tools and current research information systems are constantly enlarging their functionalities to make use of these data and extract meaning from them. In this article, we analyse how these developments in evaluative bibliometrics have contributed to an extension of indicator-based research evaluation towards data-driven research analytics. Drawing on empirical material from blogs and websites as well as from research and policy papers, we discuss how interoperability, scalability, and flexibility as material specificities of digital infrastructures generate new ways of data production and their assessment, which affect the possibilities of how academic performance can be understood and (e)valuated.

The role of Israeli researchers in the scientific literature regarding COVID-19 vaccines

BACKGROUND

The accurate and timely publication of scientific findings is a key component of the global response to the COVID-19 pandemic. This article explores the role of Israeli researchers in the scientific literature regarding COVID-19 vaccines.

METHODS

Content and bibliometric analysis of articles included in the Web of Science database regarding COVID-19 vaccines, that were published between January 2020 and June 2022.

RESULTS

The Web of Science includes 18,596 articles regarding COVID-19 vaccines that were published between January 2020 and June 2022. 536 (3%) of those articles had at least one Israeli author. These "Israeli articles" accounted for 11% of the NEJM articles on COVID-19 vaccines, 9% of such articles in Nature Medicine, and 4% of such articles in the Lancet. 80 of the 536 Israeli articles (15%) were recognized as "Highly Cited Papers" (articles that rank in the top 1% by citations for field and publication year). Most of the Israeli Highly Cited Papers (HCPs) analyzed the safety and/or efficacy of the COVID-19 vaccine developed by Pfizer and BioNTech (BNT162b2). Most of the Israeli HCPs made use of detailed and comprehensive individual data available from Israel's health plans, hospitals, or Ministry of Health. The 15% HCP rate (i.e., the number of HCPs divided by the number of all articles) for the Israeli articles was triple the HCP rate for all articles on COVID-19 vaccines (5%). A key factor contributing to Israel's prominent role in rapid publication of vaccination impact studies was Israel's being a world leader in the initial vaccination rollout, the administration of boosters, and the vaccination of pregnant women. Other contributing factors include Israeli researchers' access to well-developed electronic health record systems linking vaccinations and outcomes, the analytic strengths of leading Israeli researchers and research institutions, collaborations with leading research institutions in other countries, and the ability to quickly identify emerging research opportunities and mobilize accordingly. Recent developments in the priorities and selection criteria of leading journals have also played a role; these include an increased openness to well-designed observational studies and to manuscripts from outside of Europe and North America.

CONCLUSIONS

Israeli researchers, Israeli research institutions, and the Israeli government can, and should, take concrete steps to build upon lessons learned in the course of the recent surge of high-quality publications related to COVID-19 vaccines (such as the value of linking data across organizations). These lessons can be applied to a wide range of fields, including fields that go well beyond vaccines and pandemic responses.

On the possibilities of presenting the research performance of an institute over a long period of time: the case of the Institute of Information Science at the University of Graz in Austria

In this paper, we demonstrate how the research performance of a university institute (department) over a long period of time can be presented and evaluated. Using the example of an information science institute at a German-speaking university, namely the (former) Institute of Information Science at the University of Graz in Austria, we present the research performance of this institute over the entire duration of its existence (33 years) in different ways. In order to be able to contextualize its performance, we compare it with that of some related institutions from all over the world. Due to the high effort involved in collecting data and the lack of data availability, the comparison must be limited to a period of a few years and—with regard to the institutions from non-German-speaking countries—to the Web of Science as data source. In this international comparison, the institute in the focus of the study shows relatively poor results. As can be seen, the choice of the data source has a major influence on the evaluation results. Especially for institutes from non-English-speaking countries with publications in their respective national languages, an exclusive use of international databases, such as Web of Science or Scopus, cannot fully consider the whole research performance. The use of personal publication lists or local research databases seems almost indispensable in these cases. A major novelty of this article is the handling of a very long evaluation period and the discussion of different ways of subdividing it. With regard to the presentation of the results, in the case of a long observation period, not only should annual and overall results be presented, but also multi-year comparisons be performed. In this way, year-by-year fluctuations can be smoothed out, and longer-term developments can be well represented.

Current Status and Trends in Research on Caries Diagnosis: A Bibliometric Analysis

There are a wide variety of devices for the detection and diagnosis of caries from the initial stages. The purpose of this study was to perform a bibliometric study on research regarding caries diagnosis by identifying the contributing researchers, organizations, countries or regions, journals, and to provide an analysis of keyword co-occurrence and co-authorship networks. An advanced search was performed in Web of Science (WOS) Core Collection database, using the terms “caries diagno*” and “caries detect*” in the “topic” field, from 2013 to 2021. Bibliometric parameters were extracted using WOS’s analyze results tools and VOSviewer software. A total of 816 documents were identified. Most of them, (61.3%) are included in “Dentistry Oral Surgery & Medicine” category within WOS. The largest scientific production on the subject is observed between 2018 and 2021, with a total of 344 records. The most productive author is Mendes FM, followed by Braga MM. The journal with the most articles published on caries diagnosis is Caries Research, with 55 articles (6.74%). The terms with the highest co-occurrence refer to the validity of diagnostic methods, tools or principles used in diagnosis or general aspects related to caries detection and diagnosis.

Operational definition of complementary, alternative, and integrative medicine derived from a systematic search

BACKGROUND

Identifying what therapies constitute complementary, alternative, and/or integrative medicine (CAIM) is complex for a multitude of reasons. An operational definition is dynamic, and changes based on both historical time period and geographical location whereby many jurisdictions may integrate or consider their traditional system(s) of medicine as conventional care. To date, only one operational definition of "complementary and alternative medicine" has been proposed, by Cochrane researchers in 2011. This definition is not only over a decade old but also did not use systematic methods to compile the therapies. Furthermore, it did not capture the concept "integrative medicine", which is an increasingly popular aspect of the use of complementary therapies in practice. An updated operational definition reflective of CAIM is warranted given the rapidly increasing body of CAIM research literature published each year.

METHODS

Four peer-reviewed or otherwise quality-assessed information resource types were used to inform the development of the operational definition: peer-reviewed articles resulting from searches across seven academic databases (MEDLINE, EMBASE, AMED, PsycINFO, CINAHL, Scopus and Web of Science); the "aims and scope" webpages of peer-reviewed CAIM journals; CAIM entries found in online encyclopedias, and highly-ranked websites identified through searches of CAIM-related terms on HONcode. Screening of eligible resources, and data extraction of CAIM therapies across them, were each conducted independently and in duplicate. CAIM therapies across eligible sources were deduplicated.

RESULTS

A total of 101 eligible resources were identified: peer-reviewed articles (n = 19), journal "aims and scope" webpages (n = 22), encyclopedia entries (n = 11), and HONcode-searched websites (n = 49). Six hundred four unique CAIM terms were included in this operational definition.

CONCLUSIONS

This updated operational definition is the first to be informed by systematic methods, and could support the harmonization of CAIM-related research through the provision of a standard of classification, as well as support improved collaboration between different research groups.

Comparing standard, collaboration and fractional CNCI at the institutional level: Consequences for performance evaluation

The average Category Normalised Citation Impact (CNCI) of an institution’s publication output is a widely used indicator for research performance benchmarking. However, it combines all entity contributions, obscuring individual inputs and preventing clear insight and sound policy recommendations if it is not correctly understood. Here, variations (Fractional and Collaboration [Collab] CNCI)—which aim to address the obscurity problem—are compared to the Standard CNCI indicator for over 250 institutions, spread globally, covering a ten-year period using Web of Science data. Results demonstrate that both Fractional and Collab CNCI methods produce lower index values than Standard CNCI. Fractional and Collab results are often near-identical despite fundamentally different calculation approaches. Collab-CNCI, however, avoids assigning fractional credit (which is potentially incorrect) and is relatively easy to implement. As single metrics obscure individual inputs, institutional output is also deconstructed into five collaboration groups. These groups track the increasing international collaboration trend, particularly highly multi-lateral studies and the decrease in publications authored by single institutions. The deconstruction also shows that both Standard and Fractional CNCI increase with the level of collaboration. However, Collab-CNCI does not necessarily follow this pattern thus enabling the identification of institutions where, for example, their domestic single articles are their best performing group. Comparing CNCI variants and deconstructing portfolios by collaboration type is, when understood and used correctly, an essential tool for interpreting institutional performance and informing policy making.

Hatching start-ups for sustainable growth: a bibliometric review on business incubators

Business incubators hatch start-ups, helping them to survive their early stage and to create a solid foundation for sustainable growth by providing services and access to knowledge. The great practical relevance led to a strong interest of researchers and a high output of scholarly publications, which made the field complex and scattered. To organize the research on incubators and provide a systematic overview of the field, we conducted bibliometric performance analyses and science mappings. The performance analyses depict the temporal development of the number of incubator publications and their citations, the most cited and most productive journals, countries, and authors, and the 20 most cited articles. The author keyword co-occurrence analysis distinguishes six, and the bibliographic coupling seven research themes. Based on a content analysis of the science mappings, we propose a research framework for future research on business incubators.

Therapeutic Effects of Melatonin on Ocular Diseases: Knowledge Map and Perspective

Melatonin plays a critical role in the pathophysiological process including circadian rhythm, apoptosis, and oxidative stress. It can be synthesized in ocular tissues, and its receptors are also found in the eye, triggering more investigations concentrated on the role of melatonin in the eye. In the past decades, the protective and therapeutic potentials of melatonin for ocular diseases have been widely revealed in animal models. Herein, we construct a knowledge map of melatonin in treating ocular diseases through bibliometric analysis and review its current understanding and clinical evidence. The overall field could be divided into twelve topics through keywords co-occurrence analysis, in which the glaucoma, myopia, and retinal diseases were of greatest research interests according to the keywords burst detection. The existing clinical trials of melatonin in ocular diseases mainly focused on the glaucoma, and more research should be promoted, especially for various diseases and drug administration. We also discuss its bioavailability and further research topics including developing melatonin sensors for personalized medication, acting as stem cell therapy assistant drug, and consuming food-derived melatonin for facilitating its clinical transformation.

A comparison of systematic reviews and guideline-based systematic reviews in medical studies

The question of how citation impact relates to academic quality accompanies every decade in bibliometric research. Although experts have employed more complex conceptions of research quality for responsible evaluation, detailed analyses of how impact relates to dimensions such as methodological rigor are lacking. But the increasing number of formal guidelines for biomedical research offer not only the potential to understand the social dynamics of standardization, but also their relations to scientific rewards. By using data from Web of Science and PubMed, this study focuses on systematic reviews from biomedicine and compares this genre with those systematic reviews that applied the PRISMA reporting standard. Besides providing an overview about growth and location, it was found that the latter, more standardized type of systematic review accumulates more citations. It is argued that instead of reinforcing the traditional conception that higher impact represents higher quality, highly prolific authors could be more inclined to develop and apply new standards than more average researchers. In addition, research evaluation would benefit from a more nuanced conception of scientific output which respects the intellectual role of various document types.

Bibliometric Reports for Institutions: Best Practices in a Responsible Metrics Scenario

Carrying out bibliometric reports is one of the common tasks performed by librarians and practitioners within the framework of their professional duties. The emergence of novel data sources, the need to measure new research activities and the growing demand for fairer and more equitable evaluation within the framework of the Responsible Metrics movement has led to calls for a review of the traditional approaches to these types of reports. The main goal of this study is to outline a series of recommendations for bibliometricians, consultants and research support librarians when drafting bibliometric reports in their institutions. These best practices can significantly enhance the quality and utility of bibliometric reports, posing their practitioners as key players in the science management process.

Do researchers know what the h-index is? And how do they estimate its importance?

The h-index is a widely used scientometric indicator on the researcher level working with a simple combination of publication and citation counts. In this article, we pursue two goals, namely the collection of empirical data about researchers’ personal estimations of the importance of the h-index for themselves as well as for their academic disciplines, and on the researchers’ concrete knowledge on the h-index and the way of its calculation. We worked with an online survey (including a knowledge test on the calculation of the h-index), which was finished by 1081 German university professors. We distinguished between the results for all participants, and, additionally, the results by gender, generation, and field of knowledge. We found a clear binary division between the academic knowledge fields: For the sciences and medicine the h-index is important for the researchers themselves and for their disciplines, while for the humanities and social sciences, economics, and law the h-index is considerably less important. Two fifths of the professors do not know details on the h-index or wrongly deem to know what the h-index is and failed our test. The researchers’ knowledge on the h-index is much smaller in the academic branches of the humanities and the social sciences. As the h-index is important for many researchers and as not all researchers are very knowledgeable about this author-specific indicator, it seems to be necessary to make researchers more aware of scholarly metrics literacy.

Interpreting Bibliometric Data

Many academic analyses of good practice in the use of bibliometric data address only technical aspects and fail to account for and appreciate user requirements, expectations, and actual practice. Bibliometric indicators are rarely the only evidence put before any user group. In the present state of knowledge, it is more important to consider how quantitative evaluation can be made simple, transparent, and readily understood than it is to focus unduly on precision, accuracy, or scholarly notions of purity. We discuss how the interpretation of 'performance' from a presentation using accurate but summary bibliometrics can change when iterative deconstruction and visualization of the same dataset is applied. From the perspective of a research manager with limited resources, investment decisions can easily go awry at governmental, funding program, and institutional levels. By exploring select real-life data samples we also show how the specific composition of each dataset can influence interpretive outcomes.