Does bibliometric research confer legitimacy to research assessment practice? A sociological study of reputational control, 1972-2016

Bibliographic analysis and evaluation of the mesh keywords in the journal of prosthodontics: Implant, esthetic, and reconstructive dentistry

BACKGROUND

Various studies have been conducted to analyze data via bibliometric analysis in different dentistry, including prosthetic dentistry. However, these studies evaluated the most cited published articles in dentistry.

OBJECTIVE

To analyze and evaluate the characteristic development of several mesh keywords in the Journal of Prosthodontics - Implant, Esthetic, and Reconstructive Dentistry between 1992 and 2022.

METHODS

The Scopus databases was searched to retrieve the data related to 8 categories, including published articles, most cited documents, authors and organizations, and maximum articles cited during the evaluation period. The data retrieved were exported to a Microsoft Excel sheet and were analyzed using the bibliometrix package. The data were first analyzed for ten years and then for 30 years. The highest of 10 were reported for each category, and co-authorship, reoccurrence, and linked data were also reported.

RESULTS

2603 published documents were recorded from the Scopus databases. Articles comprised 87.9% of the total published work, followed by review articles at 5.46%. The most productive decade was between 2013 and 2022. 3793 terms were used during the evaluation period, with the maximum number in 2003-2012 (2481). A total of 5392 keywords were used during 1992-2022, with the highest number (3232) from 2013 to 2022. A total of 6108 authors were acknowledged, with the maximum number (3964) from 2013 to 2022 and the fewest (767) from 1992 to 2002. Authors from different countries increased from 17 countries in 1992-2002 to 52 countries in 2013-2022 (total = 63 countries). Similarly, the number of organizations increased from 312 organizations in 1992-2002 to 1315 organizations in 2013-2022 (total = 1976 organizations).

CONCLUSION

The current bibliometric analysis delivered a comprehensive overview about more spread research topics and its impactful role in contemporary dentistry, especially prosthetic dentistry.

Are quality assessments in science affected by anchoring effects? The proposal of a follow-up study

We plan to empirically study the assessment of scientific papers within the framework of the anchoring-and-adjustment heuristic. This study is a follow-up study which is intended to answer open questions from the previous study with the same topic Bornmann (2021) and Bornmann (2023). The previous and follow-up studies address a central question in research evaluation: does bibliometrics create the social order in science it is designed to measure or does bibliometrics reflect the given social order (which is dependent on the intrinsic quality of research)? If bibliometrics creates the social order, it can be interpreted as an anchoring-and-adjustment heuristic. In the planned study, we shall undertake a survey of corresponding authors with an available email address in the Web of Science database. The authors are asked to assess the quality of articles that they cited in previous papers. The authors are randomly assigned to different experimental settings in which they receive (or not) citation information or a numerical access code to enter the survey. The control group will not receive any further numerical information. In the statistical analyses, we estimate how (strongly) the quality assessments of the cited papers are adjusted by the respondents to the anchor value (citation counts or access code). Thus, we are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation counts), but also by numbers that are not related to quality, i.e. the access code. Strong effects of the anchors would mean that bibliometrics (or any other number) create the social order they are supposed to measure.

Bibliometric analysis and evaluation of the Journal of Prosthetic Dentistry from 1970 to 2019

STATEMENT OF PROBLEM

A comprehensive bibliometric analysis to determine different aspects of the Journal of Prosthetic Dentistry is lacking.

PURPOSE

The purpose of this bibliometric study was to analyze the characteristics of the Journal of Prosthetic Dentistry between 1970 and 2019.

MATERIAL AND METHODS

The Web of Science Core Collection was used to retrieve 9 categories of the Journal of Prosthetic Dentistry, including keywords and terms used, cited documents published, the countries and organizations of the authors, references, and sources cited during this period. Data were exported to a software program and analyzed for each 10-year period and for the entire 50 years. The highest 10 in each category were reported. Co-occurrence, couthorships, and linkage were also reported.

RESULTS

A total of 11 989 records were reached by the search on the Web of Science Core Collection database; of which, 10 638 (92.9%) were included in the analysis. Articles made up 91.1%, of all records, with 217 review documents (1.8%). The most productive decade was 1980 to 1989 with 2936 documents. The total number of citations of all documents (available period 1980 to 2019) including self-citations was 155 112. During the period 1970 to 2019, 14 837 terms were used. The total number of keywords was 4933 (available period 1990 to 2019). There were 15 382 authors, 82 countries, and 2113 organizations identified in articles published in the Journal of Prosthetic Dentistry during this period, with most from the United States. There were 43 027 authors, 95 324 references, and 14 594 sources cited in the Journal of Prosthetic Dentistry during the period surveyed.

CONCLUSIONS

This bibliometric analysis provided a comprehensive overview of the impactful role of the Journal of Prosthetic Dentistry in contemporary dentistry, particularly in the field of prosthodontics.

Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers

Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actors within several arenas. This study aims to investigate how bibliometric indicators are used in a context, which, so far, has not yet been covered by researchers, that of daily newspapers. By a content analysis of a corpus of 583 articles that appeared in four major Italian newspapers between 1990 and 2020, we chronicle the main functions that bibliometrics and bibliometric indicators played in the Italian press. Our material shows, among other things, that the public discourse developed in newspapers creates a favorable environment for bibliometrics-centered science policies, that bibliometric indicators contribute to the social construction of scientific facts in the press, especially in science news related to medicine, and that professional bibliometric expertise struggles to be represented in newspapers and hence reach the general public.

Analysis of co-authorship networks among Brazilian graduate programs in computer science

The growth and popularization of platforms on scientific production has been the subject of several studies, producing relevant analyses of co-authorship behavior among groups of researchers. Researchers and their scientific productions can be analysed as co-authorship social networks, so researchers are linked through common publications. In this context, co-authoring networks can be analysed to find patterns that can describe or characterize them. This work presents the analysis and characterization of co-authorship networks of academic Brazilian graduate programs in computer science. Data from Brazilian researchers were collected and modeled as co-authoring networks among the graduate programs that researchers take part in. Each network topology was analysed with complex network measurements and three proposed qualitative indices that evaluate the publication's quality. In addition, the co-authorship networks of the computer science graduate programs were characterized in relation to the assessment received by CAPES, which attributes a qualitative grade to the graduate programs in Brazil. The results show the most relevant topological measurements for the program's characterization and the evaluations received by the programs in different qualitative degrees, relating the main topological patterns of the co-authorship networks and the CAPES grades of the Brazilian graduate programs in computer science.

Mapping research topics at multiple levels of detail

The institutional review of interdisciplinary bodies of research lacks methods to systematically produce higher-level abstractions. Abstraction methods, like the "distant reading" of corpora, are increasingly important for knowledge discovery in the sciences and humanities. We demonstrate how abstraction methods complement the metrics on which research reviews currently rely. We model cross-disciplinary topics of research publications and projects emerging at multiple levels of detail in the context of an institutional review of the Earth Research Institute (ERI) at the University of California at Santa Barbara. From these, we design science maps that reveal the latent thematic structure of ERI's interdisciplinary research and enable reviewers to "read" a body of research at multiple levels of detail. We find that our approach provides decision support and reveals trends that strengthen the institutional review process by exposing regions of thematic expertise, distributions and clusters of work, and the evolution of these aspects.

Interpreting Bibliometric Data

Many academic analyses of good practice in the use of bibliometric data address only technical aspects and fail to account for and appreciate user requirements, expectations, and actual practice. Bibliometric indicators are rarely the only evidence put before any user group. In the present state of knowledge, it is more important to consider how quantitative evaluation can be made simple, transparent, and readily understood than it is to focus unduly on precision, accuracy, or scholarly notions of purity. We discuss how the interpretation of 'performance' from a presentation using accurate but summary bibliometrics can change when iterative deconstruction and visualization of the same dataset is applied. From the perspective of a research manager with limited resources, investment decisions can easily go awry at governmental, funding program, and institutional levels. By exploring select real-life data samples we also show how the specific composition of each dataset can influence interpretive outcomes.

How can citation impact in bibliometrics be normalized? A new approach combining citing-side normalization and citation percentiles

Since the 1980s, many different methods have been proposed to field-normalize citations. In this study, an approach is introduced that combines two previously introduced methods: citing-side normalization and citation percentiles. The advantage of combining two methods is that their advantages can be integrated in one solution. Based on citing-side normalization, each citation is field weighted and, therefore, contextualized in its field. The most important advantage of citing-side normalization is that it is not necessary to work with a specific field categorization scheme for the normalization procedure. The disadvantages of citing-side normalization—the calculation is complex and the numbers are elusive—can be compensated for by calculating percentiles based on weighted citations that result from citing-side normalization. On the one hand, percentiles are easy to understand: They are the percentage of papers published in the same year with a lower citation impact. On the other hand, weighted citation distributions are skewed distributions with outliers. Percentiles are well suited to assigning the position of a focal paper in such distributions of comparable papers. The new approach of calculating percentiles based on weighted citations is demonstrated in this study on the basis of a citation impact comparison between several countries.

Forty years of emergency medicine research: Uncovering research themes and trends through topic modeling

STUDY OBJECTIVE

Topic identification can facilitate knowledge curation, discover thematic relationships, trends, and predict future direction. We aimed to determine through an unsupervised, machine learning approach to topic modeling the most common research themes in emergency medicine over the last 40 years and summarize their trends and characteristics.

METHODS

We retrieved the complete reference entries including article abstracts from Ovid for all original research articles from 1980 to 2019 within emergency medicine for six widely-cited journals. Abstracts were processed through a natural language pipeline and analyzed by a latent Dirichlet allocation topic modeling algorithm for unsupervised topic discovery. Topics were further examined through trend analysis, word associations, co-occurrence metrics, and two-dimensional embeddings.

RESULTS

We retrieved 47,158 articles during the defined time period that were filtered to 20,528 articles for further analysis. Forty topics covering methodologic and clinical areas were discovered. These topics separated into distinct clusters when embedded in two-dimensional space and exhibited consistent patterns of interaction. We observed the greatest increase in popularity in research themes involving risk factors (0.4% to 5.2%), health utilization (1.2% to 5.0%), and ultrasound (0.7% to 3.3%), and a relative decline in research involving basic science (8.9% to 1.1%), cardiac arrest (6.5% to 2.2%), and vitals (6.3% to 1.3%) over the past 40 years. Our data show only very modest growth in mental health and substance abuse research (1.0% to 1.6%), despite ongoing crises.

CONCLUSIONS

Topic modeling via unsupervised machine learning applied to emergency medicine abstracts discovered coherent topics, trends, and patterns of interaction.

Quantitative science studies should be framed with middle-range theories and concepts from the social sciences

This paper argues that quantitative science studies should frame their data and analyses with middle-range sociological theories and concepts. We illustrate this argument with reference to the “sociology of professions,” a middle-range theoretical framework developed by Chicago sociologist Andrew Abbott. Using this framework, we counter the claim that the use of bibliometric indicators in research assessment is pervasive in all advanced economies. Rather, our comparison between the Netherlands and Italy reveals major differences in the national design of bibliometric research assessment: The Netherlands follows a model of bibliometric professionalism, whereas Italy follows a centralized bureaucratic model that co-opts academic elites. We conclude that applying the sociology of professions framework to a broader set of countries would be worthwhile, allowing the emerging bibliometric profession to be charted in a comprehensive, and preferably quantitative, fashion. We also briefly discuss other sociological middle-range concepts that could potentially guide empirical analyses in quantitative science studies.

A Bibliometric Analysis of Citation Classics in the Journal of Ultrasound in Medicine

OBJECTIVES

A bibliometric analysis of articles in the Journal of Ultrasound in Medicine (JUM) identified the journals' most impactful articles.

METHODS

A bibliometric analysis of citation classics that were published in the JUM from its inception in 1982 to 2019 was performed. All citation classics, defined as articles cited 100 or more times, were evaluated for the number of citations, citations per year, publication year, subspecialty, design, and country of origin. Characteristics were compared before and after 1998 by the Mann-Whitney test for unpaired data and 2-sample z tests of sample proportions. The Kruskal-Wallis test for nonparametric continuous data was used to compare the median number of citations per year by decade of publication.

RESULTS

A total of 7868 articles were published in the JUM between 1982 and 2019; 54 (0.7%) were citation classics. The median citation classics year of publication was 1998 (interquartile range [IQR], 1991-2003). Most citation classics originated from the United States (36 of 54 [66.7%]), were observational (47 of 54 [87%]), and were related to obstetric and gynecologic topics (16 of 54 [29.6%]). Citation classics after 1998 received significantly more citations per year (9.3 versus 4.7; P < .001), with no other differences noted. The median number of citations per year increased for each decade, with medians of 4 citations (IQR, 3.6-4.7) in 1982 to 1991 and 11.2 citations (IQR, 9-13.9) in 2002 to 2012 (P < .001).

CONCLUSIONS

This list provides insight into the most influential articles that were published in the JUM. Most citation classics were observational, were from the United States, and covered obstetric and gynecologic topics. Citation classics received more citations per year after 1998.

Professional standards in bibliometric research evaluation? A meta-evaluation of European assessment practice 2005-2019

Despite growing demand for practicable methods of research evaluation, the use of bibliometric indicators remains controversial. This paper examines performance assessment practice in Europe-first, identifying the most commonly used bibliometric methods and, second, identifying the actors who have defined wide-spread practices. The framework of this investigation is Abbott's theory of professions, and I argue that indicator-based research assessment constitutes a potential jurisdiction for both individual experts and expert organizations. This investigation was conducted using a search methodology that yielded 138 evaluation studies from 21 EU countries, covering the period 2005 to 2019. Structured content analysis revealed the following findings: (1) Bibliometric research assessment is most frequently performed in the Nordic countries, the Netherlands, Italy, and the United Kingdom. (2) The Web of Science (WoS) is the dominant database used for public research assessment in Europe. (3) Expert organizations invest in the improvement of WoS citation data, and set technical standards with regards to data quality. (4) Citation impact is most frequently assessed with reference to international scientific fields. (5) The WoS classification of science fields retained its function as a de facto reference standard for research performance assessment. A detailed comparison of assessment practices between five dedicated organizations and other individual bibliometric experts suggests that corporate ownership and limited access to the most widely used citation databases have had a restraining effect on the development and diffusion of professional bibliometric methods during this period.

Bibliometrics-based decision trees (BBDTs) based on bibliometrics-based heuristics (BBHs): Visualized guidelines for the use of bibliometrics in research evaluation

Fast-and-frugal heuristics are simple strategies that base decisions on only a few predictor variables. In so doing, heuristics may not only reduce complexity but also boost the accuracy of decisions, their speed, and transparency. In this paper, bibliometrics-based decision trees (BBDTs) are introduced for research evaluation purposes. BBDTs visualize bibliometrics-based heuristics (BBHs), which are judgment strategies solely using publication and citation data. The BBDT exemplar presented in this paper can be used as guidance to find an answer on the question in which situations simple indicators such as mean citation rates are reasonable and in which situations more elaborated indicators (i.e., [sub-]field-normalized indicators) should be applied.

Towards a second generation of 'social media metrics': Characterizing Twitter communities of attention around science

‘Social media metrics’ are bursting into science studies as emerging new measures of impact related to scholarly activities. However, their meaning and scope as scholarly metrics is still far from being grasped. This research seeks to shift focus from the consideration of social media metrics around science as mere indicators confined to the analysis of the use and visibility of publications on social media to their consideration as metrics of interaction and circulation of scientific knowledge across different communities of attention, and particularly as metrics that can also be used to characterize these communities. Although recent research efforts have proposed tentative typologies of social media users, no study has empirically examined the full range of Twitter user’s behavior within Twitter and disclosed the latent dimensions in which activity on Twitter around science can be classified. To do so, we draw on the overall activity of social media users on Twitter interacting with research objects collected from the Altmetic.com database. Data from over 1.3 million unique users, accounting for over 14 million tweets to scientific publications, is analyzed. Based on an exploratory and confirmatory factor analysis, four latent dimensions are identified: ‘Science Engagement’, ‘Social Media Capital’, ‘Social Media Activity’ and ‘Science Focus’. Evidence on the predominant type of users by each of the four dimensions is provided by means of VOSviewer term maps of Twitter profile descriptions. This research breaks new ground for the systematic analysis and characterization of social media users’ activity around science.