Including cited non-source items in a large-scale map of science: What difference does it make?

Role of citation and non-citation metrics in predicting the educational impact of textbooks

Purpose

The main objective of the present study is to determine the role of citation-based metrics (PageRank and HITS’ authority and hub scores) and non-citation metrics (Goodreads readers, reviews and ratings, textbook edition counts) in predicting educational ranks of textbooks.

Design/methodology/approach

The rankings of 1869 academic textbooks of various disciplines indexed in Scopus were extracted from the Open Syllabus Project (OSP) and compared with normalized counts of Scopus citations, scores of PageRank, authority and hub (HITS) in Scopus book-to-book citation network, Goodreads ratings and reviews, review sentiment scores and WorldCat book editions.

Findings

Prediction of the educational rank of scholarly syllabus books ranged from 32% in technology to 68% in philosophy, psychology and religion. WorldCat editions in social sciences, medicine and technology, Goodreads ratings in humanities, and book-citation-network authority scores in law and political science accounted for the strongest predictions of the educational score. Thus, each indicator of editions, Goodreads ratings, and book citation authority score alone can be used to show the rank of the academic textbooks, and if used in combination, they will help explain the educational uptake of books even better.

Originality/value

This is the first study examining the role of citation indicators, Goodreads readers, reviews and ratings in predicting the OSP rank of academic books.

Studying the cognitive relatedness between topics in the global science landscape: The case of Big Data research

Taking Big Data research as a case study, this article intends to investigate the cognitive relatedness of research topics across the global science landscape to a focal topic. Several levels of cognitive relatedness are established depending on the citation distance between the citing publications and a core set of publications. The concept of citation generation is adopted for identifying and classifying other publications with different levels of relatedness to the core set. The micro publication-level classification system of Centre for Science and Technology Studies (CWTS) is applied for determining clusters of publication sets at the topic level. The overall cognitive relatedness of micro clusters to Big Data core publications are measured based on the mean citation generation of all the publications in corresponding clusters. In addition to the given clusters, this study also explores the ‘topics’ relatedness from a semantic point of view, by extracting high-frequency title terms of publications in each generation. Results show that data analysis methods and technologies are the topics with the strongest cognitive relatedness to Big Data research, while topics on physics and astronomy studies present the weakest relatedness. This approach allows assessment of relatedness between research topics by considering the citations distribution across multiple citation generations, and can provide useful insights to study and characterise topics with fuzzy boundaries or are difficult to delineate, thus representing a novel toolset relevant in the context of studying interdisciplinary research.

Center-Periphery Structure in Research Communities

Clustering and community detection in networks are of broad interest and have been the subject of extensive research that spans several fields. We are interested in the relatively narrow question of detecting communities of scientific publications that are linked by citations. These publication communities can be used to identify scientists with shared interests who form communities of researchers. Building on the well-known k-core algorithm, we have developed a modular pipeline to find publication communities with center-periphery structure. Using a quantitative and qualitative approach, we evaluate community finding results on a citation network consisting of over 14 million publications relevant to the field of extracellular vesicles. We compare our approach to communities discovered by the widely used Leiden algorithm for community finding.

Fine-grained classification of social science journal articles using textual data: A comparison of supervised machine learning approaches

We compare two supervised machine learning algorithms—Multinomial Naïve Bayes and Gradient Boosting—to classify social science articles using textual data. The high level of granularity of the classification scheme used and the possibility that multiple categories are assigned to a document make this task challenging. To collect the training data, we query three discipline specific thesauri to retrieve articles corresponding to specialties in the classification. The resulting data set consists of 113,909 records and covers 245 specialties, aggregated into 31 subdisciplines from three disciplines. Experts were consulted to validate the thesauri-based classification. The resulting multilabel data set is used to train the machine learning algorithms in different configurations. We deploy a multilabel classifier chaining model, allowing for an arbitrary number of categories to be assigned to each document. The best results are obtained with Gradient Boosting. The approach does not rely on citation data. It can be applied in settings where such information is not available. We conclude that fine-grained text-based classification of social sciences publications at a subdisciplinary level is a hard task, for humans and machines alike. A combination of human expertise and machine learning is suggested as a way forward to improve the classification of social sciences documents.

A principled methodology for comparing relatedness measures for clustering publications

There are many different relatedness measures, based for instance on citation relations or textual similarity, that can be used to cluster scientific publications. We propose a principled methodology for evaluating the accuracy of clustering solutions obtained using these relatedness measures. We formally show that the proposed methodology has an important consistency property. The empirical analyses that we present are based on publications in the fields of cell biology, condensed matter physics, and economics. Using the BM25 text-based relatedness measure as the evaluation criterion, we find that bibliographic coupling relations yield more accurate clustering solutions than direct citation relations and cocitation relations. The so-called extended direct citation approach performs similarly to or slightly better than bibliographic coupling in terms of the accuracy of the resulting clustering solutions. The other way around, using a citation-based relatedness measure as evaluation criterion, BM25 turns out to yield more accurate clustering solutions than other text-based relatedness measures.

Enhancing direct citations: A comparison of relatedness measures for community detection in a large set of PubMed publications

The effects of enhancing direct citations, with respect to publication–publication relatedness measurement, by indirect citation relations (bibliographic coupling, cocitation, and extended direct citations) and text relations on clustering solution accuracy are analyzed. For comparison, we include each approach that is involved in the enhancement of direct citations. In total, we investigate the relative performance of seven approaches. To evaluate the approaches we use a methodology proposed by earlier research. However, the evaluation criterion used is based on MeSH, one of the most sophisticated publication-level classification schemes available. We also introduce an approach, based on interpolated accuracy values, by which overall relative clustering solution accuracy can be studied. The results show that the cocitation approach has the worst performance, and that the direct citations approach is outperformed by the other five investigated approaches. The extended direct citations approach has the best performance, followed by an approach in which direct citations are enhanced by the BM25 textual relatedness measure. An approach that combines direct citations with bibliographic coupling and cocitation performs slightly better than the bibliographic coupling approach, which in turn has a better performance than the BM25 approach.

Node2vec Representation for Clustering Journals and as A Possible Measure of Diversity

Purpose

To investigate the effectiveness of using node2vec on journal citation networks to represent journals as vectors for tasks such as clustering, science mapping, and journal diversity measure.

Design/methodology/approach

Node2vec is used in a journal citation network to generate journal vector representations.

Findings

1. Journals are clustered based on the node2vec trained vectors to form a science map. 2. The norm of the vector can be seen as an indicator of the diversity of journals. 3. Using node2vec trained journal vectors to determine the Rao-Stirling diversity measure leads to a better measure of diversity than that of direct citation vectors.

Research limitations

All analyses use citation data and only focus on the journal level.

Practical implications

Node2vec trained journal vectors embed rich information about journals, can be used to form a science map and may generate better values of journal diversity measures.

Originality/value

The effectiveness of node2vec in scientometric analysis is tested. Possible indicators for journal diversity measure are presented.

Dynamics of co-authorship and productivity across different fields of scientific research

We aimed to assess which factors correlate with collaborative behavior and whether such behavior associates with scientific impact (citations and becoming a principal investigator). We used the R index which is defined for each author as log(N_p)/log(I₁), where I₁ is the number of co-authors who appear in at least I₁ papers written by that author and N_p are his/her total papers. Higher R means lower collaborative behavior, i.e. not working much with others, or not collaborating repeatedly with the same co-authors. Across 249,054 researchers who had published ≥30 papers in 2000–2015 but had not published anything before 2000, R varied across scientific fields. Lower values of R (more collaboration) were seen in physics, medicine, infectious disease and brain sciences and higher values of R were seen for social science, computer science and engineering. Among the 9,314 most productive researchers already reaching N_p ≥ 30 and I₁ ≥ 4 by the end of 2006, R mostly remained stable for most fields from 2006 to 2015 with small increases seen in physics, chemistry, and medicine. Both US-based authorship and male gender were associated with higher values of R (lower collaboration), although the effect was small. Lower values of R (more collaboration) were associated with higher citation impact (h-index), and the effect was stronger in certain fields (physics, medicine, engineering, health sciences) than in others (brain sciences, computer science, infectious disease, chemistry). Finally, for a subset of 400 U.S. researchers in medicine, infectious disease and brain sciences, higher R (lower collaboration) was associated with a higher chance of being a principal investigator by 2016. Our analysis maps the patterns and evolution of collaborative behavior across scientific disciplines.

Chronological corpora curve clustering: From scientific corpora construction to knowledge dynamics discovery through word life-cycles clustering

Aim of this procedural method is to construct well-founded corpora of scientific literature, and, hence, to track the evolution of knowledge fields from the reconstruction and clustering of words’ life-cycles. The method contains:

•

an original selection process of relevant keywords involving the identification of relevant stems and stem n-grams through a matching with item lists of relevant glossaries;

•

several types of normalization of temporal trajectories of word raw frequencies

•

a properly customized clustering of word life-cycles, with a graphical extensive investigation of the best candidates for cluster number, to unveil the important dynamics and decipher the history of a scientific field.

Citation-based clustering of publications using CitNetExplorer and VOSviewer

Clustering scientific publications in an important problem in bibliometric research. We demonstrate how two software tools, CitNetExplorer and VOSviewer, can be used to cluster publications and to analyze the resulting clustering solutions. CitNetExplorer is used to cluster a large set of publications in the field of astronomy and astrophysics. The publications are clustered based on direct citation relations. CitNetExplorer and VOSviewer are used together to analyze the resulting clustering solutions. Both tools use visualizations to support the analysis of the clustering solutions, with CitNetExplorer focusing on the analysis at the level of individual publications and VOSviewer focusing on the analysis at an aggregate level. The demonstration provided in this paper shows how a clustering of publications can be created and analyzed using freely available software tools. Using the approach presented in this paper, bibliometricians are able to carry out sophisticated cluster analyses without the need to have a deep knowledge of clustering techniques and without requiring advanced computer skills.

Multiple Citation Indicators and Their Composite across Scientific Disciplines

Many fields face an increasing prevalence of multi-authorship, and this poses challenges in assessing citation metrics. Here, we explore multiple citation indicators that address total impact (number of citations, Hirsch H index [H]), co-authorship adjustment (Schreiber Hm index [Hm]), and author order (total citations to papers as single; single or first; or single, first, or last author). We demonstrate the correlation patterns between these indicators across 84,116 scientists (those among the top 30,000 for impact in a single year [2013] in at least one of these indicators) and separately across 12 scientific fields. Correlation patterns vary across these 12 fields. In physics, total citations are highly negatively correlated with indicators of co-authorship adjustment and of author order, while in other sciences the negative correlation is seen only for total citation impact and citations to papers as single author. We propose a composite score that sums standardized values of these six log-transformed indicators. Of the 1,000 top-ranked scientists with the composite score, only 322 are in the top 1,000 based on total citations. Many Nobel laureates and other extremely influential scientists rank among the top-1,000 with the composite indicator, but would rank much lower based on total citations. Conversely, many of the top 1,000 authors on total citations have had no single/first/last-authored cited paper. More Nobel laureates of 2011–2015 are among the top authors when authors are ranked by the composite score than by total citations, H index, or Hm index; 40/47 of these laureates are among the top 30,000 by at least one of the six indicators. We also explore the sensitivity of indicators to self-citation and alphabetic ordering of authors in papers across different scientific fields. Multiple indicators and their composite may give a more comprehensive picture of impact, although no citation indicator, single or composite, can be expected to select all the best scientists.

Citation indicators addressing total impact, co-authorship, and author positions offer complementary insights about impact. This article shows that a composite score including six citation indicators identifies extremely influential scientists better than single indicators.

Multiple citation indicators are used in science and scientific evaluation. With an increasing proportion of papers co-authored by many researchers, it is important to account for the relative contributions of different co-authors. We explored multiple citation indicators that address total impact, co-authorship adjustment, and author order (in particular, single, first, or last position authorships, since these positions suggest pivotal contributions to the work). We evaluated the top 30,000 scientists in 2013 based on each of six citation indicators (84,116 total scientists assessed) and also developed a composite score that combines the six indicators. Different scientists populated the top ranks when different indicators were used. Many Nobel laureates and other influential scientists rank among the top 1,000 with the composite indicator, but rank much lower based on total citations. Conversely, many of the top 1,000 authors on total citations had no single/first/last-authored cited paper. More Nobel laureates are among the top authors when authors are ranked by the composite score than by single indicators. Multiple indicators and their composite give a more comprehensive picture of impact, although no method can pick all the best scientists.

Clustering Scientific Publications Based on Citation Relations: A Systematic Comparison of Different Methods

Clustering methods are applied regularly in the bibliometric literature to identify research areas or scientific fields. These methods are for instance used to group publications into clusters based on their relations in a citation network. In the network science literature, many clustering methods, often referred to as graph partitioning or community detection techniques, have been developed. Focusing on the problem of clustering the publications in a citation network, we present a systematic comparison of the performance of a large number of these clustering methods. Using a number of different citation networks, some of them relatively small and others very large, we extensively study the statistical properties of the results provided by different methods. In addition, we also carry out an expert-based assessment of the results produced by different methods. The expert-based assessment focuses on publications in the field of scientometrics. Our findings seem to indicate that there is a trade-off between different properties that may be considered desirable for a good clustering of publications. Overall, map equation methods appear to perform best in our analysis, suggesting that these methods deserve more attention from the bibliometric community.