Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

Biodiversity conservation in the context of climate change: Facing challenges and management strategies

Biodiversity conservation amidst the uncertainty of climate change presents unique challenges that necessitate precise management strategies. The study reported here was aimed at refining understanding of these challenges and to propose specific, actionable management strategies. Employing a quantitative literature analysis, we meticulously examined 1268 research articles from the Web of Science database between 2005 and 2023. Through Cite Spaces and VOS viewer software, we conducted a bibliometric analysis and thematic synthesis to pinpoint emerging trends, key themes, and the geographical distribution of research efforts. Our methodology involved identifying patterns within the data, such as frequency of keywords, co-authorship networks, and citation analysis, to discern the primary focus areas within the field. This approach allowed us to distinguish between research concentration areas, specifically highlighting a predominant interest in Environmental Sciences Ecology (67.59 %) and Biodiversity Conservation (22.63 %). The identification of adaptive management practices and ecosystem services maintenance are central themes in the research from 2005 to 2023. Moreover, challenges such as understanding phenological shifts, invasive species dynamics, and anthropogenic pressures critically impact biodiversity conservation efforts. Our findings underscore the urgent need for precise, data-driven decision-making processes in the face of these challenges. Addressing the gaps identified, our study proposes targeted solutions, including the establishment of germplasm banks for at-risk species, the development of advanced genomic and microclimate models, and scenario analysis to predict and mitigate future conservation challenges. These strategies are aimed at enhancing the resilience of biodiversity against the backdrop of climate change through integrated, evidence-based approaches. By leveraging the compiled and analyzed data, this study offers a foundational framework for future research and practical action in biodiversity conservation strategies, demonstrating a path forward through detailed analysis and specified solutions.

The SCOPE framework - implementing ideals of responsible research assessment

Background

Research and researchers are heavily evaluated, and over the past decade it has become widely acknowledged that the consequences of evaluating the research enterprise and particularly individual researchers are considerable. This has resulted in the publishing of several guidelines and principles to support moving towards more responsible research assessment (RRA). To ensure that research evaluation is meaningful, responsible, and effective the International Network of Research Management Societies (INORMS) Research Evaluation Group created the SCOPE framework enabling evaluators to deliver on existing principles of RRA. SCOPE bridges the gap between principles and their implementation by providing a structured five-stage framework by which evaluations can be designed and implemented, as well as evaluated.

Methods

SCOPE is a step-by-step process designed to help plan, design, and conduct research evaluations as well as check effectiveness of existing evaluations. In this article, four case studies are presented to show how SCOPE has been used in practice to provide value-based research evaluation.

Results

This article situates SCOPE within the international work towards more meaningful and robust research evaluation practices and shows through the four case studies how it can be used by different organisations to develop evaluations at different levels of granularity and in different settings.

Conclusions

The article demonstrates that the SCOPE framework is rooted firmly in the existing literature. In addition, it is argued that it does not simply translate existing principles of RRA into practice, but provides additional considerations not always addressed in existing RRA principles and practices thus playing a specific role in the delivery of RRA. Furthermore, the use cases show the value of SCOPE across a range of settings, including different institutional types, sizes, and missions.

Anchoring effects in the assessment of papers: An empirical survey of citing authors

In our study, we have empirically studied the assessment of cited papers within the framework of the anchoring-and-adjustment heuristic. We are interested in the question whether the assessment of a paper can be influenced by numerical information that act as an anchor (e.g. citation impact). We have undertaken a survey of corresponding authors with an available email address in the Web of Science database. The authors were asked to assess the quality of papers that they cited in previous papers. Some authors were assigned to three treatment groups that receive further information alongside the cited paper: citation impact information, information on the publishing journal (journal impact factor) or a numerical access code to enter the survey. The control group did not receive any further numerical information. We are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation impact or journal impact), but also by numbers that are not related to quality, i.e. the access code. Our results show that the quality assessments of papers seem to depend on the citation impact information of single papers. The other information (anchors) such as an arbitrary number (an access code) and journal impact information did not play a (important) role in the assessments of papers. The results point to a possible anchoring bias caused by insufficient adjustment: it seems that the respondents assessed cited papers in another way when they observed paper impact values in the survey. We conclude that initiatives aiming at reducing the use of journal impact information in research evaluation either were already successful or overestimated the influence of this information.

Mapping the research trends of third language acquisition: A bibliometric analysis based on Scopus

L3 acquisition is gaining prominence in the academic community. The cardinal aim of this study is to conduct a bibliometric analysis of research on L3 acquisition. 425 documents from the Scopus database were analyzed with Bibliometrix. To gain a general and systematic overview of research on L3 acquisition, we grounded our study on three main levels of bibliometric analysis: performance analysis, collaboration analysis, and thematic map. By doing so, we identified the most influential sources, authors, affiliations, countries, and documents, the scientific network among different constituents, as well as the evolution of research trends. The results suggest that L3 acquisition has experienced three periods: initial phase (1984-2008), development phase (2009-2014), and burst phase (2015-2022). The results also indicate that: (1) The International Journal of Multilingualism is the most steady source contributing to this field. (2) Relevant contributors for each period are recognized, including established and emerging researchers. (3) European countries such as Spain, United Kingdom, Norway, Sweden, and Poland are at the forefront of publication. (4) Collaboration increases over time but is becoming increasingly oriented in European and Anglophone countries. (5) The research hotspots have shifted throughout time, encompass a broad range of fields, and continue to grow. The study results provide insights into the evolving trends of L3 acquisition studies and data to assist researchers in identifying research gaps.

Citizen Environmental Behavior From the Perspective of Psychological Distance Based on a Visual Analysis of Bibliometrics and Scientific Knowledge Mapping

Global warming and other climate issues seriously threaten global sustainable development. As citizen environmental behavior can have a positive impact on the environment, it is of great theoretical significance and practical reference value to study the impact of psychological distance theory on citizen environmental behavior. This study obtained 2,633 related studies from 1980 to 2020 from the Web of Science as research objects, and used CiteSpace, VOSviewer, Netdraw, and other software programs to perform a bibliometric analysis, which can show the complex relationship implied in the citation, intuitively grasp the development context and hot frontier in this field, and help scholars better study citizen environmental behavior. The results and conclusions are as follows. (1) The related research covered three periods: infancy, growth, and outbreak. (2) Chronologically, the relevant research evolved from building models to analyze the impact of environmental enrichment on human environmental behavior, to studying the motivation behind citizen environmental behavior, and finally to paying attention to sustainable development. (3) A cluster analysis of high-frequency keywords identified three research hotspots: “drivers of psychological distance of environmental change perception on citizen environmental behavior,” “impact of social distance on adolescents’ behavior,” and “construal level theory and citizen green behavior.” Based on these findings, possible future research directions were identified, including changing from a single theory to a combination of multiple theories to comprehensively study citizen environmental protection behavior; analyzing the motivation of citizen environmental behavior and summarizing the general motivation of environmental behavior according to its internal relationship; and determining how to narrow the global psychological distance, strengthen the awareness of the community of human destiny, and explore the establishment of an efficient global climate cooperation mechanism.

Anchoring effects in the assessment of papers: The proposal for an empirical survey of citing authors

In our planned study, we shall empirically study the assessment of cited papers within the framework of the anchoring-and-adjustment heuristic. We are interested in the question whether citation decisions are (mainly) driven by the quality of cited references. The design of our study is oriented towards the study by Teplitskiy, Duede [10]. We shall undertake a survey of corresponding authors with an available email address in the Web of Science database. The authors are asked to assess the quality of papers that they cited in previous papers. Some authors will be assigned to three treatment groups that receive further information alongside the cited paper: citation information, information on the publishing journal (journal impact factor), or a numerical access code to enter the survey. The control group will not receive any further numerical information. In the statistical analyses, we estimate how (strongly) the quality assessments of the cited papers are adjusted by the respondents to the anchor value (citation, journal, or access code). Thus, we are interested in whether possible adjustments in the assessments can not only be produced by quality-related information (citation or journal), but also by numbers that are not related to quality, i.e. the access code. The results of the study may have important implications for quality assessments of papers by researchers and the role of numbers, citations, and journal metrics in assessment processes.

Big Data Research in Fighting COVID-19: Contributions and Techniques

The COVID-19 pandemic has induced many problems in various sectors of human life. After more than one year of the pandemic, many studies have been conducted to discover various technological innovations and applications to combat the virus that has claimed many lives. The use of Big Data technology to mitigate the threats of the pandemic has been accelerated. Therefore, this survey aims to explore Big Data technology research in fighting the pandemic. Furthermore, the relevance of Big Data technology was analyzed while technological contributions to five main areas were highlighted. These include healthcare, social life, government policy, business and management, and the environment. The analytical techniques of machine learning, deep learning, statistics, and mathematics were discussed to solve issues regarding the pandemic. The data sources used in previous studies were also presented and they consist of government officials, institutional service, IoT generated, online media, and open data. Therefore, this study presents the role of Big Data technologies in enhancing the research relative to COVID-19 and provides insights into the current state of knowledge within the domain and references for further development or starting new studies are provided.

Interpreting Bibliometric Data

Many academic analyses of good practice in the use of bibliometric data address only technical aspects and fail to account for and appreciate user requirements, expectations, and actual practice. Bibliometric indicators are rarely the only evidence put before any user group. In the present state of knowledge, it is more important to consider how quantitative evaluation can be made simple, transparent, and readily understood than it is to focus unduly on precision, accuracy, or scholarly notions of purity. We discuss how the interpretation of 'performance' from a presentation using accurate but summary bibliometrics can change when iterative deconstruction and visualization of the same dataset is applied. From the perspective of a research manager with limited resources, investment decisions can easily go awry at governmental, funding program, and institutional levels. By exploring select real-life data samples we also show how the specific composition of each dataset can influence interpretive outcomes.

How can citation impact in bibliometrics be normalized? A new approach combining citing-side normalization and citation percentiles

Since the 1980s, many different methods have been proposed to field-normalize citations. In this study, an approach is introduced that combines two previously introduced methods: citing-side normalization and citation percentiles. The advantage of combining two methods is that their advantages can be integrated in one solution. Based on citing-side normalization, each citation is field weighted and, therefore, contextualized in its field. The most important advantage of citing-side normalization is that it is not necessary to work with a specific field categorization scheme for the normalization procedure. The disadvantages of citing-side normalization—the calculation is complex and the numbers are elusive—can be compensated for by calculating percentiles based on weighted citations that result from citing-side normalization. On the one hand, percentiles are easy to understand: They are the percentage of papers published in the same year with a lower citation impact. On the other hand, weighted citation distributions are skewed distributions with outliers. Percentiles are well suited to assigning the position of a focal paper in such distributions of comparable papers. The new approach of calculating percentiles based on weighted citations is demonstrated in this study on the basis of a citation impact comparison between several countries.

An evaluation of percentile measures of citation impact, and a proposal for making them better

Percentiles are statistics pointing to the standing of a paper’s citation impact relative to other papers in a given citation distribution. Percentile Ranks (PRs) often play an important role in evaluating the impact of researchers, institutions, and similar lines of study. Because PRs are so important for the assessment of scholarly impact, and because citations differ greatly across time and fields, various percentile approaches have been proposed to time- and field-normalize citations. Unfortunately, current popular methods often face significant problems in time- and field-normalization, including when papers are assigned to multiple fields or have been published by more than one unit (e.g., researchers or countries). They also face problems for estimating citation counts for pre-defined PRs (e.g., the 90th PR). We offer a series of guidelines and procedures that, we argue, address these problems and others and provide a superior means to make the use of percentile methods more accurate and informative. In particular, we introduce two approaches, CP-IN and CP-EX, that should be preferred in bibliometric studies because they consider the complete citation distribution and can be accurately interpreted. Both approaches are based on cumulative frequencies in percentages (CPs). The paper further shows how bar graphs and beamplots can present PRs in a more meaningful and accurate manner.

Does the hα-index reinforce the Matthew effect in science? The introduction of agent-based simulations into scientometrics

Recently, Hirsch (2019a) proposed a new variant of the h-index called the hα-index. The hα-index was criticized by Leydesdorff, Bornmann, and Opthof (2019) . One of their most important points is that the index reinforces the Matthew effect in science. The Matthew effect was defined by Merton (1968) as follows: “the Matthew effect consists in the accruing of greater increments of recognition for particular scientific contributions to scientists of considerable repute and the withholding of such recognition from scientists who have not yet made their mark” (p. 58). We follow up on the point about the Matthew effect in the current study by using a recently developed Stata command (h_index) and R package (hindex), which can be used to simulate h-index and hα-index applications in research evaluation. The user can investigate under which conditions hα reinforces the Matthew effect. The results of our study confirm what Leydesdorff et al. (2019) expected: The hα-index reinforces the Matthew effect. This effect can be intensified if strategic behavior of the publishing scientists and cumulative advantage effects are additionally considered in the simulation.

Bibliometrics-based decision trees (BBDTs) based on bibliometrics-based heuristics (BBHs): Visualized guidelines for the use of bibliometrics in research evaluation

Fast-and-frugal heuristics are simple strategies that base decisions on only a few predictor variables. In so doing, heuristics may not only reduce complexity but also boost the accuracy of decisions, their speed, and transparency. In this paper, bibliometrics-based decision trees (BBDTs) are introduced for research evaluation purposes. BBDTs visualize bibliometrics-based heuristics (BBHs), which are judgment strategies solely using publication and citation data. The BBDT exemplar presented in this paper can be used as guidance to find an answer on the question in which situations simple indicators such as mean citation rates are reasonable and in which situations more elaborated indicators (i.e., [sub-]field-normalized indicators) should be applied.