Researchers' attitudes towards the h-index on Twitter 2007-2020: criticism and acceptance

The h-index is an indicator of the scientific impact of an academic publishing career. Its hybrid publishing/citation nature and inherent bias against younger researchers, women, people in low resourced countries, and those not prioritizing publishing arguably give it little value for most formal and informal research evaluations. Nevertheless, it is well-known by academics, used in some promotion decisions, and is prominent in bibliometric databases, such as Google Scholar. In the context of this apparent conflict, it is important to understand researchers’ attitudes towards the h-index. This article used public tweets in English to analyse how scholars discuss the h-index in public: is it mentioned, are tweets about it positive or negative, and has interest decreased since its shortcomings were exposed? The January 2021 Twitter Academic Research initiative was harnessed to download all English tweets mentioning the h-index from the 2006 start of Twitter until the end of 2020. The results showed a constantly increasing number of tweets. Whilst the most popular tweets unapologetically used the h-index as an indicator of research performance, 28.5% of tweets were critical of its simplistic nature and others joked about it (8%). The results suggest that interest in the h-index is still increasing online despite scientists willing to evaluate the h-index in public tending to be critical. Nevertheless, in limited situations it may be effective at succinctly conveying the message that a researcher has had a successful publishing career.

Pot, kettle: Nonliteral titles aren’t (natural) science

Researchers may be tempted to attract attention through poetic titles for their publications, but would this be mistaken in some fields? Although poetic titles are known to be common in medicine, it is not clear whether the practice is widespread elsewhere. This article investigates the prevalence of poetic expressions in journal article titles from 1996–2019 in 3.3 million articles from all 27 Scopus broad fields. Expressions were identified by manually checking all phrases with at least five words that occurred at least 25 times, finding 149 stock phrases, idioms, sayings, literary allusions, film names, and song titles or lyrics. The expressions found are most common in the social sciences and the humanities. They are also relatively common in medicine, but almost absent from engineering and the natural and formal sciences. The differences may reflect the less hierarchical and more varied nature of the social sciences and humanities, where interesting titles may attract an audience. In engineering, natural science, and formal science fields, authors should take extra care with poetic expressions in case their choice is judged inappropriate. This includes interdisciplinary research overlapping these areas. Conversely, reviewers of interdisciplinary research involving the social sciences should be more tolerant of poetic license.

Coronavirus research before 2020 is more relevant than ever, especially when interpreted for COVID-19

The speed with which biomedical specialists were able to identify and characterize COVID-19 was partly due to prior research with other coronaviruses. Early epidemiological comparisons with Severe Acute Respiratory Syndrome (SARS) and Middle East Respiratory Syndrome (MERS), also made it easier to predict COVID-19’s likely spread and lethality. This article assesses whether academic interest in prior coronavirus research has translated into interest in the primary source material, using Mendeley reader counts for early academic impact evidence. The results confirm that SARS and MERS research in 2008–2017 experienced anomalously high increases in Mendeley readers in April–May 2020. Nevertheless, studies learning COVID-19 lessons from SARS and MERS or using them as a benchmark for COVID-19 have generated much more academic interest than primary studies of SARS or MERS. Thus, research that interprets prior relevant research for new diseases when they are discovered seems to be particularly important to help researchers to understand its implications in the new context.

A thematic analysis of highly retweeted early COVID-19 tweets: consensus, information, dissent and lockdown life

Purpose

Public attitudes towards COVID-19 and social distancing are critical in reducing its spread. It is therefore important to understand public reactions and information dissemination in all major forms, including on social media. This article investigates important issues reflected on Twitter in the early stages of the public reaction to COVID-19.

Design/methodology/approach

A thematic analysis of the most retweeted English-language tweets mentioning COVID-19 during March 10–29, 2020.

Findings

The main themes identified for the 87 qualifying tweets accounting for 14 million retweets were: lockdown life; attitude towards social restrictions; politics; safety messages; people with COVID-19; support for key workers; work; and COVID-19 facts/news.

Research limitations/implications

Twitter played many positive roles, mainly through unofficial tweets. Users shared social distancing information, helped build support for social distancing, criticised government responses, expressed support for key workers and helped each other cope with social isolation. A few popular tweets not supporting social distancing show that government messages sometimes failed.

Practical implications

Public health campaigns in future may consider encouraging grass roots social web activity to support campaign goals. At a methodological level, analysing retweet counts emphasised politics and ignored practical implementation issues.

Originality/value

This is the first qualitative analysis of general COVID-19-related retweeting.

COVID-19 publications: Database coverage, citations, readers, tweets, news, Facebook walls, Reddit posts

The COVID-19 pandemic requires a fast response from researchers to help address biological, medical, and public health issues to minimize its impact. In this rapidly evolving context, scholars, professionals, and the public may need to identify important new studies quickly. In response, this paper assesses the coverage of scholarly databases and impact indicators during March 21, 2020 to April 18, 2020. The rapidly increasing volume of research is particularly accessible through Dimensions, and less through Scopus, the Web of Science, and PubMed. Google Scholar’s results included many false matches. A few COVID-19 papers from the 21,395 in Dimensions were already highly cited, with substantial news and social media attention. For this topic, in contrast to previous studies, there seems to be a high degree of convergence between articles shared in the social web and citation counts, at least in the short term. In particular, articles that are extensively tweeted on the day first indexed are likely to be highly read and relatively highly cited 3 weeks later. Researchers needing wide scope literature searches (rather than health-focused PubMed or medRxiv searches) should start with Dimensions (or Google Scholar) and can use tweet and Mendeley reader counts as indicators of likely importance.

A gender equality paradox in academic publishing: Countries with a higher proportion of female first-authored journal articles have larger first-author gender disparities between fields

Current attempts to address the shortfall of female researchers in science, technology, engineering, and mathematics (STEM) have not yet succeeded, despite other academic subjects having female majorities. This article investigates the extent to which gender disparities are subject-wide or nation-specific by a first-author gender comparison of 30 million articles from all 27 Scopus broad fields within the 31 countries with the most Scopus-indexed articles 2014–2018. The results show overall and geocultural patterns as well as individual national differences. Almost half of the subjects were always more male (seven; e.g., Mathematics) or always more female (six; e.g., Immunology & Microbiology) than the national average. A strong overall trend (Spearman correlation 0.546) is for countries with a higher proportion of female first-authored research to also have larger differences in gender disparities between fields (correlation 0.314 for gender ratios). This confirms the international gender equality paradox previously found for degree subject choices: Increased gender equality overall associates with moderately greater gender differentiation between subjects. This is consistent with previous United States-based claims that gender differences in academic careers are partly due to (socially constrained) gender differences in personal preferences. Radical solutions may therefore be needed for some STEM subjects to overcome gender disparities.

Greater female first author citation advantages do not associate with reduced or reducing gender disparities in academia

Ongoing problems attracting women into many Science, Technology, Engineering and Mathematics (STEM) subjects have many potential explanations. This article investigates whether the possible undercitation of women associates with lower proportions of, or increases in, women in a subject. It uses six million articles published in 1996–2012 across up to 331 fields in six mainly English-speaking countries: Australia, Canada, Ireland, New Zealand, the United Kingdom and the United States. The proportion of female first- and last-authored articles in each year was calculated and 4,968 regressions were run to detect first-author gender advantages in field normalized article citations. The proportion of female first authors in each field correlated highly between countries and the female first-author citation advantages derived from the regressions correlated moderately to strongly between countries, so both are relatively field specific. There was a weak tendency in the United States and New Zealand for female citation advantages to be stronger in fields with fewer women, after excluding small fields, but there was no other association evidence. There was no evidence of female citation advantages or disadvantages to be a cause or effect of changes in the proportions of women in a field for any country. Inappropriate uses of career-level citations are a likelier source of gender inequities.

All downhill from the PhD? The typical impact trajectory of U.S. academic careers

Within academia, mature researchers tend to be more senior, but do they also tend to write higher impact articles? This article assesses long-term publishing (16+ years) United States (U.S.) researchers, contrasting them with shorter-term publishing researchers (1, 6, or 10 years). A long-term U.S. researcher is operationalized as having a first Scopus-indexed journal article in exactly 2001 and one in 2016–2019, with U.S. main affiliations in their first and last articles. Researchers publishing in large teams (11+ authors) were excluded. The average field and year normalized citation impact of long- and shorter-term U.S. researchers’ journal articles decreases over time relative to the national average, with especially large falls for the last articles published, which may be at least partly due to a decline in self-citations. In many cases researchers start by publishing above U.S. average citation impact research and end by publishing below U.S. average citation impact research. Thus, research managers should not assume that senior researchers will usually write the highest impact papers.

Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000Research post-publication open peer review publishing model

As part of moves towards open knowledge practices, making peer review open is cited as a way to enable fuller scrutiny and transparency of assessments around research. There are now many flavours of open peer review in use across scholarly publishing, including where reviews are fully attributable and the reviewer is named. This study examines whether there is any evidence of bias in two areas of common critique of open, non-anonymous (named) peer review – and used in the post-publication, peer review system operated by the open-access scholarly publishing platform F1000Research. First, is there evidence of potential bias where a reviewer based in a specific country assesses the work of an author also based in the same country? Second, are reviewers influenced by being able to see the comments and know the origins of a previous reviewer? Based on over 4 years of open peer review data, we found some weak evidence that being based in the same country as an author may influence a reviewer’s decision, while there was insufficient evidence to conclude that being able to read an existing published review prior to submitting a review encourages conformity. Thus, while immediate publishing of peer review reports appears to be unproblematic, caution may be needed when selecting same-country reviewers in open systems if other studies confirm these results.

How does nursing research differ internationally? A bibliometric analysis of six countries

BACKGROUND

International nursing research comparisons can give a new perspective on a nation's output by identifying strengths and weaknesses.

AIM

This article compares strengths in nursing research between six mainly English-speaking nations (Australia, Canada, Ireland, New Zealand, United Kingdom and United States).

METHODS

Journal authorship (percentage of first authorship by nationality) and article keywords were compared for Scopus-indexed journal articles 2008-2018. Three natural language processing strategies were assessed for identifying statistically significant international differences in the use of keywords or phrases.

RESULTS

Journal author nationality was not a good indicator of international differences in research specialisms, but keyword and phrase differences were more promising especially if both are used. For this, the part of speech tagging and lemmatisation text processing strategies were helpful but not named entity recognition. The results highlight aspects of nursing research that were absent in some countries, such as papers about nursing administration and management.

CONCLUSION

Researchers outside the United States should consider the importance of researching specific patient groups, diseases, treatments, skills, research methods and social perspectives for unresearched gaps with national relevance. From a methods perspective, keyword and phrase differences are useful to reveal international differences in nursing research topics.

Automatically detecting open academic review praise and criticism

Purpose

Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.

Design/methodology/approach

PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.

Findings

PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.

Originality/value

PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.

Google Books, Scopus, Microsoft Academic and Mendeley for impact assessment of doctoral dissertations: A multidisciplinary analysis of the UK

A research doctorate normally culminates in publishing a dissertation reporting a substantial body of novel work. In the absence of a suitable citation index, this article explores the relative merits of alternative methods for the large-scale assessment of dissertation impact, using 150,740 UK doctoral dissertations from 2009–2018. Systematic methods for this were designed for Google Books, Scopus, Microsoft Academic, and Mendeley. Less than 1 in 8 UK doctoral dissertations had at least one Scopus (12%), Microsoft Academic (11%), or Google Books citation (9%), or at least one Mendeley reader (5%). These percentages varied substantially by subject area and publication year. Google Books citations were more common in the Arts and Humanities (18%), whereas Scopus and Microsoft Academic citations were more numerous in Engineering (24%). In the Social Sciences, Google Books (13%) and Scopus (12%) citations were important and in Medical Sciences, Scopus and Microsoft Academic citations to dissertations were rare (6%). Few dissertations had Mendeley readers (from 3% in Science to 8% in the Social Sciences) and further analysis suggests that Google Scholar finds more citations, but does not report information about all dissertations within a repository and is not a practical tool for large-scale impact assessment

Gender differences in citation impact for 27 fields and six English-speaking countries 1996–2014

Initiatives addressing the lack of women in many academic fields, and the general lack of senior women, need to be informed about the causes of any gender differences that may affect career progression, including citation impact. Previous research about gender differences in journal article citation impact has found the direction of any difference to vary by country and field, but has usually avoided discussions of the magnitude and wider significance of any differences and has not been systematic in terms of fields and/or time. This study investigates differences in citation impact between male and female first-authored research for 27 broad fields and six large English-speaking countries (Australia, Canada, Ireland, New Zealand, the UK, and the USA) from 1996 to 2014. The results show an overall female first author citation advantage, although in most broad fields it is reversed in all countries for some years. International differences include Medicine having a female first author citation advantage for all years in Australia, but a male citation advantage for most years in Canada. There was no general trend for the gender difference to increase or decrease over time. The average effect size is small, however, and unlikely to have a substantial influence on overall gender differences in researcher careers.

How common are explicit research questions in journal articles?

Although explicitly labeled research questions seem to be central to some fields, others do not need them. This may confuse authors, editors, readers, and reviewers of multidisciplinary research. This article assesses the extent to which research questions are explicitly mentioned in 17 out of 22 areas of scholarship from 2000 to 2018 by searching over a million full-text open access journal articles. Research questions were almost never explicitly mentioned (under 2%) by articles in engineering and physical, life, and medical sciences, and were the exception (always under 20%) for the broad fields in which they were least rare: computing, philosophy, theology, and social sciences. Nevertheless, research questions were increasingly mentioned explicitly in all fields investigated, despite a rate of 1.8% overall (1.1% after correcting for irrelevant matches). Other terminology for an article’s purpose may be more widely used instead, including aims, objectives, goals, hypotheses, and purposes, although no terminology occurs in a majority of articles in any broad field tested. Authors, editors, readers, and reviewers should therefore be aware that the use of explicitly labeled research questions or other explicit research purpose terminology is non-standard in most or all broad fields, although it is becoming less rare.

Large publishing consortia produce higher citation impact research but coauthor contributions are hard to evaluate

This paper introduces a simple agglomerative clustering method to identify large publishing consortia with at least 20 authors and 80% shared authorship between articles. Based on Scopus journal articles from 1996–2018, under these criteria, nearly all (88%) of the large consortia published research with citation impact above the world average, with the exceptions being mainly the newer consortia, for which average citation counts are unreliable. On average, consortium research had almost double (1.95) the world average citation impact on the log scale used (Mean Normalised Log Citation Score). At least partial alphabetical author ordering was the norm in most consortia. The 250 largest consortia were for nuclear physics and astronomy, involving expensive equipment, and for predominantly health-related issues in genomics, medicine, public health, microbiology and neuropsychology. For the health-related issues, except for the first and last few authors, authorship seem to primarily indicate contributions to the shared project infrastructure necessary to gather the raw data. It is impossible for research evaluators to identify the contributions of individual authors in the huge alphabetical consortia of physics and astronomy and problematic for the middle and end authors of health-related consortia. For small-scale evaluations, authorship contribution statements could be used when available.

Mendeley reader counts for US computer science conference papers and journal articles

Although bibliometrics are normally applied to journal articles when used to support research evaluations, conference papers are at least as important in fast-moving computing-related fields. It is therefore important to assess the relative advantages of citations and altmetrics for computing conference papers to make an informed decision about which, if any, to use. This paper compares Scopus citations with Mendeley reader counts for conference papers and journal articles that were published between 1996 and 2018 in 11 computing fields and that had at least one US author. The data showed high correlations between Scopus citation counts and Mendeley reader counts in all fields and most years, but with few Mendeley readers for older conference papers and few Scopus citations for new conference papers and journal articles. The results therefore suggest that Mendeley reader counts have a substantial advantage over citation counts for recently published conference papers due to their greater speed, but are unsuitable for older conference papers.

Author gender differences in psychology citation impact 1996-2018

Academic psychology in the USA is a gender success story in terms of overturning its early male dominance but there are still relatively few senior female psychology researchers. To assess whether there are gender differences in citation impact that might help to explain either of these trends, this study investigates psychology articles since 1996. Seven out of eight Scopus psychology categories had a majority of female first-authored journal articles by 2018. From regression analyses of first and last author gender and team size, female first authors associate with a slightly higher average citation impact, but extra authors have a 10 times stronger association with higher average citation impact. Last author gender has little association with citation impact. Female first authors are more likely to be in larger teams and if team size is attributed to the first author's work, then their apparent influence of female first authors on citation impact doubles. While gender differences in average citation impact are too small to account for gender-related trends in academic psychology, they warn that male-dominated citation-based ranking lists of psychologists do not reflect the state of psychology research today.