Discussing some basic critique on Journal Impact Factors: revision of earlier comments

Correlations between the selection of topics by news media and scientific journals

The aim of this study is to reveal a robust correlation between the amount of attention international journalism devotes to scientific papers and the amount of attention scientific journals devote to the respective topics. Using a Mainstream-Media-Score (MSM) ≥ 100 (which we regard as an indicator for news media attention) from the altmetrics provider Altmetric, we link 983 research articles with 185,166 thematically similar articles from the PubMed database (which we use to operationalize attention from scientific journals). The method we use is to test whether there is a concomitant increase in scientific attention after a research article has received popular media coverage. To do so, we compare the quotient of the number of thematically similar articles published in scientific journals during the period before and after the publication of an MSM ≥ 100 article. Our main result shows that in 59 percent of cases, more thematically similar articles were published in scientific journals after a scientific paper received noteworthy news media coverage than before (p < 0.01). In this context, we neither found significant differences between various types of scientific journal (p = 0.3) nor between scientific papers that were originally published in renowned opinion-leading journals or in less renowned, non-opinion-leading journals (p = 0.1). Our findings indicate a robust correlation between the choice of topics in the mass media and in research. However, our study cannot clarify whether this correlation occurs because researchers and/or scientific journals are oriented towards public relevance (publicity effect) or whether the correlation is due to the parallelism of relevance attributions in quality journalism and research (earmark hypothesis). We infer that topics of social relevance are (more) likely to be picked up by popular media as well as by scientific journals. Altogether, our study contributes new empirical findings to the relationship between topic selection in journalism and in research.

Early Impact of the Western Journal of Emergency Medicine CDEM/CORD Special Issue in Educational Research & Practice

Introduction

In 2015, with a stated goal of disseminating best teaching practices and developing a community of educational scholars, the Council of Emergency Medicine Directors (CORD) and the Clerkship Directors of Emergency Medicine (CDEM) created an annual Special Issue in Educational Research and Practice (Special Issue) in cooperation with the Western Journal of Emergency Medicine. The intention of this study was to analyze the impact of this effort to date.

Methods

Bibliometric data was gathered on all four special issues, 2015–2019, from the Web of Science and then verified with the eScholarship website. Authorship, academic affiliation, date published, article type, and format were tabulated for descriptive analysis. Using metrics from Google Scholar, alternative scholarly impact metrics (altmetrics), and the eScholarship website, the authors identified top articles and grouped them into themes.

Results

Of the 136 articles included in the first four years of the Special Issue, 126 represented peer-reviewed publications with an overall acceptance rate of 25.0% (126/505). Authors from this cohort represented 103 of the 182 (56.6%) Accreditation Council for Graduate Medical Education (ACGME) programs in existence at the time of the inaugural issue. Multi-institutional studies represented 34.9% (44/126) of the peer-reviewed publications. Traditional and alternative publication metrics are reported to assess the impact of articles from the Special Issues.

Conclusion

The Special Issue is a proven outlet to share best practices, innovations, and research related to education. Additionally, the infrastructure of this process promotes the development of individual faculty and a community of teaching scholars.

Redefining the pharmacology and pharmacy subject category in the journal citation reports using medical subject headings (MeSH)

Background The Journal Citation Reports (JCR) Pharmacology and Pharmacy subject category is heterogeneous. The inclusion of journals with basic and clinical scopes, which have different citation patterns, compromises comparability of impact factors among journals within the category. Objective To subdivide the Pharmacology and Pharmacy category into basic pharmacology, clinical pharmacology, and pharmacy based on the analyses of Medical Subject Headings (MeSH) as a proxy of journals' scopes. Setting JCR. Method All articles, and respective MeSH, published in 2013, 2014, and 2015 in all journals included in the 2014 JCR Pharmacology and Pharmacy category were retrieved from PubMed. Several models using a combination of the 14 MeSH categories and specific MeSH tree branches were tested using hierarchical cluster analysis. Main outcome measure Distribution of journals across the subcategories of the JCR Pharmacology and Pharmacy subject category. Results A total of 107,847 articles from 214 journals were included. Nine different models combining the MeSH categories M (Persons) and N (Health Care) with specific MeSH tree branches (selected ad-hoc) and Pharmacy-specific MeSH (identified in previous research) consistently grouped 142 journals (66.4%) in homogeneous groups reflecting their basic and clinical pharmacology, and pharmacy scopes. Ultimately, journals were clustered into: 150 in basic pharmacology, 43 in clinical pharmacology, 16 in basic pharmacology and clinical pharmacology, and 5 in pharmacy. Conclusion The reformulation of the Pharmacology and Pharmacy category into three categories was demonstrated by the consistent results obtained from testing nine different clustering models using the MeSH terms assigned to their articles.

Impact factor of medical education journals and recently developed indices: Can any of them support academic promotion criteria?

Journal Impact Factor (JIF) has been used in assessing scientific journals. Other indices, h- and g-indices and Article Influence Score (AIS), have been developed to overcome some limitations of JIF. The aims of this study were, first, to critically assess the use of JIF and other parameters related to medical education research, and second, to discuss the capacity of these indices in assessing research productivity as well as their utility in academic promotion. The JIF of 16 medical education journals from 2000 to 2011 was examined together with the research evidence about JIF in assessing research outcomes of medical educators. The findings were discussed in light of the nonnumerical criteria often used in academic promotion. In conclusion, JIF was not designed for assessing individual or group research performance, and it seems unsuitable for such purposes. Although the g- and h-indices have demonstrated promising outcomes, further developments are needed for their use as academic promotion criteria. For top academic positions, additional criteria could include leadership, evidence of international impact, and contributions to the advancement of knowledge with regard to medical education.

Journal Impact Factor: Do the Numerator and Denominator Need Correction?

To correct the incongruence of document types between the numerator and denominator in the traditional impact factor (IF), we make a corresponding adjustment to its formula and present five corrective IFs: IFTotal/Total, IFTotal/AREL, IFAR/AR, IFAREL/AR, and IFAREL/AREL. Based on a survey of researchers in the fields of ophthalmology and mathematics, we obtained the real impact ranking of sample journals in the minds of peer experts. The correlations between various IFs and questionnaire score were analyzed to verify their journal evaluation effects. The results show that it is scientific and reasonable to use five corrective IFs for journal evaluation for both ophthalmology and mathematics. For ophthalmology, the journal evaluation effects of the five corrective IFs are superior than those of traditional IF: the corrective effect of IFAR/AR is the best, IFAREL/AR is better than IFTotal/Total, followed by IFTotal/AREL, and IFAREL/AREL. For mathematics, the journal evaluation effect of traditional IF is superior than those of the five corrective IFs: the corrective effect of IFTotal/Total is best, IFAREL/AR is better than IFTotal/AREL and IFAREL/AREL, and the corrective effect of IFAR/AR is the worst. In conclusion, not all disciplinary journal IF need correction. The results in the current paper show that to correct the IF of ophthalmologic journals may be valuable, but it seems to be meaningless for mathematic journals.

Bibliometric analysis of the top-cited gastroenterology and hepatology articles

OBJECTIVE

To identify the top-cited articles in gastroenterology and hepatology, and analyse their characteristics.

METHODS

Two searches were conducted in the Science Citation Index Expanded database; a search of 69 journals under the category 'Gastroenterology and Hepatology' (list A) and a keyword search of all journals (list B). The search results were analysed and the inter-rater coefficient of agreement between evaluators was measured using Cohen κ.

RESULTS

The number of citations varied from 1049 to 2959 in list A and from 1929 to 5500 in list B. In both lists, the majority of articles were research papers. No significant correlations were found between the number of citations and the number of years since publication (R(2)=0.00992, p=0.473 and R(2)=0.00202, p=0.757, respectively). However, the mean number of citations of papers published before the year 2000 was lower than those published after 2000 (36.70 ± 19.31 vs 106.03 ± 39.22). No correlation was found between number of authors and the number of citations (R(2)=0.04352, p=0.130), but strong correlations were found between the number of institutes involved or number of countries and the number of citations (R(2)=0.275, p<0.001 and R(2)=0.16181, p=0.003, respectively). Females were under-represented in authorship (45 vs 254, p=0.004). Only 21 papers (of 54) in list A were supported by grants. No correlation was found between number of grants received and the number of citations (R(2)=0.02573, p=0.247). The inter-rater agreement between evaluators had a Cohen κ coefficient 0.76-0.84.

CONCLUSIONS

Top-cited articles were not only published in highly ranked journals specialising in Gastroenterology and Hepatology but also in 14 journals not specialised in this field. The number of citations correlated with the number of institutes and the number of countries involved but not with the number of grants received or the number of authors. Females were under-represented in the authorship.

High-ranked social science journal articles can be identified from early citation information

Do citations accumulate too slowly in the social sciences to be used to assess the quality of recent articles? I investigate whether this is the case using citation data for all articles in economics and political science published in 2006 and indexed in the Web of Science. I find that citations in the first two years after publication explain more than half of the variation in cumulative citations received over a longer period. Journal impact factors improve the correlation between the predicted and actual future ranks of journal articles when using citation data from 2006 alone but the effect declines sharply thereafter. Finally, more than half of the papers in the top 20% in 2012 were already in the top 20% in the year of publication (2006).

The first three years of the Journal of Global Health: Assessing the impact

The Journal of Global Health (JoGH) is three years old. To assess its impact, we analysed online access to JoGH's articles using PubMed Central and Google Analytics tools. Moreover, we tracked citations that JoGH received in 2013 using ISI Web of Knowledge^SM and Google Scholar® tools. The 66 items (articles, viewpoints and editorials) published between June 2011 and December 2013 were accessed more than 50 000 times during 2013, from more than 160 countries of the world. Seven among the 13 most accessed papers were focused on global, regional and national epidemiological estimates of important infectious diseases. JoGH articles published in 2011 and 2012 received 77 citations in Journal Citation Reports® (JCR)–indexed journals in 2013 to 24 original research articles, setting our first, unofficial impact factor at 3.208. In addition, JoGH received 11 citations during 2013 to its 12 original research papers published during 2013, resulting in an immediacy index of 0.917. The number of external, non–commissioned submissions that we consider to be of high quality is continuously increasing, leading to current JoGH's rejection rate of about 80%. The current citation analysis raises favourable expectations for the JoGH's overall impact on the global health community in future years.