Questioning the Impact of the Impact Factor. A Brief Review and Future Directions

Assessing the impact of history of medicine research: A scientometric and altmetric analysis

Background and Aims

After conducting a comprehensive literature search of two medical electronic databases, PubMed and Embase, as well as two citation databases, Web of Science Core Collections (WoS) and Scopus, we aimed to conduct an Altmetric and Scientometric analysis of the History of Medicine literature in medical research.

Methods

The following software tools were used for analyzing the retrieved records from PubMed and Embase databases and conducting a collaboration analysis to identify the countries involved in scientific medical papers, as well as clustering keywords to reveal the trend of History of Medicine research for the future. These software tools (VOSviewer 1.6.18 and Spss 16) allowed the researchers to visualize bibliometric networks, perform statistical analysis, and identify patterns and trends in the data.

Results

Our analysis revealed 53,771 records from PubMed and 54,405 records from EMBASE databases retrieved in the field of History of Medicine by 105,286 contributed authors in WoS. We identified 157 countries that collaborated on scientific medical papers. By clustering 59,995 keywords, we were able to reveal the trend of History of Medicine research for the future. Our findings showed a positive association between traditional bibliometrics and social media metrics such as the Altmetric Attention Score in the History of Medicine literature (p < 0.05).

Conclusion

Sharing research findings of articles in social scientific networks will increase the visibility of scientific works in History of Medicine research, which is one of the most important factors influencing the citation of articles. Additionally, our overview of the literature in the medical field allowed us to identify and examine gaps in the History of Medicine research.

Variations in Citations Across Databases: Implications for Journal Impact Factors

The Journal Impact Factor (JIF) is a widely used metric for ranking journals based on the number of citations garnered by papers published over a specific timeframe. To assess the accuracy of JIF values, I compared citation counts for 30 of my own publications across six major bibliography databases: CrossRef, Web of Science, Publisher records, Google Scholar, PubMed and Scopus. The analysis revealed noteworthy variations in citation counts, ranging from 10% to over 50% between the lowest and highest citation counts. Google Scholar records the highest citation numbers, while PubMed reported the lowest. Notably, Web of Science, whose citation data are used in JIF calculations, tend to underestimate citation counts compared to other databases. These observations raise concerns about the accuracy of JIF calculation based on Web of Science's citation data. The real JIF values for most journals would differ from those annually reported by Clarivate's journal citation reports (JCR). These citation discrepancies underscore the importance of comprehensive data collection and the necessity to include additional citation sources. Not because a paper is cited in one journal rather than another should it have a less or more citation weight. Ultimately, one citation remains one citation, regardless of its origin. Clarivate Analytics may thus need to consider integrating all citation sources for more accurate JIF values. Alternatively, Google Scholar could potentially develop its own journal or citation impact based on its extensive journal citation records. However, while making adjustments to how the Journal Impact Factor is calculated can make it more mathematically precise, it doesn't address the fundamental biases built into the metric. Even with refinements, the Journal Impact Factor will remain skewed due to how it's defined and used.

Only half of the authors of overviews of exercise-related interventions use some strategy to manage overlapping primary studies-a metaresearch study

OBJECTIVES

The conduct of systematic reviews (SRs) and overviews share several similarities. However, because the unit of analysis for overviews is the SRs, there are some unique challenges. One of the most critical issues to manage when conducting an overview is the overlap of data across the primary studies included in the SRs. This metaresearch study aimed to describe the frequency of strategies to manage the overlap in overviews of exercise-related interventions.

STUDY DESIGN AND SETTING

A systematic search in MEDLINE (Ovid), Embase (Ovid), Cochrane Library, Epistemonikos, and other sources was conducted from inception to June 2022. We included overviews of SRs that considered primary studies and evaluated the effectiveness of exercise-related interventions for any health condition. The overviews were screened by two authors independently, and the extraction was performed by one author and checked by a second. We found 353 overviews published between 2005 and 2022 that met the inclusion criteria.

RESULTS

One hundred and sixty-four overviews (46%) used at least one strategy to visualize, quantify, or resolve overlap, with a matrix (32/164; 20%), absolute frequency (34/164; 21%), and authors' algorithms (24/164; 15%) being the most used methods, respectively. From 2016 onwards, there has been a trend toward increasing the use of some strategies to manage overlap. Of the 108 overviews that used some strategy to resolve the overlap, ie, avoiding double or multiple counting of primary study data, 79 (73%) succeeded. In overviews where no strategies to manage overlap were reported (n = 189/353; 54%), 16 overview authors (8%) recognized this as a study limitation.

CONCLUSION

Although there is a trend toward increasing its use, only half of the authors of the overviews of exercise-related interventions used a strategy to visualize, quantify, or resolve overlap in the primary studies' data. In the future, authors should report such strategies to communicate more valid results.

What makes an orthopaedic paper highly citable? A bibliometric analysis of top orthopeadic journals with 10-year follow up

PURPOSE

To examine a series of papers from top ranked orthopaedic journals with respect to the number of citations over a 10-year observation period to identify factors that lead to high citation rates.

METHODS

The Web of Science database was consulted to identify all published papers from the first-year term of 2010 (January-May) from four top orthopaedic journals: AJSM, Arthroscopy, JBJS Am and KSSTA. The database was used to analyze and compare the papers with respect to their characteristics and citations up to 2019. Basic information for each paper was collected including the author, country, study type and average citations per year (ACY). The most (Top20%) and least (Bottom20%) frequently cited papers were identified and differences were extracted.

RESULTS

Five hundred sixteen papers were included with a total of 19,261 citations. Most of the published papers were from the United States (n = 245). On average, a paper received 37.3 citations over the 10-year observation period. The most cited paper was cited 322 times. The most cited study type was randomized controlled trial (RCT) (Ø80.8). The Top20% papers were cited 37 times more often than the Bottom20%. Among the Top20%, the largest group was cohort study (n = 20) followed by case series (n = 19). Among others, the number of authors, the number of keywords and the number of references significantly correlated with the number of citations (p < 0.001).

CONCLUSIONS

Factors influencing citation frequency were identified.

Bibliometric and visualization analysis of research trend in mental health problems of children and adolescents during the COVID-19 pandemic

Objectives

To analyze the evolution of research on children and adolescents mental health issues during COVID-19 pandemic and discuss research hotspots and cutting-edge developments.

Methods

The literature obtained from the web of science core collection as of June 28, 2022, was analyzed using Citespace, VOSviewer bibliometric visualization mapping software.

Results

A total of 6,039 relevant papers were found, of which 5,594 were included in the study. The number of literatures is growing since 2020; and the country, institution, and journal publications were analyzed. The co-citation analysis shows that there are more research articles among the highly cited articles and a lack of systematic reviews that use critical thinking for review. In the cluster analysis, mental health and life change were the most representative. The timeline view of the keywords shows that Online learning (#0), Public health (#1), and Mental health (#2) are the three largest clusters and shows the change over time.

Conclusion

This study helped analyze the mental health of children and adolescents during the COVID-19 pandemic and identified hot trends and shortcomings, which are important references for the theoretical basis of future research and decision making and technical guidance for systematic reviews.

Integrity of the editing and publishing process is the basis for improving an academic journal’s Impact Factor

BACKGROUND

Journal Impact Factor™ (JIF) is often used to evaluate the relative reputation and quality of academic journals in their respective fields, and can greatly influence the quality and scope of subsequent manuscript submissions. Therefore, many if not all academic journals are interested in increasing their JIF, to improve their academic impact.

AIM

To determine the importance of the integrity of the editorial and publication process in improving the academic influence of academic journals and the JIF of academic journals.

METHODS

In this paper, we describe our statistical analysis of bibliometric factors - including the 2021 JIFs released in the Journal Citation Report™ 2022, discipline rankings, received and published articles in 2019-2021, and webpage visits and downloads - for seven journals published by Baishideng Publishing Group (Baishideng) and indexed in Science Citation Index Expanded™; ultimately, we introduce and discuss the editing and publishing processes of Baishideng’s journals in their entirety, as they form the basis for our objective of safeguarding and bolstering integrity in academic publication.

RESULTS

For the seven journals assessed, their 2021 JIFs were basically unchanged from 2020, with the current metric ranging from 5.374 for World Journal of Gastroenterology (WJG) to 1.534 for World Journal of Clinical Cases (WJCC). Further assessments of the journals’ bibliometrics from 2019 to 2020, showed that World Journal of Stem Cells has the highest self-citation rate (1.43%) and World Journal of Gastrointestinal Surgery has the lowest (0.21%). Additionally, the total 3012 articles published during this period were cited by more than 20000 articles in approximately 8000 academic journals. Of note, the 1102 articles published in WJG were cited by articles in 3059 journals, among which 171 journals have a JIF of > 10, including internationally renowned academic journals such as CA-A Cancer Journal for Clinicians (2021 JIF 286.130, record count: 1), Lancet (2021 JIF 202.731, record count: 4), Nature Reviews Immunology (2021 JIF 108.555, record count: 2), Nature Reviews Gastroenterology & Hepatology (2021 JIF 73.082, record count: 9), Lancet Gastroenterology & Hepatology (2021 JIF 45.042, record count: 8), Gastroenterology (2021 JIF 33.883, record count: 19), and Gut (2021 JIF 31.793, record count: 21). This suggests that Baishideng’s journals have been widely recognized for their academic quality. In the Reference Citation Analysis (RCA) database, all seven Baishideng-published journals obtained a 2022 Journal Article Influence Index (JAII). For example, WJG has a 2022 JAII of 22.048, ranking 18^th out of 102 journals in the field of gastroenterology & hepatology in the RCA, with 469909 total citations (6/102) and 21313 total articles (5/102). The numbers of manuscripts received and published in 2021 were both higher than those in 2019-2020. For example, WJCC received a total of 3650 manuscripts in 2021, which is 91.1% higher than those in 2019-2020 (average: 1910 papers/year). In 2021, WJCC published 1296 articles, representing an increase of 105.1% compared to those in 2019-2020 (average: 632 articles/year). The numbers of webpage visits and downloads received by the seven journals have increased year by year. For example, the number of total visits received by WJG in 2019-2021 was 1974052 in 2019, 2317835 in 2020 (increased by 17.4% compared with that in 2019), and 2652555 in 2021 (increased by 4.4% compared with that in 2020). The visitors were from more than 220 countries and regions worldwide, such as the United States, China, and the United Kingdom. Open access (OA) plays a vital role in improving the quality, efficiency, transparency, and integrity of academic journal publishing. From 2019 to 2021, a total of 5543 OA articles were published in the seven journals, of which 2083 (37.6%) were invited and published free-of-charge. During the same period, 1683 articles were published in WJG, and the authors were from more than 70 countries and regions. For the total 5543 articles published in the seven journals from 2019 to 2021, 3903 article quality tracking reports were received after the online publication of these articles. The quality of the articles was further evaluated through the Baishideng’s article quality and author evaluation tracking system, with 4655 articles (84.0%) having received author evaluation and feedback, which contributes to tracking metrics for authors’ satisfaction with the collective publication processes. From March 25, 2021 to June 28, 2022, the seven journals received a total of 424 reader evaluations and 229 letters from readers; this subsequent reader engagement demonstrates that the popularity of the published articles and the volume of their readership audience were improved through the reader evaluation system.

CONCLUSION

Ultimately, the findings from our bibliometric assessments indicate that establishing, promoting and actively practicing processes that safeguard and bolster the integrity of the editing and publication process also help to improve the academic influence of academic journals, which itself is the cornerstone for improving JIF.

The Influence of International Collaboration on the Scientific Impact in V4 Countries

Several strategies are used by researchers and research facilities to increase their scientific production and consequent research quality. Bibliometric records show that coauthorship and the number of participating organizations in research publications are steadily increasing; however, the effect of collaboration varies across disciplines, and the corresponding author’s country appears to influence research impact. This finding inspired our research question for this study: How does international cooperation affect scientific impact, and does the affiliation of corresponding authors influence citation impact indicators at the level of individual publications? To this end, we provide a comparative evaluation of research articles published in Q1 journals among Visegrad Group countries (Czech Republic, Hungary, Poland and Slovakia) in Medical and Health sciences between 2017 and 2021. The study investigates the relationship between collaboration type (national vs. international) and scientific impact (impact factor of the journal and category normalized citation impact or research papers), as well as the impact of the country of the corresponding author’s affiliation on quantitative quality of individual papers. We show that Q1 research papers in international collaboration have a higher scientific impact than papers published in national partnerships. Moreover, the corresponding authors’ country of affiliation significantly affects scientific impact.

Methodological approaches for assessing certainty of the evidence in umbrella reviews: A scoping review

INTRODUCTION

The number of umbrella reviews (URs) that compiled systematic reviews and meta-analysis (SR-MAs) has increased dramatically over recent years. No formal guidance for assessing the certainty of evidence in URs of meta-analyses exists nowadays. URs of non-interventional studies help establish evidence linking exposure to certain health outcomes in a population. This study aims to identify and describe the methodological approaches for assessing the certainty of the evidence in published URs of non-interventions.

METHODS

We searched from 3 databases including PubMed, Embase, and The Cochrane Library from May 2010 to September 2021. We included URs that included SR-MAs of studies with non-interventions. Two independent reviewers screened and extracted data. We compared URs characteristics stratified by publication year, journal ranking, journal impact factor using Chi-square test.

RESULTS

Ninety-nine URs have been included. Most were SR-MAs of observational studies evaluating association of non-modifiable risk factors with some outcomes. Only half (56.6%) of the included URs assessed the certainty of the evidence. The most frequently used criteria is credibility assessment (80.4%), followed by GRADE approach (14.3%). URs published in journals with higher journal impact factor assessed certainty of evidence than URs published in lower impact group (77.1 versus 37.2% respectively, p < 0.05). However, criteria for credibility assessment used in four of the seven URs that were published in top ranking journals were slightly varied.

CONCLUSIONS

Half of URs of MAs of non-interventional studies have assessed the certainty of the evidence, in which criteria for credibility assessment was the commonly used method. Guidance and standards are required to ensure the methodological rigor and consistency of certainty of evidence assessment for URs.