Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences

Evaluation and Comparison of Academic Impact and Disruptive Innovation Level of Medical Journals: Bibliometric Analysis and Disruptive Evaluation

BACKGROUND

As an important platform for researchers to present their academic findings, medical journals have a close relationship between their evaluation orientation and the value orientation of their published research results. However, the differences between the academic impact and level of disruptive innovation of medical journals have not been examined by any study yet.

OBJECTIVE

This study aims to compare the relationships and differences between the academic impact, disruptive innovation levels, and peer review results of medical journals and published research papers. We also analyzed the similarities and differences in the impact evaluations, disruptive innovations, and peer reviews for different types of medical research papers and the underlying reasons.

METHODS

The general and internal medicine Science Citation Index Expanded (SCIE) journals in 2018 were chosen as the study object to explore the differences in the academic impact and level of disruptive innovation of medical journals based on the OpenCitations Index of PubMed open PMID-to-PMID citations (POCI) and H1Connect databases, respectively, and we compared them with the results of peer review.

RESULTS

First, the correlation coefficients of the Journal Disruption Index (JDI) with the Journal Cumulative Citation for 5 years (JCC5), Journal Impact Factor (JIF), and Journal Citation Indicator (JCI) were 0.677, 0.585, and 0.621, respectively. The correlation coefficient of the absolute disruption index (Dz) with the Cumulative Citation for 5 years (CC5) was 0.635. However, the average difference in the disruptive innovation and academic influence rankings of journals reached 20 places (about 17.5%). The average difference in the disruptive innovation and influence rankings of research papers reached about 2700 places (about 17.7%). The differences reflect the essential difference between the two evaluation systems. Second, the top 7 journals selected based on JDI, JCC5, JIF, and JCI were the same, and all of them were H-journals. Although 8 (8/15, 53%), 96 (96/150, 64%), and 880 (880/1500, 58.67%) of the top 0.1%, top 1%, and top 10% papers selected based on Dz and CC5, respectively, were the same. Third, research papers with the "changes clinical practice" tag showed only moderate innovation (4.96) and impact (241.67) levels but had high levels of peer-reviewed recognition (6.00) and attention (2.83).

CONCLUSIONS

The results of the study show that research evaluation based on innovative indicators is detached from the traditional impact evaluation system. The 3 evaluation systems (impact evaluation, disruptive innovation evaluation, and peer review) only have high consistency for authoritative journals and top papers. Neither a single impact indicator nor an innovative indicator can directly reflect the impact of medical research for clinical practice. How to establish an integrated, comprehensive, scientific, and reasonable journal evaluation system to improve the existing evaluation system of medical journals still needs further research.

A memory-theoretic account of citation propagation

Despite the common assumption that citations are indicative of an article's scientific merit, increasing evidence indicates that citation counts are largely driven by variables unrelated to quality. In this article, we treat people's decisions of what to cite as an instance of memory retrieval and show that observed citation patterns are well accounted for by a model of memory. The proposed exposure model anticipates that small alterations in factors that affect people's ability to retrieve to-be-cited articles from memory early in their life cycle are magnified over time and can lead to the emergence of highly cited papers. This effect occurs even when there is no variation in the starting point exposure probabilities (i.e. when assuming a level playing field where all articles are treated equally and of equal 'quality'), and is exacerbated by natural variation in retrievability of articles due to encoding. We discuss the implications of the model within the context of research evaluation and hiring, tenure and promotion decisions.

Towards responsible research assessment: How to reward research quality

Researchers would be more willing to prioritize research quality over quantity if the incentive structure of the academic system aligned with this goal. The winner of a 2023 Einstein Foundation Award for Promoting Quality in Research explains how they rose to this challenge.

The landscape of open science in behavioral addiction research: Current practices and future directions

Open science refers to a set of practices that aim to make scientific research more transparent, accessible, and reproducible, including pre-registration of study protocols, sharing of data and materials, the use of transparent research methods, and open access publishing. In this commentary, we describe and evaluate the current state of open science practices in behavioral addiction research. We highlight the specific value of open science practices for the field; discuss recent field-specific meta-scientific reviews that show the adoption of such practices remains in its infancy; address the challenges to engaging with open science; and make recommendations for how researchers, journals, and scientific institutions can work to overcome these challenges and promote high-quality, transparently reported behavioral addiction research. By collaboratively promoting open science practices, the field can create a more sustainable and productive research environment that benefits both the scientific community and society as a whole.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: The Granada statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: The Granada Statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

Improving the quality of publications in and advancing the paradigms of clinical and social Pharmacy Practice Research: The Granada statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work. © 2023 The Author(s) Published by Elsevier Inc, Springer Nature, Brazilian Society of Hospital Pharmacy and Health Services, Elsevier Inc, Royal Pharmaceutical Society, Biomedcentral, Sociedad Española de Farmacia Hospitalaria (S.E.F.H), Pharmaceutical Care España Foundation, European Association of Hospital Pharmacists, Faculty of Pharmacy.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: the Granada Statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: the Granada Statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as 'the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on healthcare systems, medicine use, and patient care'. Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other healthcare areas (ie, medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: The Granada statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

A discipline-wide investigation of the replicability of Psychology papers over the past two decades

Conjecture about the weak replicability in social sciences has made scholars eager to quantify the scale and scope of replication failure for a discipline. Yet small-scale manual replication methods alone are ill-suited to deal with this big data problem. Here, we conduct a discipline-wide replication census in science. Our sample (N = 14,126 papers) covers nearly all papers published in the six top-tier Psychology journals over the past 20 y. Using a validated machine learning model that estimates a paper's likelihood of replication, we found evidence that both supports and refutes speculations drawn from a relatively small sample of manual replications. First, we find that a single overall replication rate of Psychology poorly captures the varying degree of replicability among subfields. Second, we find that replication rates are strongly correlated with research methods in all subfields. Experiments replicate at a significantly lower rate than do non-experimental studies. Third, we find that authors' cumulative publication number and citation impact are positively related to the likelihood of replication, while other proxies of research quality and rigor, such as an author's university prestige and a paper's citations, are unrelated to replicability. Finally, contrary to the ideal that media attention should cover replicable research, we find that media attention is positively related to the likelihood of replication failure. Our assessments of the scale and scope of replicability are important next steps toward broadly resolving issues of replicability.

Improving the quality of publications in and advancing the paradigms of clinical and social pharmacy practice research: The Granada Statements

Pharmacy and pharmaceutical sciences embrace a series of different disciplines. Pharmacy practice has been defined as "the scientific discipline that studies the different aspects of the practice of pharmacy and its impact on health care systems, medicine use, and patient care". Thus, pharmacy practice studies embrace both clinical pharmacy and social pharmacy elements. Like any other scientific discipline, clinical and social pharmacy practice disseminates research findings using scientific journals. Clinical pharmacy and social pharmacy journal editors have a role in promoting the discipline by enhancing the quality of the articles published. As has occurred in other health care areas (i.e., medicine and nursing), a group of clinical and social pharmacy practice journal editors gathered in Granada, Spain to discuss how journals could contribute to strengthening pharmacy practice as a discipline. The result of that meeting was compiled in these Granada Statements, which comprise 18 recommendations gathered into six topics: the appropriate use of terminology, impactful abstracts, the required peer reviews, journal scattering, more effective and wiser use of journal and article performance metrics, and authors' selection of the most appropriate pharmacy practice journal to submit their work.

The credibility crisis and democratic governance: how to reform university governance to be compatible with the nature of science

To address the credibility crisis facing many disciplines, change is needed at the institutional level. Science will only function optimally if the culture by which it is governed becomes aligned with the way of thinking required in science itself. The paper suggests a series of graduated reforms to university governance, to radically reform the operation of universities. The reforms are based on existing established open democratic practices. The aim is for universities to become consistent with the flourishing of science and research more generally.

Forecasting the publication and citation outcomes of COVID-19 preprints

Many publications on COVID-19 were released on preprint servers such as medRxiv and bioRxiv. It is unknown how reliable these preprints are, and which ones will eventually be published in scientific journals. In this study, we use crowdsourced human forecasts to predict publication outcomes and future citation counts for a sample of 400 preprints with high Altmetric score. Most of these preprints were published within 1 year of upload on a preprint server (70%), with a considerable fraction (45%) appearing in a high-impact journal with a journal impact factor of at least 10. On average, the preprints received 162 citations within the first year. We found that forecasters can predict if preprints will be published after 1 year and if the publishing journal has high impact. Forecasts are also informative with respect to Google Scholar citations within 1 year of upload on a preprint server. For both types of assessment, we found statistically significant positive correlations between forecasts and observed outcomes. While the forecasts can help to provide a preliminary assessment of preprints at a faster pace than traditional peer-review, it remains to be investigated if such an assessment is suited to identify methodological problems in preprints.