Changing the Status Quo: The 100 Most-Disruptive Papers in Urology?

Evaluation and Comparison of Academic Impact and Disruptive Innovation Level of Medical Journals: Bibliometric Analysis and Disruptive Evaluation

BACKGROUND

As an important platform for researchers to present their academic findings, medical journals have a close relationship between their evaluation orientation and the value orientation of their published research results. However, the differences between the academic impact and level of disruptive innovation of medical journals have not been examined by any study yet.

OBJECTIVE

This study aims to compare the relationships and differences between the academic impact, disruptive innovation levels, and peer review results of medical journals and published research papers. We also analyzed the similarities and differences in the impact evaluations, disruptive innovations, and peer reviews for different types of medical research papers and the underlying reasons.

METHODS

The general and internal medicine Science Citation Index Expanded (SCIE) journals in 2018 were chosen as the study object to explore the differences in the academic impact and level of disruptive innovation of medical journals based on the OpenCitations Index of PubMed open PMID-to-PMID citations (POCI) and H1Connect databases, respectively, and we compared them with the results of peer review.

RESULTS

First, the correlation coefficients of the Journal Disruption Index (JDI) with the Journal Cumulative Citation for 5 years (JCC5), Journal Impact Factor (JIF), and Journal Citation Indicator (JCI) were 0.677, 0.585, and 0.621, respectively. The correlation coefficient of the absolute disruption index (Dz) with the Cumulative Citation for 5 years (CC5) was 0.635. However, the average difference in the disruptive innovation and academic influence rankings of journals reached 20 places (about 17.5%). The average difference in the disruptive innovation and influence rankings of research papers reached about 2700 places (about 17.7%). The differences reflect the essential difference between the two evaluation systems. Second, the top 7 journals selected based on JDI, JCC5, JIF, and JCI were the same, and all of them were H-journals. Although 8 (8/15, 53%), 96 (96/150, 64%), and 880 (880/1500, 58.67%) of the top 0.1%, top 1%, and top 10% papers selected based on Dz and CC5, respectively, were the same. Third, research papers with the "changes clinical practice" tag showed only moderate innovation (4.96) and impact (241.67) levels but had high levels of peer-reviewed recognition (6.00) and attention (2.83).

CONCLUSIONS

The results of the study show that research evaluation based on innovative indicators is detached from the traditional impact evaluation system. The 3 evaluation systems (impact evaluation, disruptive innovation evaluation, and peer review) only have high consistency for authoritative journals and top papers. Neither a single impact indicator nor an innovative indicator can directly reflect the impact of medical research for clinical practice. How to establish an integrated, comprehensive, scientific, and reasonable journal evaluation system to improve the existing evaluation system of medical journals still needs further research.

Disruption of trauma research: an analysis of the top cited versus disruptive trauma research publications

Introduction

The analysis of surgical research using bibliometric measures has become increasingly prevalent. Absolute citation counts (CC) or indices are commonly used markers of research quality but may not adequately capture the most impactful research. A novel scoring system, the disruptive score (DS) has been found to identity academic work that either changes paradigms (disruptive (DIS) work) or entrenches ideas (developmental (DEV) work). We sought to analyze the most DIS and DEV versus most cited research in civilian trauma.

Methods

The top papers by DS and by CC from trauma and surgery journals were identified via a professional literature search. The identified publications were then linked to the National Institutes of Health iCite tool to quantify total CC and related metrics. The top 100 DIS and DEV publications by DS were analyzed based on the area of focus, citation, and perceived clinical impact, and compared with the top 100 papers by CC.

Results

32 293 articles published between 1954 and 2014 were identified. The most common publication location of selected articles was published in Journal of Trauma (31%). Retrospective reviews (73%) were common in DIS (73%) and top CC (67%) papers, while DEV papers were frequently case reports (49%). Only 1 publication was identified in the top 100 DIS and top 100 CC lists. There was no significant correlation between CC and DS among the top 100 DIS papers (r=0.02; p=0.85), and only a weak correlation between CC and DS score (r=0.21; p<0.05) among the top 100 DEV papers.

Conclusion

The disruption score identifies a unique subset of trauma academia. The most DIS trauma literature is highly distinct and has little overlap with top trauma publications identified by standard CC metrics, with no significant correlation between the CC and DS.

Level of evidence

Level IV.

A construction and empirical research of the journal disruption index based on open citation data

For many years, the journal evaluation system has been centered on impact indicators, resulting in evaluation results that do not reflect the academic innovation of journals. To solve this issue, this study attempts to construct the Journal Disruption Index (JDI) from the perspective of measuring the disruption of each journal article. In the actual study, we measured the disruption of articles of 22 selected virology journals based on the OpenCitations Index of Crossref open DOI-to-DOI citations (COCI) first. Then we calculated the JDI of 22 virology journals based on the absolute disruption index (D Z ) of the articles. Finally, we conducted an empirical study on the differences and correlations between the impact indicators and disruption indicators as well as the evaluation effect of the disruption index. The results of the study show: (1) There are large differences in the ranking of journals based on disruption indicators and impact indicators. Among the 22 journals, 12 are ranked higher by JDI than Cumulative Impact Factor for 5 years (CIF5), the Journal Index for PR6 (JIPR6) and average Percentile in Subject Area (aPSA). The ranking difference of 17 journals between the two kinds of indicators is greater than or equal to 5. (2) There is a medium correlation between disruption indicators and impact indicators at the level of journals and papers. JDI is moderately correlated with CIF5, JIPR6 and aPSA, with correlation coefficients of 0.486, 0.471 and - 0.448, respectively. D Z was also moderately correlated with Cumulative Citation (CC), Percentile Ranking with 6 Classifications (PR6) and Percentile in Subject Area (PSA) with correlation coefficients of 0.593, 0.575 and - 0.593, respectively. (3) Compared with traditional impact indicators, the results of journal disruption evaluation are more consistent with the evaluation results of experts' peer review. JDI reflects the innovation level of journals to a certain extent, which is helpful to promote the evaluation of innovation in sci-tech journals.

Using the Disruptive Score to Identify Publications That Changed Plastic Surgery Practice

PURPOSE

The impact of academic publications is often characterized by the total number of future citations. However, this metric does not adequately characterize the true impact in terms of changing practices or paradigms. A new metric called the "disruption score" (DS) has been developed and validated in nonsurgical publications. This study aims to use the DS to identify the most disruptive publications in plastic surgery.The DS, a ratio of 2 numbers, varies between -1 and +1. Scores closer to -1 are developing papers that summarize the known literature while papers closer to +1 are disruptive-they result in a paradigm shift in the field of study.

METHODS

A search was performed for all articles from 1954 to 2014 in the following journals: Plastic and Reconstructive Surgery; Aesthetic Surgery Journal; Journal of Plastic, Reconstructive, and Aesthetic Surgery; Annals of Plastic Surgery; Aesthetic Plastic Surgery; Clinics in Plastic Surgery; and Plastic Surgery. The disruptive score was calculated for each article.The top 100 papers ranked by DS were examined and any editorials/viewpoints, publications with less than 26 citations, or less than 3 references were excluded because of their subjective nature and smaller academic contribution. The remaining 64 publications were analyzed for topic, study type, and citation count.

RESULTS

A total of 32,622 articles were found with a DS range from 0.385 to 0.923. The mean score of the top 64 articles was 0.539 with an average citation count of 195 and 9 references. Plastic and Reconstructive Surgery had the most disruptive papers with 50. There were no randomized controlled trials with a majority of the studies being technical descriptions or case series.

CONCLUSIONS

There are many ways to measure academic success, but there are fewer ways to measure the impact of academic contributions. The DS is a novel measurement that can demonstrate when an article results in a paradigm shift as opposed to just total citation count. When applied to the plastic surgery literature, the DS demonstrates that technical innovation and creativity are the most academically impactful. Future evaluations of academic success should include the DS to measure the quality of academic contributions.

The Most Disruptive Publications in Craniofacial Surgery (1954-2014)

BACKGROUND

Emphasis on the quantification and qualification of scientific literature has increased over recent years. The newly validated disruption score is a bibliometric measure that identifies groundbreaking research that eclipses prior research in a specific field of study.

MATERIALS AND METHODS

The 100 most disruptive craniofacial surgery publications were identified through query of the 4 top craniofacial journals and 10 of the most prominent Plastic and Reconstructive Surgery journals, looking at craniofacial specific publications.

RESULTS

Presented is the compilation and analysis of the 100 most disruptive publications in the field of craniofacial surgery compared to the 100 most cited publications between 1954 and 2014. Plastic and Reconstructive Surgery had the most papers in the top 100 (n = 56) followed by Journal of Oral and Maxillofacial Surgery (n = 22), and British Journal of Surgery (n = 12). The correlation coefficient between disruption scores and citation counts was -0.001 and -0.07 among all papers, and the top 100 most disruptive papers, respectively. For craniofacial journals, Journal of Craniofacial Surgery had the highest average disruption score for all published papers. The most common decade represented in the top 100 was the 1980's (n = 32) and the least common was the 2000's (n = 14). Randomized controlled trials did not comprise a large amount of either the most disruptive (n = 1) or most cited (n = 2) lists.

CONCLUSIONS

This is the first use of disruption index score to describe craniofacial surgery research. The disruption score can help recognize paradigm shifts and innovative research in this unique surgical subspecialty.

The 100 Most-disruptive Articles in Plastic and Reconstructive Surgery and Sub-specialties (1954-2014)

Alternative bibliometrics have recently been the subject of significantly increased interest. The disruption index is a new bibliometric that was recently applied to surgery and urology and identifies papers that shift paradigms and eclipse previous research in a given field.

METHODS

The 100 most-disruptive publications in the 14 most prominent plastic and reconstructive surgery and subspecialty journals were identified.

RESULTS

We present the 100 most-disruptive studies as well as the 100 most-cited studies for comparison in n=14 of the most popular plastic and reconstructive surgery (and subspecialty) journals between 1954 and 2014. The 100 most-disruptive publications in these journals were more disruptive than 99.8% of all PubMed papers. Plastic and Reconstructive Surgery (PRS) had the most papers in the top 100 (n=64) followed by British Journal of Plastic Surgery (currently Journal of Plastic, Reconstructive & Aesthetic Surgery, n=15), and Journal of Oral and Maxillofacial Surgery (n=7). PRS had 9 of the top 10 papers. However, Clinics in Plastic Surgery had the highest average disruption score for all its published papers (0.0029). The correlation coefficient linking disruption scores and citation counts was 0.01 and 0.11, respectively. The most common decade represented in the top 100 was the 1980's (n=31) and the least common was the 2000's (n=9).

CONCLUSIONS

This is the first application of the disruption index to plastic and reconstructive surgery. The disruption score provides a unique ability to identify research that has shifted paradigms and driven the innovation that defines our specialty.