1
|
Puzzitiello RN, Lachance AD, Michalowski A, Menendez ME, Salzler MJ. Comparing Orthopaedic Randomized Control Trials Published in High-Impact Medical and Orthopaedic Journals. J Am Acad Orthop Surg 2023; 31:e974-e983. [PMID: 37722064 DOI: 10.5435/jaaos-d-22-00604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/03/2022] [Accepted: 07/25/2023] [Indexed: 09/20/2023] Open
Abstract
INTRODUCTION Orthopaedic studies published in high-impact medical journals are often believed to have a high prevalence of negative or neutral results and possess methodological characteristics that may bias toward nonsurgical treatments. The purpose of this study was to compare study characteristics, methodologic quality, exposure, and outcome direction among orthopaedic randomized control trials (RCTs) published in high-impact medical and orthopaedic journals and to identify study attributes associated with greater impact. METHODS RCTs published between January 2010 and December 2020 in the five medical journals and 10 orthopaedic journals with the highest 5-year impact factors were analyzed. Inclusion criteria were RCTs reporting on orthopaedic surgical intervention compared with nonsurgical or less-invasive surgical procedures. Study characteristics, methodologic quality (Jadad scale), outcomes, and altmetric data were collected. Primary outcomes were categorized as positive (favoring surgical/more-extensive surgery), negative (favoring nonsurgical/less-extensive surgery), or neutral. RESULTS One hundred twenty-eight RCTs were analyzed; 26 from medical and 102 from orthopaedic journals. Studies published in medical journals included more authors ( P < 0.001), larger sample sizes ( P < 0.001), more institutions ( P < 0.001), and more often received funding ( P < 0.001). The average Jadad scale did not significantly differ between journals ( P = 0.14). The direction of the primary study outcome did not differ between journals ( P = 0.22). Average AAS and annual citation rates were higher in RCTs published in medical journals ( P < 0.001). Publication in a medical journal was the only covariate associated with higher annual citation rates ( P < 0.001) and AAS ( P < 0.001) on multivariable analyses. DISCUSSION High-impact medical journals do not publish orthopaedic RCTs with negative or neutral findings at a rate that significantly differs from orthopaedic journals. However, the higher impact and digital coverage of the studies published in medical journals may disproportionally influence the practices of nonorthopaedic providers. Raising awareness of critical findings published in orthopaedic journals may be particularly important for improving healthcare policies and orthopaedic referral patterns for musculoskeletal problems.
Collapse
Affiliation(s)
- Richard N Puzzitiello
- From the Department of Orthopaedics (Puzzitiello, Lachance, Michalowski, and Salzler), Tufts Medical Center, Boston, MA and the Midwest Orthopaedics at Rush, Rush University Medical Center, Chicago, IL (Menendez)
| | | | | | | | | |
Collapse
|
2
|
Loomba RS, Villarreal EG, Patel RD, Santos-Cantu D, Alanis-Garza C, Flores S, Farias JS, Jacobs JP. Altmetric score, reads, and citations in paediatric cardiology: do they correlate and what do they mean? Cardiol Young 2022; 33:1-6. [PMID: 35718987 DOI: 10.1017/s1047951122001780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
INTRODUCTION The primary objective of this study was to determine whether Altmetric score, number of reads, and citations for paediatric cardiology manuscripts correlate with one another. A secondary objective was to determine the extent to which factors mediated citation number for paediatric cardiology manuscripts. METHODS Data for this study came from manuscripts published in Cardiology in the Young (2010-2021). Data were extracted by using data shared on the journal website. Spearman's correlation analyses were conducted between manuscript reads, citations, and Altmetric score. Regression analyses were conducted with number of citations as the dependent variable and year of publication, publication type, number of reads, and Altmetric score as independent variables. RESULTS A total of 2642 manuscripts were included in the final analyses. Reads and citations had poor correlation (r-value 0.32); reads and Altmetric score had negligible correlation (r-value 0.26); and Altmetric score and citations had negligible correlation (r-value 0.07). Year of publication was independently associated with number of citations (β -0.95, p-value <0.01). Manuscript type was independently associated with number of citations (β 1.04, p-value <0.01). Number of reads was independently associated with citations (β 0.01, p-value <0.01). Altmetric score was independently associated with number of citations (β 0.05, p-value <0.01). CONCLUSION This study describes the correlation of reads, citations, and Altmetric score in manuscripts published in Cardiology in the Young, demonstrating poor correlation, at best, between these metrics. Each bibliometric index seems to represent a different phenomenon of manuscript consumption. No single bibliometric index in isolation offers ample representation of manuscript consumption.
Collapse
Affiliation(s)
- Rohit S Loomba
- Division of Cardiology, Advocate Children's Hospital, Oak Lawn, Illinois, USA
- Department of Pediatrics, Chicago Medical School/Rosalind Franklin University of Medicine and Science, North Chicago, Illinois, USA
| | - Enrique G Villarreal
- Tecnologico de Monterrey, Escuela de Medicina y Ciencias de la Salud, Monterrey, Nuevo Leon, Mexico
| | - Riddhi D Patel
- Division of Cardiology, Advocate Children's Hospital, Oak Lawn, Illinois, USA
| | - Daniela Santos-Cantu
- Tecnologico de Monterrey, Escuela de Medicina y Ciencias de la Salud, Monterrey, Nuevo Leon, Mexico
| | - Cordelia Alanis-Garza
- Tecnologico de Monterrey, Escuela de Medicina y Ciencias de la Salud, Monterrey, Nuevo Leon, Mexico
| | - Saul Flores
- Section of Critical Care Medicine and Cardiology, Texas Children's Hospital, Houston, Texas, USA
- Department of Pediatrics, Baylor College of Medicine, Houston, Texas, USA
| | - Juan S Farias
- Tecnologico de Monterrey, Escuela de Medicina y Ciencias de la Salud, Monterrey, Nuevo Leon, Mexico
| | - Jeffrey P Jacobs
- Congenital Heart Center, Division of Cardiovascular Surgery, Departments of Surgery and Pediatrics, University of Florida, Gainesville, Florida, USA
| |
Collapse
|
3
|
Katsura M, Kuriyama A, Tada M, Tsujimoto Y, Luo Y, Yamamoto K, So R, Aga M, Matsushima K, Fukuma S, Furukawa TA. High variability in results and methodological quality among overlapping systematic reviews on the same topics in surgery: a meta-epidemiological study. Br J Surg 2021; 108:1521-1529. [PMID: 34791075 DOI: 10.1093/bjs/znab328] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2021] [Accepted: 08/27/2021] [Indexed: 01/04/2023]
Abstract
BACKGROUND Redundant publication of systematic reviews and meta-analyses (SRs/MAs) on the same topic presents an increasing burden for clinicians. The aim of this study was to describe variabilities in effect size and methodological quality of overlapping surgery-related SRs/MAs and to investigate factors associated with their postpublication citations. METHODS PubMed/MEDLINE was searched to identify SRs/MAs of RCTs on thoracoabdominal surgeries published in 2015. Previous SRs/MAs on the same topics published within the preceding 5 years (2011-2015) were identified and 5-year citation counts (through to 2020) were evaluated. Discrepancies in pooled effect sizes and their methodological quality using A Measurement Tool to Assess Systematic Reviews (AMSTAR) among overlapping SRs/MAs were assessed. The SR/MA-level factors associated with 5-year citation counts were explored, using a mixed-effects regression model with a random intercept for surgical topics. RESULTS A total of 57 surgery-related SRs/MAs (48 topics) published in 2015 were identified, and 146 SRs/MAs had overlapping publications on 29 topics (60.4 per cent of all topics) in the preceding 5 years. There was considerable variability in methodological quality of SRs/MAs and coverage probability for relevant RCTs, resulting in discrepant effect size estimates for the same topic. High quality (AMSTAR score 8-11) was independently associated with higher 5-year citation counts (coefficient = 32.82; 95 per cent c.i. 15.63 to 50.02; P < 0.001). CONCLUSION Overlapping SRs/MAs with high variability in results and methodological quality were common in surgery. A high-quality SR/MA score was an independent predictor of more frequent citations. Researchers and journal editors should concentrate their efforts on limiting publications to higher-quality reviews.
Collapse
Affiliation(s)
- Morihiro Katsura
- Human Health Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan.,Department of Surgery, Okinawa Chubu Hospital, Okinawa, Japan
| | - Akira Kuriyama
- Emergency and Critical Care Centre, Kurashiki Central Hospital, Okayama, Japan
| | - Masafumi Tada
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.,Department of Neurology, Emergency Medicine, Nagoya City University East Medical Centre, Nagoya, Japan
| | - Yasushi Tsujimoto
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.,Department of Nephrology and Dialysis, Kyoritsu Hospital, Hyogo, Japan
| | - Yan Luo
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| | - Kazumichi Yamamoto
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.,Institute for Airway Disease, Hyogo, Japan
| | - Ryuhei So
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan.,Okayama Psychiatric Medical Centre, Okayama, Japan
| | - Masaharu Aga
- Department of Respiratory Medicine, Yokohama Municipal Citizen's Hospital, Kanagawa, Japan
| | - Kazuhide Matsushima
- Division of Acute Care Surgery, University of Southern California, Los Angeles, California, USA
| | - Shingo Fukuma
- Human Health Sciences, Kyoto University Graduate School of Medicine, Kyoto, Japan
| | - Toshi A Furukawa
- Department of Health Promotion and Human Behavior, Kyoto University Graduate School of Medicine/School of Public Health, Kyoto, Japan
| |
Collapse
|
4
|
Schniedermann A. A comparison of systematic reviews and guideline-based systematic reviews in medical studies. Scientometrics 2021. [DOI: 10.1007/s11192-021-04199-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Abstract
AbstractThe question of how citation impact relates to academic quality accompanies every decade in bibliometric research. Although experts have employed more complex conceptions of research quality for responsible evaluation, detailed analyses of how impact relates to dimensions such as methodological rigor are lacking. But the increasing number of formal guidelines for biomedical research offer not only the potential to understand the social dynamics of standardization, but also their relations to scientific rewards. By using data from Web of Science and PubMed, this study focuses on systematic reviews from biomedicine and compares this genre with those systematic reviews that applied the PRISMA reporting standard. Besides providing an overview about growth and location, it was found that the latter, more standardized type of systematic review accumulates more citations. It is argued that instead of reinforcing the traditional conception that higher impact represents higher quality, highly prolific authors could be more inclined to develop and apply new standards than more average researchers. In addition, research evaluation would benefit from a more nuanced conception of scientific output which respects the intellectual role of various document types.
Collapse
|
5
|
Ho L, Ke FYT, Wong CHL, Wu IXY, Cheung AKL, Mao C, Chung VCH. Low methodological quality of systematic reviews on acupuncture: a cross-sectional study. BMC Med Res Methodol 2021; 21:237. [PMID: 34717563 PMCID: PMC8557536 DOI: 10.1186/s12874-021-01437-0] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2021] [Accepted: 10/14/2021] [Indexed: 01/08/2023] Open
Abstract
BACKGROUND While well-conducted systematic reviews (SRs) can provide the best evidence on the potential effectiveness of acupuncture, limitations on the methodological rigour of SRs may impact the trustworthiness of their conclusions. This cross-sectional study aimed to evaluate the methodological quality of a representative sample of SRs on acupuncture effectiveness. METHODS Cochrane Database of Systematic Reviews, MEDLINE, and EMBASE were searched for SRs focusing on the treatment effect of manual acupuncture or electro-acupuncture published during January 2018 and March 2020. Eligible SRs must contain at least one meta-analysis and be published in English language. Two independent reviewers extracted the bibliographical characteristics of the included SRs with a pre-designed questionnaire and appraised the methodological quality of the studies with the validated AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews 2). The associations between bibliographical characteristics and methodological quality ratings were explored using Kruskal-Wallis rank tests and Spearman's rank correlation coefficients. RESULTS A total of 106 SRs were appraised. Only one (0.9%) SR was of high overall methodological quality, zero (0%) was of moderate-quality, six (5.7%) and 99 (93.4%) were of low-quality and critically low-quality respectively. Among appraised SRs, only ten (9.4%) provided an a priori protocol, four (3.8%) conducted a comprehensive literature search, five (4.7%) provided a list of excluded studies, and six (5.7%) performed meta-analysis appropriately. Cochrane SRs, updated SRs, and SRs that did not search non-English databases had relatively higher overall quality. CONCLUSIONS Methodological quality of SRs on acupuncture is unsatisfactory. Future reviewers should improve critical methodological aspects of publishing protocols, performing comprehensive search, providing a list of excluded studies with justifications for exclusion, and conducting appropriate meta-analyses. These recommendations can be implemented via enhancing the technical competency of reviewers in SR methodology through established education approaches as well as quality gatekeeping by journal editors and reviewers. Finally, for evidence users, skills in SR critical appraisal remain to be essential as relevant evidence may not be available in pre-appraised formats.
Collapse
Affiliation(s)
- Leonard Ho
- School of Chinese Medicine, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Fiona Y T Ke
- The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Charlene H L Wong
- The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Irene X Y Wu
- Xiangya School of Public Health, Central South University, 5/F, 238 Shang-Ma-Yuan-Ling Alley, Kai-Fu District, Changsha, Hunan, China.
| | - Andy K L Cheung
- The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Chen Mao
- Department of Epidemiology, School of Public Health, Southern Medical University, Guangzhou, China
| | - Vincent C H Chung
- School of Chinese Medicine, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong.,The Jockey Club School of Public Health and Primary Care, Faculty of Medicine, The Chinese University of Hong Kong, Shatin, Hong Kong
| |
Collapse
|
6
|
Altmetrics Attention Scores for Randomized Controlled Trials in Total Joint Arthroplasty Are Reflective of High Scientific Quality: An Altmetrics-Based Methodological Quality and Bias Analysis. JOURNAL OF THE AMERICAN ACADEMY OF ORTHOPAEDIC SURGEONS GLOBAL RESEARCH AND REVIEWS 2020; 4:e20.00187. [PMID: 33278182 PMCID: PMC7714052 DOI: 10.5435/jaaosglobal-d-20-00187] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 12/03/2022]
Abstract
The Altmetric Attention Score (AAS) has been associated with citation rates across medical and surgical disciplines. However, factors that drive high AAS remain poorly understood and there remains multiple pitfalls to correlating these metrics alone with the quality of a study. The purpose of the current study was to determine the relationship between methodologic and study biases and the AAS in randomized controlled trials (RCTs) published in total joint arthroplasty journals.
Collapse
|
7
|
Kazi AA, Vahidi NA, Sinkovich J, Coelho DH. Does quality of research in otolaryngology correlate with academic impact? Laryngoscope Investig Otolaryngol 2020; 5:819-823. [PMID: 33134528 PMCID: PMC7585250 DOI: 10.1002/lio2.444] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2020] [Accepted: 07/25/2020] [Indexed: 11/05/2022] Open
Abstract
OBJECTIVE To determine if the quality of otolaryngology-related journal articles correlates with traditional measures of article impact. METHODS All articles published by Laryngoscope in 2011 were categorized according to level of evidence (LOE) according to the Oxford Center for Evidence Based Medicine rubric. Articles without a level of evidence assigned were alphabetically subcategorized type with letters A-D corresponding to Contemporary Reports, Case Reports, Basic Science or Animal Studies, and Other respectively. Citations per article were then recorded per article each year from 2012 to 2018. RESULTS A total of 494 articles were included for analysis, 315 had numerical LOE and 179 had alphabetical LOE. There was a strong negative correlation between numerical LOE and median and interquartile number of citations (R = -.9014, P = .037). Overall, numerical LOE had a significantly higher median number of citations per article compared with the non-number/alphabetical group (14 vs 6, P < .001). CONCLUSIONS Higher quality research as determined by level of evidence is in fact being cited more frequently than lower quality articles. Although the scope of this study was relatively limited, these data suggest that better designed studies may exhibit greater impact by traditional measures. Such findings should serve as an impetus for (and validation of) continued pursuit of high LOE research. LEVEL OF EVIDENCE NA.
Collapse
Affiliation(s)
- Aasif A. Kazi
- Department of Otolaryngology—Head and Neck SurgeryVirginia Commonwealth University School of MedicineRichmondVirginiaUSA
| | - Nima A. Vahidi
- Department of Otolaryngology—Head and Neck SurgeryVirginia Commonwealth University School of MedicineRichmondVirginiaUSA
| | - John Sinkovich
- Department of Otolaryngology—Head and Neck SurgeryVirginia Commonwealth University School of MedicineRichmondVirginiaUSA
| | - Daniel H. Coelho
- Department of Otolaryngology—Head and Neck SurgeryVirginia Commonwealth University School of MedicineRichmondVirginiaUSA
| |
Collapse
|
8
|
Quinn TJ, McMahon D. Commentary: Evaluating the Landscape of Clinical Research in Neurosurgery. Neurosurgery 2019; 85:E494-E495. [PMID: 31120109 DOI: 10.1093/neuros/nyz155] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/22/2019] [Accepted: 01/23/2019] [Indexed: 11/14/2022] Open
Affiliation(s)
- Terence J Quinn
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, United Kingdom
| | - David McMahon
- Institute of Cardiovascular and Medical Sciences, University of Glasgow, Glasgow, United Kingdom
| |
Collapse
|
9
|
New quality and quantity indices in science (NewQIS): results of the first decade-project progress review. Scientometrics 2019; 121:451-478. [PMID: 32214551 PMCID: PMC7089293 DOI: 10.1007/s11192-019-03188-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/03/2019] [Indexed: 12/19/2022]
Abstract
Strategies employing information science and scientometric approaches were introduced to science policy and management over the past decades. As a rapidly evolving field, new bibliometric parameters are proposed and discussed continuously and the fields also benefits from the introduction of novel visualization techniques. The present article summarizes the experiences with a platform that combines geographical mapping with scientometrics. It was established between 2005 and 2008 at the Charité in Berlin and termed “New Quality and Quantity Indices in Science” (NewQIS), consisting of the integration of common scientometric parameters such as the h-index and novel visualization techniques including density equalizing mapping. NewQIS was used to assess socio-economic important fields of medicine and sciences. Within NewQIS studies, research activities, citation patterns and their relation to socio-economic figures were analyzed with regard to time periods, countries, continents or even single cities. Within the decade after its establishment, more than 80 NewQIS articles were peer-reviewed and published. Being a non-funded low budget project, it was used by many medical students to conduct their MD thesis. The narrow technical frame led to the chance of a comparison of research output between different fields of science. This article summarizes NewQIS 1.0 activities, discusses its limits and gives a look into the future of NewQIS 2.0 with a target of 200 evaluated entities of the biomedical field of sciences.
Collapse
|
10
|
On the potential distortions of highly cited papers in emerging research fields: A critical appraisal. Behav Brain Sci 2019. [DOI: 10.1017/s0140525x18002807] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Abstract
Citation-based metrics are increasingly used as a proxy to define representative, considerable, or significant papers. We challenge this belief by taking into account factors that may play a role in providing citations to a manuscript and whether/how those highly cited studies could shape a scientific field. A different approach to summarisation of relevant core publications within a topic is proposed.
Collapse
|