A scoping review of competencies for scientific editors of biomedical journals

A large language model's assessment of methodology reporting in head and neck surgery

OBJECTIVE

The aim of this study was to assess the ability of a Large Language Model - ChatGPT 3.5 to appraise the quality of scientific methodology reporting in head and neck specific scientific literature.

METHODS

Authors asked ChatGPT 3.5 to create a grading system for scientific reporting of research methods. The language model produced a system with a max of 60 points. Individual scores were provided for Study Design and Description, Data Collection and Measurement, Statistical Analysis, Ethical Considerations, and Overall Clarity and Transparency. Twenty articles were selected at random from The American Head and Neck Society's (AHNS) fellowship curriculum 2.0 for interrogation and each 'Methods' section was input into ChatGPT 3.5 for scoring. Analysis of variance (ANOVA) was performed between different scoring categories and a post-hoc tukey HSD test was performed.

RESULTS

Twenty articles were assessed, eight were categorized as very good and nine as good based on cumulative score. Lowest mean score was noted with category of statistical analysis (Mean = 0.49, SD = 0.02). On ANOVA a significant difference between means of the different scoring categories was noted, F(4, 95) = 13.4, p ≤ 0.05. On post-hoc Tukey HSD test, mean scores for categories of data collection (Mean = 0.58, SD = 0.06) and statistical analysis (Mean = 0.49, SD = 0.02) were significantly lower when compared to other categories.

CONCLUSION

This article showcases the feasibility of employing a large language model such as ChatGPT 3.5 to assess the methods sections in head and neck academic writing.

LEVEL OF EVIDENCE: 4

Advancing Self-Evaluative and Self-Regulatory Mechanisms of Scholarly Journals: Editors’ Perspectives on What Needs to Be Improved in the Editorial Process

Meticulous self-evaluative practices in the offices of academic periodicals can be helpful in reducing widespread uncertainty about the quality of scholarly journals. This paper summarizes the results of the second part of a qualitative worldwide study among 258 senior editors of scholarly journals across disciplines. By means of a qualitative questionnaire, the survey investigated respondents’ perceptions of needed changes in their own editorial workflow that could, according to their beliefs, positively affect the quality of their journals. The results show that the most relevant past improvements indicated by respondents were achieved by: (a) raising the required quality criteria for manuscripts, by defining standards for desk rejection and/or shaping the desired qualities of the published material, and (b) guaranteeing a rigorous peer review process. Respondents believed that, currently, three areas have the most pressing need for amendment: ensuring higher overall quality of published articles (26% of respondents qualified this need as very high or high), increasing the overall quality of peer-review reports (23%), and raising reviewers’ awareness of the required quality standards (20%). Bivariate analysis shows that respondents who work with non-commercial publishers reported an overall greater need to improve implemented quality assessment processes. Work overload, inadequate reward systems, and a lack of time for development activities were cited by respondents as the greatest obstacles to implementing necessary amendments.

Improving the quality of toxicology and environmental health systematic reviews: What journal editors can do

Systematic reviews are fast increasing in prevalence in the toxicology and environmental health literature. However, how well these complex research projects are being conducted and reported is unclear. Since editors have an essential role in ensuring the scientific quality of manuscripts being published in their journals, a workshop was convened where editors, systematic review practitioners, and research quality control experts could discuss what editors can do to ensure the systematic reviews they publish are of sufficient scientific quality. Interventions were explored along four themes: setting standards; reviewing protocols; optimizing editorial workflows; and measuring the effectiveness of editorial interventions. In total, 58 editorial interventions were proposed. Of these, 26 were shortlisted for being potentially effective, and 5 were prioritized as short-term actions that editors could relatively easily take to improve the quality of published systematic reviews. Recent progress in improving systematic reviews is summarized, and outstanding challenges to further progress are highlighted.

Do not make clinical decisions based on abstracts of healthcare research: A systematic review

OBJECTIVE

To summarize the reporting quality of healthcare abstracts and inconsistencies between abstracts and full texts.

STUDY DESIGN AND SETTING

This systematic review included overviews of randomized controlled trials (RCTs) and systematic reviews (SRs) that summarized data of healthcare abstracts on reporting of abstracts and consistency of abstracts with the full text. Searches were performed in PubMed, CENTRAL, Cochrane Library and EMBASE databases from 1900 to February 2019. Two authors screened the overviews and extracted the data. All analyses were descriptive and divided into two main groups: abstracts' reporting quality and abstracts' consistency with the full text. Abstracts were considered poorly reported and inconsistent with the full text if more than 5% of abstracts' information was not fully reported or not consistent with the full text.

RESULTS

27 overviews analyzing 5,194 RCTs and 866 SRs were retrieved for reporting quality of abstracts. A total of 22 overviews analyzing 2,025 RCTs and 551 SRs were included for consistency of abstracts with the full text. Abstracts across all healthcare areas presented poor reporting quality and were inconsistent with the full texts, with results and conclusions as the most inconsistent sections.

CONCLUSION

Abstracts of healthcare RCTs and SRs have shown a large room for improvement in reporting quality and consistency with the full text. Authors, journal editors and reviewers need to give the highest priority to this matter.

A scoping review on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals

BACKGROUND

Although peer reviewers play a key role in the manuscript review process, their roles and tasks are poorly defined. Clarity around this issue is important as it may influence the quality of peer reviewer reports. This scoping review explored the roles and tasks of peer reviewers of biomedical journals.

METHODS

Comprehensive literature searches were conducted in Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, EMBASE, MEDLINE, PsycINFO, Scopus and Web of Science from inception up to May 2017. There were no date and language restrictions. We also searched for grey literature. Studies with statements mentioning roles, tasks and competencies pertaining to the role of peer reviewers in biomedical journals were eligible for inclusion. Two reviewers independently performed study screening and selection. Relevant statements were extracted, collated and classified into themes.

RESULTS

After screening 2763 citations and 600 full-text papers, 209 articles and 13 grey literature sources were included. A total of 1426 statements related to roles were extracted, resulting in 76 unique statements. These were grouped into 13 emergent themes: proficient experts in their field (3 items), dutiful/altruistic towards scientific community (7 items), familiar with journal (2 items), unbiased and ethical professionals (18 items), self-critical professionals (4 items), reliable professionals (7 items), skilled critics (15 items), respectful communicators (6 items), gatekeepers (2 items), educators (2 items), advocates for author/editor/reader (3 items) and advisors to editors (2 items). Roles that do not fall within the remit of peer reviewers were also identified (5 items). We also extracted 2026 statements related to peer reviewers' tasks, resulting in 73 unique statements. These were grouped under six themes: organisation and approach to reviewing (10 items), make general comments (10 items), assess and address content for each section of the manuscript (36 items), address ethical aspects (5 items), assess manuscript presentation (8 items) and provide recommendations (4 items).

CONCLUSIONS

Peer reviewers are expected to perform a large number of roles and tasks for biomedical journals. These warrant further discussion and clarification in order not to overburden these key actors.

Editors' perspectives on the peer-review process in biomedical journals: protocol for a qualitative study

INTRODUCTION

Despite dealing with scientific output and potentially having an impact on the quality of research published, the manuscript peer-review process itself has at times been criticised for being 'unscientific'. Research indicates that there are social and subjective dimensions of the peer-review process that contribute to this perception, including how key stakeholders-namely authors, editors and peer reviewers-communicate. In particular, it has been suggested that the expected roles and tasks of stakeholders need to be more clearly defined and communicated if the manuscript review process is to be improved. Disentangling current communication practices, and outlining the specific roles and tasks of the main actors, might be a first step towards establishing the design of interventions that counterbalance social influences on the peer-review process.The purpose of this article is to present a methodological design for a qualitative study exploring the communication practices within the manuscript review process of biomedical journals from the journal editors' point of view.

METHODS AND ANALYSIS

Semi-structured interviews will be carried out with editors of biomedical journals between October 2017 and February 2018. A heterogeneous sample of participants representing a wide range of biomedical journals will be sought through purposive maximum variation sampling, drawing from a professional network of contacts, publishers, conference participants and snowballing.Interviews will be thematically analysed following the method outlined by Braun and Clarke. The qualitative data analysis software NVivo V.11 will be used to aid data management and analysis.

ETHICS AND DISSEMINATION

This research project was evaluated and approved by the University of Split, Medical School Ethics Committee (2181-198-03-04-17-0029) in May 2017. Findings will be disseminated through a publication in a peer-reviewed journal and presentations during conferences.

Updated Editorial Guidance for Quality and Reliability of Research Output

Over the past few years, updated editorial policy statements of several associations have provided a platform for improving the quality of scientific research and publishing. The updates have particularly pointed to the need for following research reporting standards, authorship and contributorship regulations, implementing digital tools for the identification and crediting academic contributors, and moving towards optimal ethical open-access models. This article overviews some of the recent editorial policy statements of global editorial associations and reflects on the role of the regional counterparts in advancing scholarly publishing. One of the globally promoted documents is the Recommendations of the International Committee of Medical Journal Editors (ICMJE). Its latest versions contain statements on proper research reporting, reviewing, editing, and publishing. Points on ethical target journals and 'predatory' sources are also available. This year, in a move to update its editorial policy, the Committee on Publication Ethics (COPE) released the Core Practices, comprehensively reflecting on the major issues in publication ethics. Updated joint statements of medical writers associations are also available to implement transparent policy on contributorship in sponsor-supported research projects and related reports. Several suggestions are put forward to improve global editorial statements on online profiling, crediting, and referencing. It is also highlighted that knowledge and implementation of updated editorial guidance is essential for editors' good standing.

What is a predatory journal? A scoping review

Background: There is no standardized definition of what a predatory journal is, nor have the characteristics of these journals been delineated or agreed upon. In order to study the phenomenon precisely a definition of predatory journals is needed. The objective of this scoping review is to summarize the literature on predatory journals, describe its epidemiological characteristics, and to extract empirical descriptions of potential characteristics of predatory journals. Methods: We searched five bibliographic databases: Ovid MEDLINE, Embase Classic + Embase, ERIC, and PsycINFO, and Web of Science on January 2 nd, 2018. A related grey literature search was conducted March 27 th, 2018. Eligible studies were those published in English after 2012 that discuss predatory journals. Titles and abstracts of records obtained were screened. We extracted epidemiological characteristics from all search records discussing predatory journals. Subsequently, we extracted statements from the empirical studies describing empirically derived characteristics of predatory journals. These characteristics were then categorized and thematically grouped.   Results: 920 records were obtained from the search. 344 of these records met our inclusion criteria. The majority of these records took the form of commentaries, viewpoints, letters, or editorials (78.44%), and just 38 records were empirical studies that reported empirically derived characteristics of predatory journals. We extracted 109 unique characteristics from these 38 studies, which we subsequently thematically grouped into six categories: journal operations, article, editorial and peer review, communication, article processing charges, and dissemination, indexing and archiving, and five descriptors.    Conclusions: This work identified a corpus of potential characteristics of predatory journals. Limitations of the work include our restriction to English language articles, and the fact that the methodological quality of articles included in our extraction was not assessed. These results will be provided to attendees at a stakeholder meeting seeking to develop a standardized definition for what constitutes a predatory journal.

What is a predatory journal? A scoping review

Background: There is no standardized definition of what a predatory journal is, nor have the characteristics of these journals been delineated or agreed upon. In order to study the phenomenon precisely a definition of predatory journals is needed. The objective of this scoping review is to summarize the literature on predatory journals, describe its epidemiological characteristics, and to extract empirical descriptions of potential characteristics of predatory journals. Methods: We searched five bibliographic databases: Ovid MEDLINE, Embase Classic + Embase, ERIC, and PsycINFO, and Web of Science on January 2 nd, 2018. A related grey literature search was conducted March 27 th, 2018. Eligible studies were those published in English after 2012 that discuss predatory journals. Titles and abstracts of records obtained were screened. We extracted epidemiological characteristics from all search records discussing predatory journals. Subsequently, we extracted statements from the empirical studies describing empirically derived characteristics of predatory journals. These characteristics were then categorized and thematically grouped.   Results: 920 records were obtained from the search. 344 of these records met our inclusion criteria. The majority of these records took the form of commentaries, viewpoints, letters, or editorials (78.44%), and just 38 records were empirical studies that reported empirically derived characteristics of predatory journals. We extracted 109 unique characteristics from these 38 studies, which we subsequently thematically grouped into six categories: journal operations, article, editorial and peer review, communication, article processing charges, and dissemination, indexing and archiving, and five descriptors.    Conclusions: This work identified a corpus of potential characteristics of predatory journals. Limitations of the work include our restriction to English language articles, and the fact that the methodological quality of articles included in our extraction was not assessed. These results will be provided to attendees at a stakeholder meeting seeking to develop a standardized definition for what constitutes a predatory journal.

Gatekeepers of Reward: a Pilot Study on the Ethics of Editing and Competing Evaluations of Value

The reward infrastructure in science centres on publication, in which journal editors play a key role. Reward distribution hinges on value assessments performed by editors, who draw from plural value systems to judge manuscripts. This conceptual paper examines the numerous biases and other factors that affect editorial decisions. Hybrid and often conflicting value systems contribute to an infrastructure in which editors manage reward through editorial review, commissioned commentaries and reviews and weighing of peer review judgments. Taken together, these systems and processes push the editor into a role resembling censorship. Editors and authors both experience this phenomenon as an unintended side-effect of the reward infrastructure in science. To work towards a more constructive editor-author relationship, we propose a conversation, an exchange between editor and author in which value is collectively assessed (or constructed) as obligatory passage points in the publishing process are traversed. This paper contributes to the discourse on editorial practices by problematising editorial paradigms in a new way and suggesting solutions to entrenched problems.

Improving the biomedical research literature: insights from authors' editors can help journal editors define and refine their core competencies

A team of stakeholders in biomedical publishing recently proposed a set of core competencies for journal editors, as a resource that can inform training programs for editors and ultimately improve the quality of the biomedical research literature. This initiative, still in its early stages, would benefit from additional sources of expert information. Based on our experiences as authors’ editors, we offer two suggestions on how to strengthen these competencies so that they better respond to the needs of readers and authors – the main users of and contributors to research journals. First, journal editors should be able to ensure that authors are given useful feedback on the language and writing in submitted manuscripts, beyond a (possibly incorrect) blanket judgement of whether the English is “acceptable” or not. Second, journal editors should be able to deal effectively with inappropriate text re-use and plagiarism. These additional competencies would, we believe, be valued by other stakeholders in biomedical research publication as markers of editorial quality.

A scoping review protocol on the roles and tasks of peer reviewers in the manuscript review process in biomedical journals

INTRODUCTION

The primary functions of peer reviewers are poorly defined. Thus far no body of literature has systematically identified the roles and tasks of peer reviewers of biomedical journals. A clear establishment of these can lead to improvements in the peer review process. The purpose of this scoping review is to determine what is known on the roles and tasks of peer reviewers.

METHODS

We will use the methodological framework first proposed by Arksey and O'Malley and subsequently adapted by Levac et al and the Joanna Briggs Institute. The scoping review will include all study designs, as well as editorials, commentaries and grey literature. The following eight electronic databases will be searched (from inception to May 2017): Cochrane Library, Cumulative Index to Nursing and Allied Health Literature, Educational Resources Information Center, EMBASE, MEDLINE, PsycINFO, Scopus and Web of Science. Two reviewers will use inclusion and exclusion criteria based on the 'Population-Concept-Context' framework to independently screen titles and abstracts of articles considered for inclusion. Full-text screening of relevant eligible articles will also be carried out by two reviewers. The search strategy for grey literature will include searching in websites of existing networks, biomedical journal publishers and organisations that offer resources for peer reviewers. In addition we will review journal guidelines to peer reviewers on how to perform the manuscript review. Journals will be selected using the 2016 journal impact factor. We will identify and assess the top five, middle five and lowest-ranking five journals across all medical specialties.

ETHICS AND DISSEMINATION

This scoping review will undertake a secondary analysis of data already collected and does not require ethical approval. The results will be disseminated through journals and conferences targeting stakeholders involved in peer review in biomedical research.

Core competencies for scientific editors of biomedical journals: consensus statement

BACKGROUND

Scientific editors are responsible for deciding which articles to publish in their journals. However, we have not found documentation of their required knowledge, skills, and characteristics, or the existence of any formal core competencies for this role.

METHODS

We describe the development of a minimum set of core competencies for scientific editors of biomedical journals.

RESULTS

The 14 key core competencies are divided into three major areas, and each competency has a list of associated elements or descriptions of more specific knowledge, skills, and characteristics that contribute to its fulfillment.

CONCLUSIONS

We believe that these core competencies are a baseline of the knowledge, skills, and characteristics needed to perform competently the duties of a scientific editor at a biomedical journal.

An international survey and modified Delphi process revealed editors' perceptions, training needs, and ratings of competency-related statements for the development of core competencies for scientific editors of biomedical journals

Background: Scientific editors (i.e., those who make decisions on the content and policies of a journal) have a central role in the editorial process at biomedical journals. However, very little is known about the training needs of these editors or what competencies are required to perform effectively in this role. Methods: We conducted a survey of perceptions and training needs among scientific editors from major editorial organizations around the world, followed by a modified Delphi process in which we invited the same scientific editors to rate the importance of competency-related statements obtained from a previous scoping review. Results: A total of 148 participants completed the survey of perceptions and training needs. At least 80% of participants agreed on six of the 38 skill and expertise-related statements presented to them as being important or very important to their role as scientific editors. At least 80% agreed on three of the 38 statements as necessary skills they perceived themselves as possessing (well or very well).  The top five items on participants' list of top training needs were training in statistics, research methods, publication ethics, recruiting and dealing with peer reviewers, and indexing of journals. The three rounds of the Delphi were completed by 83, 83, and 73 participants, respectively, which ultimately produced a list of 23 "highly rated" competency-related statements and another 86 "included" items. Conclusion: Both the survey and the modified Delphi process will be critical for understanding knowledge and training gaps among scientific editors when designing curriculum around core competencies in the future.

CARE guidelines for case reports: explanation and elaboration document

BACKGROUND

Well-written and transparent case reports (1) reveal early signals of potential benefits, harms, and information on the use of resources; (2) provide information for clinical research and clinical practice guidelines, and (3) inform medical education. High-quality case reports are more likely when authors follow reporting guidelines. During 2011-2012, a group of clinicians, researchers, and journal editors developed recommendations for the accurate reporting of information in case reports that resulted in the CARE (CAse REport) Statement and Checklist. They were presented at the 2013 International Congress on Peer Review and Biomedical Publication, have been endorsed by multiple medical journals, and translated into nine languages.

OBJECTIVES

This explanation and elaboration document has the objective to increase the use and dissemination of the CARE Checklist in writing and publishing case reports.

ARTICLE DESIGN AND SETTING

Each item from the CARE Checklist is explained and accompanied by published examples. The explanations and examples in this document are designed to support the writing of high-quality case reports by authors and their critical appraisal by editors, peer reviewers, and readers.

RESULTS AND CONCLUSION

This article and the 2013 CARE Statement and Checklist, available from the CARE website [www.care-statement.org] and the EQUATOR Network [www.equator-network.org], are resources for improving the completeness and transparency of case reports.

Randomized controlled trials in central vascular access devices: A scoping review

BACKGROUND

Randomized controlled trials evaluate the effectiveness of interventions for central venous access devices, however, high complication rates remain. Scoping reviews map the available evidence and demonstrate evidence deficiencies to focus ongoing research priorities.

METHOD

A scoping review (January 2006-December 2015) of randomized controlled trials evaluating the effectiveness of interventions to improve central venous access device outcomes; including peripherally inserted central catheters, non-tunneled, tunneled and totally implanted venous access catheters. MeSH terms were used to undertake a systematic search with data extracted by two independent researchers, using a standardized data extraction form.

RESULTS

In total, 178 trials were included (78 non-tunneled [44%]; 40 peripherally inserted central catheters [22%]; 20 totally implanted [11%]; 12 tunneled [6%]; 6 non-specified [3%]; and 22 combined device trials [12%]). There were 119 trials (68%) involving adult participants only, with 18 (9%) pediatric and 20 (11%) neonatal trials. Insertion-related themes existed in 38% of trials (67 RCTs), 35 RCTs (20%) related to post-insertion patency, with fewer trials on infection prevention (15 RCTs, 8%), education (14RCTs, 8%), and dressing and securement (12 RCTs, 7%). There were 46 different study outcomes reported, with the most common being infection outcomes (161 outcomes; 37%), with divergent definitions used for catheter-related bloodstream and other infections.

CONCLUSION

More high quality randomized trials across central venous access device management are necessary, especially in dressing and securement and patency. These can be encouraged by having more studies with multidisciplinary team involvement and consumer engagement. Additionally, there were extensive gaps within population sub-groups, particularly in tunneled devices, and in pediatrics and neonates. Finally, outcome definitions need to be unified for results to be meaningful and comparable across studies.

Surveys on Reporting Guideline Usage in Dental Journals

The objectives of this study were 1) to find out if and how authors and peer reviewers for dental journals are encouraged to use reporting guidelines (RGs); 2) to identify factors related to RG endorsement; and 3) to assess the knowledge, opinions, and future plans of dental journal editors in chief (EICs) on RGs. A total of 109 peer-reviewed and original research-oriented dental journals that were indexed in the MEDLINE and/or SCIE database in 2015 were included. The "instructions to authors" and "instructions to reviewers" of these journals were identified and retrieved from journals' official websites. Any mention of RGs or other related policies were sought and extracted. In addition, an anonymous survey of the EICs of the included journals was conducted with a validated questionnaire. All 109 journals provided "instructions to authors," among which 55 (50.5%) mentioned RGs. Only the CONSORT (45.0%), PRISMA (13.8%), and STROBE (12.8%) guidelines were mentioned by >10% of the included journals. Statistical analyses suggest that RGs were more frequently mentioned by SCIE-indexed journals (P < 0.001), higher-impact journals (P = 0.002), and journals that endorsed the ICMJE recommendations (P < 0.001). "Instructions to reviewers" were available online for only 9 journals (8.3%), 3 of which mentioned RGs. For the EIC survey, the response rate was 32.1% (35 of 109). Twenty-six editors (74.3%) stated that they knew what RGs were before receiving our questionnaire. Twenty-four editors (68.6%) believed that RGs should be adopted by all refereed dental journals where appropriate. RGs are important tools for enhancing research reporting and reducing avoidable research waste, but currently they are not widely endorsed by dental journals. Joint efforts by all stakeholders to further promote RG usage in dentistry are needed.