ChatGPT in academic publishing: An ally or an adversary?

Using the Chat Generative Pre-trained Transformer in academic writing in health: a scoping review

OBJECTIVE

to map the scientific literature regarding the use of the Chat Generative Pre-trained Transformer, ChatGPT, in academic writing in health.

METHOD

this was a scoping review, following the JBI methodology. Conventional databases and gray literature were included. The selection of studies was applied after removing duplicates and individual and paired evaluation. Data were extracted based on an elaborate script, and presented in a descriptive, tabular and graphical format.

RESULTS

the analysis of the 49 selected articles revealed that ChatGPT is a versatile tool, contributing to scientific production, description of medical procedures and preparation of summaries aligned with the standards of scientific journals. Its application has been shown to improve the clarity of writing and benefits areas such as innovation and automation. Risks were also observed, such as the possibility of lack of originality and ethical issues. Future perspectives highlight the need for adequate regulation, agile adaptation and the search for an ethical balance in incorporating ChatGPT into academic writing.

CONCLUSION

ChatGPT presents transformative potential in academic writing in health. However, its adoption requires rigorous human supervision, solid regulation, and transparent guidelines to ensure its responsible and beneficial use by the scientific community.

An evaluation of AI generated literature reviews in musculoskeletal radiology

PURPOSE

The use of artificial intelligence (AI) tools to aid in summarizing information in medicine and research has recently garnered a huge amount of interest. While tools such as ChatGPT produce convincing and naturally sounding output, the answers are sometimes incorrect. Some of these drawbacks, it is hoped, can be avoided by using programmes trained for a more specific scope. In this study we compared the performance of a new AI tool (the-literature.com) to the latest version OpenAI's ChatGPT (GPT-4) in summarizing topics that the authors have significantly contributed to.

METHODS

The AI tools were asked to produce a literature review on 7 topics. These were selected based on the research topics that the authors were intimately familiar with and have contributed to through their own publications. The output produced by the AI tools were graded on a 1-5 Likert scale for accuracy, comprehensiveness, and relevance by two fellowship trained consultant radiologists.

RESULTS

The-literature.com produced 3 excellent summaries, 3 very poor summaries not relevant to the prompt, and one summary, which was relevant but did not include all relevant papers. All of the summaries produced by GPT-4 were relevant, but fewer relevant papers were identified. The average Likert rating was for the-literature was 2.88 and 3.86 for GPT-4. There was good agreement between the ratings of both radiologists (ICC = 0.883).

CONCLUSION

Summaries produced by AI in its current state require careful human validation. GPT-4 on average provides higher quality summaries. Neither tool can reliably identify all relevant publications.

Could ChatGPT Pass the UK Radiology Fellowship Examinations?

RATIONALE AND OBJECTIVES

Chat Generative Pre-trained Transformer (ChatGPT) is an artificial intelligence (AI) tool which utilises machine learning to generate original text resembling human language. AI models have recently demonstrated remarkable ability at analysing and solving problems, including passing professional examinations. We investigate the performance of ChatGPT on some of the UK radiology fellowship equivalent examination questions.

METHODS

ChatGPT was asked to answer questions from question banks resembling the Fellowship of the Royal College of Radiologists (FRCR) examination. The entire physics part 1 question bank (203 5-part true/false questions) was answered by the GPT-4 model and answers recorded. 240 single best answer questions (SBAs) (representing the true length of the FRCR 2A examination) were answered by both GPT-3.5 and GPT-4 models.

RESULTS

ChatGPT 4 answered 74.8% of part 1 true/false statements correctly. The spring 2023 passing mark of the part 1 examination was 75.5% and ChatGPT thus narrowly failed. In the 2A examination, ChatGPT 3.5 answered 50.8% SBAs correctly, while GPT-4 answered 74.2% correctly. The winter 2022 2A pass mark was 63.3% and thus GPT-4 clearly passed.

CONCLUSION

AI models such as ChatGPT are able to answer the majority of questions in an FRCR style examination. It is reasonable to assume that further developments in AI will be more likely to succeed in comprehending and solving questions related to medicine, specifically clinical radiology.

ADVANCES IN KNOWLEDGE

Our findings outline the unprecedented capabilities of AI, adding to the current relatively small body of literature on the subject, which in turn can play a role medical training, evaluation and practice. This can undoubtedly have implications for radiology.

Exploring the potential of ChatGPT in the peer review process: An observational study

BACKGROUND

Peer review is the established method for evaluating the quality and validity of research manuscripts in scholarly publishing. However, scientific peer review faces challenges as the volume of submitted research has steadily increased in recent years. Time constraints and peer review quality assurance can place burdens on reviewers, potentially discouraging their participation. Some artificial intelligence (AI) tools might assist in relieving these pressures. This study explores the efficiency and effectiveness of one of the artificial intelligence (AI) chatbots, ChatGPT (Generative Pre-trained Transformer), in the peer review process.

METHODS

Twenty-one peer-reviewed research articles were anonymised to ensure unbiased evaluation. Each article was reviewed by two humans and by versions 3.5 and 4.0 of ChatGPT. The AI was instructed to provide three positive and three negative comments on the articles and recommend whether they should be accepted or rejected. The human and AI results were compared using a 5-point Likert scale to determine the level of agreement. The correlation between ChatGPT responses and the acceptance or rejection of the papers was also examined.

RESULTS

Subjective review similarity between human reviewers and ChatGPT showed a mean score of 3.6/5 for ChatGPT 3.5 and 3.76/5 for ChatGPT 4.0. The correlation between human and AI review scores was statistically significant for ChatGPT 3.5, but not for ChatGPT 4.0.

CONCLUSION

ChatGPT can complement human scientific peer review, enhancing efficiency and promptness in the editorial process. However, a fully automated AI review process is currently not advisable, and ChatGPT's role should be regarded as highly constrained for the present and near future.

Assessing ChatGPT's ability to pass the FRCS orthopaedic part A exam: A critical analysis

AI technology has made significant advancements in recent years, with the notable development of ChatGPT in November 2022. Users have observed evidence of deductive reasoning, logical thinking, and coherent thought in ChatGPT's responses. This study aimed to determine if ChatGPT has the capability to pass the Orthopaedic Fellow of the Royal College of Surgeons (FRCS Orth) Part A exam.

METHODS

To assess ChatGPT4's ability to pass the Orthopaedic FRCS Orth Part A exam, a study was conducted using 240 mock FRCS Orth Part A questions. The study evaluated the accuracy of ChatGPT's answers and the response time for each question. Descriptive statistics were employed to analyse the chatbot's performance.

RESULTS

The evaluation revealed that ChatGPT4 achieved an overall score of 67.5% on Part A of the exam. However, ChatGPT4 did not meet the overall pass mark required for the FRCS Orth Part A exam.

CONCLUSION

This study demonstrates that ChatGPT was unable to pass the FRCS Orthopaedic examination. Several factors contributed to this outcome, including the lack of critical or high-order thinking abilities, limited clinical expertise, and the inability to meet the rigorous requirements of the exam.