The use of artificial intelligence to improve the scientific writing of non-native english speakers

PICOT questions and search strategies formulation: A novel approach using artificial intelligence automation

AIM

The aim of this study was to evaluate and compare artificial intelligence (AI)-based large language models (LLMs) (ChatGPT-3.5, Bing, and Bard) with human-based formulations in generating relevant clinical queries, using comprehensive methodological evaluations.

METHODS

To interact with the major LLMs ChatGPT-3.5, Bing Chat, and Google Bard, scripts and prompts were designed to formulate PICOT (population, intervention, comparison, outcome, time) clinical questions and search strategies. Quality of the LLMs responses was assessed using a descriptive approach and independent assessment by two researchers. To determine the number of hits, PubMed, Web of Science, Cochrane Library, and CINAHL Ultimate search results were imported separately, without search restrictions, with the search strings generated by the three LLMs and an additional one by the expert. Hits from one of the scenarios were also exported for relevance evaluation. The use of a single scenario was chosen to provide a focused analysis. Cronbach's alpha and intraclass correlation coefficient (ICC) were also calculated.

RESULTS

In five different scenarios, ChatGPT-3.5 generated 11,859 hits, Bing 1,376,854, Bard 16,583, and an expert 5919 hits. We then used the first scenario to assess the relevance of the obtained results. The human expert search approach resulted in 65.22% (56/105) relevant articles. Bing was the most accurate AI-based LLM with 70.79% (63/89), followed by ChatGPT-3.5 with 21.05% (12/45), and Bard with 13.29% (42/316) relevant hits. Based on the assessment of two evaluators, ChatGPT-3.5 received the highest score (M = 48.50; SD = 0.71). Results showed a high level of agreement between the two evaluators. Although ChatGPT-3.5 showed a lower percentage of relevant hits compared to Bing, this reflects the nuanced evaluation criteria, where the subjective evaluation prioritized contextual accuracy and quality over mere relevance.

CONCLUSION

This study provides valuable insights into the ability of LLMs to formulate PICOT clinical questions and search strategies. AI-based LLMs, such as ChatGPT-3.5, demonstrate significant potential for augmenting clinical workflows, improving clinical query development, and supporting search strategies. However, the findings also highlight limitations that necessitate further refinement and continued human oversight.

CLINICAL RELEVANCE

AI could assist nurses in formulating PICOT clinical questions and search strategies. AI-based LLMs offer valuable support to healthcare professionals by improving the structure of clinical questions and enhancing search strategies, thereby significantly increasing the efficiency of information retrieval.

Artificial Intelligence: A Challenge to Scientific Communication

Recent years have seen formidable advances in artificial intelligence. Developments include a large number of specialised systems either existing or planned for use in scientific research, data analysis, translation, text production and design with grammar checking and stylistic revision, plagiarism detection, and scientific review in addition to general-purpose AI systems for searching the internet and generative AI systems for texts, images, videos, and musical compositions. These systems promise more ease and simplicity in many aspects of work. Blind trust in AI systems with uncritical, careless use of AI results is dangerous, as these systems do not have any inherent understanding of the content they process or generate, but only simulate this understanding by reproducing statistical patterns extracted from training data. This article discusses the potential and risk of using AI in scientific communication and explores potential systemic consequences of widespread AI implementation in this context.

Introducing Our Custom GPT: An Example of the Potential Impact of Personalized GPT Builders on Scientific Writing

BACKGROUND

The rapid progression of artificial intelligence (AI) and large language models (LLMs), such as ChatGPT, has contributed to increase its utility and popularity in various fields. Discourse about AI's potential role in different aspects of scientific literature such as writing, data analysis, and literature review, is growing as the programs continue to improve their capabilities. This study utilizes a recently released ChatGPT tool that allows users to create customized GPTs to highlight the potential of customizable GPTs tailored to prepare and write research manuscripts.

METHODS

We developed our 2 GPTs, Neurosurgical Research Paper Writer and Medi Research Assistant, through iterative refinement of ChatGPT 4.0's tool, GPT Builder. This process included providing specific and thorough instructions along with repetitive testing and feedback-driven adjustments to finalize a version of the model that fit our needs.

RESULTS

The GPT models that we created were able to efficiently and consistently produce accurate outputs from inputted prompts based on their specific configurations. It effectively analyzed existing literature that it found and synthesized information in ways that were reliable and written in ways comparable to manuscripts authored by scientific professionals.

CONCLUSIONS

While the ability of modern AI to generate scientific manuscripts has shown significant progress, the persistence of fallacies and miscalculations suggest that the development of GPTs requires extensive calibration before achieving greater reliability and consistency. Nevertheless, the prospective horizon of AI-driven research holds promise in streamlining the publication workflow and increasing accessibility to novel research.

Reference Hallucination Score for Medical Artificial Intelligence Chatbots: Development and Usability Study

BACKGROUND

Artificial intelligence (AI) chatbots have recently gained use in medical practice by health care practitioners. Interestingly, the output of these AI chatbots was found to have varying degrees of hallucination in content and references. Such hallucinations generate doubts about their output and their implementation.

OBJECTIVE

The aim of our study was to propose a reference hallucination score (RHS) to evaluate the authenticity of AI chatbots' citations.

METHODS

Six AI chatbots were challenged with the same 10 medical prompts, requesting 10 references per prompt. The RHS is composed of 6 bibliographic items and the reference's relevance to prompts' keywords. RHS was calculated for each reference, prompt, and type of prompt (basic vs complex). The average RHS was calculated for each AI chatbot and compared across the different types of prompts and AI chatbots.

RESULTS

Bard failed to generate any references. ChatGPT 3.5 and Bing generated the highest RHS (score=11), while Elicit and SciSpace generated the lowest RHS (score=1), and Perplexity generated a middle RHS (score=7). The highest degree of hallucination was observed for reference relevancy to the prompt keywords (308/500, 61.6%), while the lowest was for reference titles (169/500, 33.8%). ChatGPT and Bing had comparable RHS (β coefficient=-0.069; P=.32), while Perplexity had significantly lower RHS than ChatGPT (β coefficient=-0.345; P<.001). AI chatbots generally had significantly higher RHS when prompted with scenarios or complex format prompts (β coefficient=0.486; P<.001).

CONCLUSIONS

The variation in RHS underscores the necessity for a robust reference evaluation tool to improve the authenticity of AI chatbots. Further, the variations highlight the importance of verifying their output and citations. Elicit and SciSpace had negligible hallucination, while ChatGPT and Bing had critical hallucination levels. The proposed AI chatbots' RHS could contribute to ongoing efforts to enhance AI's general reliability in medical research.

Assessing the Accuracy, Completeness, and Reliability of Artificial Intelligence-Generated Responses in Dentistry: A Pilot Study Evaluating the ChatGPT Model

BACKGROUND

Artificial intelligence (AI) can be a tool in the diagnosis and acquisition of knowledge, particularly in dentistry, sparking debates on its application in clinical decision-making.

OBJECTIVE

This study aims to evaluate the accuracy, completeness, and reliability of the responses generated by Chatbot Generative Pre-Trained Transformer (ChatGPT) 3.5 in dentistry using expert-formulated questions.

MATERIALS AND METHODS

Experts were invited to create three questions, answers, and respective references according to specialized fields of activity. The Likert scale was used to evaluate agreement levels between experts and ChatGPT responses. Statistical analysis compared descriptive and binary question groups in terms of accuracy and completeness. Questions with low accuracy underwent re-evaluation, and subsequent responses were compared for improvement. The Wilcoxon test was utilized (α = 0.05).

RESULTS

Ten experts across six dental specialties generated 30 binary and descriptive dental questions and references. The accuracy score had a median of 5.50 and a mean of 4.17. For completeness, the median was 2.00 and the mean was 2.07. No difference was observed between descriptive and binary responses for accuracy and completeness. However, re-evaluated responses showed a significant improvement with a significant difference in accuracy (median 5.50 vs. 6.00; mean 4.17 vs. 4.80; p=0.042) and completeness (median 2.0 vs. 2.0; mean 2.07 vs. 2.30; p=0.011). References were more incorrect than correct, with no differences between descriptive and binary questions.

CONCLUSIONS

ChatGPT initially demonstrated good accuracy and completeness, which was further improved with machine learning (ML) over time. However, some inaccurate answers and references persisted. Human critical discernment continues to be essential to facing complex clinical cases and advancing theoretical knowledge and evidence-based practice.

Bridging Gaps: A Quality Improvement Project for the Continuing Medical Education on Stick (CMES) Program

BACKGROUND

Aimed at bridging the gap in continuing medical education (CME) resource availability in low- and middle-income countries (LMICs), the "Continuing Medical Education on Stick" (CMES) program introduces two technological solutions: a universal serial bus (USB) drive and the CMES-Pi computer facilitating access to monthly updated CME content without data cost. Feedback from users suggests a lack of content on tropical infectious diseases (IDs) and content from a Western perspective, which may be less relevant in LMIC settings.

METHODS

This quality improvement project was intended to identify areas for improvement of the CMES database to better meet the educational needs of users. We compared the CMES content with the American Board of Emergency Medicine (ABEM) Exam content outline to identify gaps. The curriculum map of the CMES library, encompassing content from 2019 to 2024, was reviewed. An anonymous survey was conducted among 47 global users to gather feedback on unmet educational needs and suggestions for content improvements. All healthcare workers who were members of the CMES WhatsApp group were eligible to participate in the survey.

RESULTS

The curriculum map included 2,572 items categorized into 23 areas. The comparison with the ABEM outline identified gaps in several clinical areas, including procedures, traumatic disorders, and geriatrics, which were represented -5%, -5%, and -4% in the CMES library compared with the ABEM outline, respectively. Free responses from users highlighted a lack of content on practical skills, such as electrocardiogram (ECG) interpretation and management of tropical diseases. Respondents identified emergency medical services (EMS)/prehospital care (81%), diagnostic imaging (62%), and toxicology/pharmacology (40%) as the most beneficial areas for clinical practice. In response to feedback from users, new content was added to the CMES platform on the management of sickle cell disease and dermatologic conditions in darkly pigmented skin. Furthermore, a targeted podcast series called "ID for Users of the CMES Program (ID4U)" has been launched, focusing on tropical and locally relevant ID, with episodes now being integrated into the CMES platform.

CONCLUSIONS

The project pinpointed critical gaps in emergency medicine (EM) content pertinent to LMICs and led to targeted enhancements in the CMES library. Ongoing updates will focus on including more prehospital medicine, diagnostic imaging, and toxicology content. Further engagement with users and education on utilizing the CMES platform will be implemented to maximize its educational impact. Future adaptations will consider local relevance over the ABEM curriculum to better serve the diverse needs of global users.

Characterizing the Increase in Artificial Intelligence Content Detection in Oncology Scientific Abstracts From 2021 to 2023

PURPOSE

Artificial intelligence (AI) models can generate scientific abstracts that are difficult to distinguish from the work of human authors. The use of AI in scientific writing and performance of AI detection tools are poorly characterized.

METHODS

We extracted text from published scientific abstracts from the ASCO 2021-2023 Annual Meetings. Likelihood of AI content was evaluated by three detectors: GPTZero, Originality.ai, and Sapling. Optimal thresholds for AI content detection were selected using 100 abstracts from before 2020 as negative controls, and 100 produced by OpenAI's GPT-3 and GPT-4 models as positive controls. Logistic regression was used to evaluate the association of predicted AI content with submission year and abstract characteristics, and adjusted odds ratios (aORs) were computed.

RESULTS

Fifteen thousand five hundred and fifty-three abstracts met inclusion criteria. Across detectors, abstracts submitted in 2023 were significantly more likely to contain AI content than those in 2021 (aOR range from 1.79 with Originality to 2.37 with Sapling). Online-only publication and lack of clinical trial number were consistently associated with AI content. With optimal thresholds, 99.5%, 96%, and 97% of GPT-3/4-generated abstracts were identified by GPTZero, Originality, and Sapling respectively, and no sampled abstracts from before 2020 were classified as AI generated by the GPTZero and Originality detectors. Correlation between detectors was low to moderate, with Spearman correlation coefficient ranging from 0.14 for Originality and Sapling to 0.47 for Sapling and GPTZero.

CONCLUSION

There is an increasing signal of AI content in ASCO abstracts, coinciding with the growing popularity of generative AI models.

A review of top cardiology and cardiovascular medicine journal guidelines regarding the use of generative artificial intelligence tools in scientific writing

BACKGROUND

Generative Artificial Intelligence (AI) tools have experienced rapid development over the last decade and are gaining increasing popularity as assistive models in academic writing. However, the ability of AI to generate reliable and accurate research articles is a topic of debate. Major scientific journals have issued policies regarding the contribution of AI tools in scientific writing.

METHODS

We conducted a review of the author and peer reviewer guidelines of the top 25 Cardiology and Cardiovascular Medicine journals as per the 2023 SCImago rankings. Data were obtained though reviewing journal websites and directly emailing the editorial office. Descriptive data regarding journal characteristics were coded on SPSS. Subgroup analyses of the journal guidelines were conducted based on the publishing company policies.

RESULTS

Our analysis revealed that all scientific journals in our study permitted the documented use of AI in scientific writing with certain limitations as per ICMJE recommendations. We found that AI tools cannot be included in the authorship or be used for image generation, and that all authors are required to assume full responsibility of their submitted and published work. The use of generative AI tools in the peer review process is strictly prohibited.

CONCLUSION

Guidelines regarding the use of generative AI in scientific writing are standardized, detailed, and unanimously followed by all journals in our study according to the recommendations set forth by international forums. It is imperative to ensure that these policies are carefully followed and updated to maintain scientific integrity.

Between human and AI: assessing the reliability of AI text detection tools

OBJECTIVE

Large language models (LLMs) such as ChatGPT-4 have raised critical questions regarding their distinguishability from human-generated content. In this research, we evaluated the effectiveness of online detection tools in identifying ChatGPT-4 vs human-written text.

METHODS

A two texts produced by ChatGPT-4 using differing prompts and one text created by a human author were analytically assessed using the following online detection tools: GPTZero, ZeroGPT, Writer ACD, and Originality.

RESULTS

The findings revealed a notable variance in the detection capabilities of the employed detection tools. GPTZero and ZeroGPT exhibited inconsistent assessments regarding the AI-origin of the texts. Writer ACD predominantly identified texts as human-written, whereas Originality consistently recognized the AI-generated content in both samples from ChatGPT-4. This highlights Originality's enhanced sensitivity to patterns characteristic of AI-generated text.

CONCLUSION

The study demonstrates that while automatic detection tools may discern texts generated by ChatGPT-4 significant variability exists in their accuracy. Undoubtedly, there is an urgent need for advanced detection tools to ensure the authenticity and integrity of content, especially in scientific and academic research. However, our findings underscore an urgent need for more refined detection methodologies to prevent the misdetection of human-written content as AI-generated and vice versa.

Potential applications and implications of large language models in primary care

The recent release of highly advanced generative artificial intelligence (AI) chatbots, including ChatGPT and Bard, which are powered by large language models (LLMs), has attracted growing mainstream interest over its diverse applications in clinical practice, including in health and healthcare. The potential applications of LLM-based programmes in the medical field range from assisting medical practitioners in improving their clinical decision-making and streamlining administrative paperwork to empowering patients to take charge of their own health. However, despite the broad range of benefits, the use of such AI tools also comes with several limitations and ethical concerns that warrant further consideration, encompassing issues related to privacy, data bias, and the accuracy and reliability of information generated by AI. The focus of prior research has primarily centred on the broad applications of LLMs in medicine. To the author's knowledge, this is, the first article that consolidates current and pertinent literature on LLMs to examine its potential in primary care. The objectives of this paper are not only to summarise the potential benefits, risks and challenges of using LLMs in primary care, but also to offer insights into considerations that primary care clinicians should take into account when deciding to adopt and integrate such technologies into their clinical practice.

Artificial intelligence (AI) and ChatGPT involvement in scientific and medical writing, a new concern for researchers. A scoping review

Purpose

The study aims to evaluate PubMed publications on ChatGPT or artificial intelligence (AI) involvement in scientific or medical writing and investigate whether ChatGPT or AI was used to create these articles or listed as authors.

Design/methodology/approach

This scoping review was conducted according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews (PRISMA-ScR) guidelines. A PubMed database search was performed for articles published between January 1 and November 29, 2023, using appropriate search terms; both authors performed screening and selection independently.

Findings

From the initial search results of 127 articles, 41 were eligible for final analysis. Articles were published in 34 journals. Editorials were the most common article type, with 15 (36.6%) articles. Authors originated from 27 countries, and authors from the USA contributed the most, with 14 (34.1%) articles. The most discussed topic was AI tools and writing capabilities in 19 (46.3%) articles. AI or ChatGPT was involved in manuscript preparation in 31 (75.6%) articles. None of the articles listed AI or ChatGPT as an author, and in 19 (46.3%) articles, the authors acknowledged utilizing AI or ChatGPT.

Practical implications

Researchers worldwide are concerned with AI or ChatGPT involvement in scientific research, specifically the writing process. The authors believe that precise and mature regulations will be developed soon by journals, publishers and editors, which will pave the way for the best usage of these tools.

Originality/value

This scoping review expressed data published on using AI or ChatGPT in various scientific research and writing aspects, besides alluding to the advantages, disadvantages and implications of their usage.