A role for artificial intelligence chatbots in the writing of scientific articles

Letters generated by ChatGPT: Author who?

AIM

ChatGPT's role in medical writing is a topic of discussion. I experimented whether ChatGPT almost automatically generates Correspondence or Letter addressed to a "translated" article, and thereby wish to arouse discussion regarding ChatGPT use in medical writing.

METHODS

I input an English article of mine into ChatGPT, tasking it with generating an English Disagreement Letter (Letter 1). Next, I tasked ChatGPT with translating the manuscript addressed to from English-French-Spanish-German. Then, I once again tasked ChatGPT with generating an English Disagreement Letter addressed to a German manuscript (triplicate translated manuscript) (Letter 2).

RESULTS

Letters 1 and 2 are readable and reasonable, shooting the point that the author (myself) felt as the weakness of the article. Letters addressed to French (single translation) and to Spanish (double translation) and longer Letters (corresponding to Letters 1 and 2) are also readable, and thus stand.

CONCLUSIONS

Solely based on this experiment, one may be able to write a letter even without understanding the meaning of the paper being addressed, let alone the language of the paper. Although this humble experiment does not conclude anything, I plea for a comprehensive discussion on the implications of these findings.

Identification of Human-Generated vs AI-Generated Research Abstracts by Health Care Professionals

This survey study assesses the ability of health care professionals to discern whether abstracts were written by investigators or by an artificial intelligence (AI) chatbot.

Exploring the Limits of Artificial Intelligence for Referencing Scientific Articles

OBJECTIVE

 To evaluate the reliability of three artificial intelligence (AI) chatbots (ChatGPT, Google Bard, and Chatsonic) in generating accurate references from existing obstetric literature.

STUDY DESIGN

 Between mid-March and late April 2023, ChatGPT, Google Bard, and Chatsonic were prompted to provide references for specific obstetrical randomized controlled trials (RCTs) published in 2020. RCTs were considered for inclusion if they were mentioned in a previous article that primarily evaluated RCTs published by the top medical and obstetrics and gynecology journals with the highest impact factors in 2020 as well as RCTs published in a new journal focused on publishing obstetric RCTs. The selection of the three AI models was based on their popularity, performance in natural language processing, and public availability. Data collection involved prompting the AI chatbots to provide references according to a standardized protocol. The primary evaluation metric was the accuracy of each AI model in correctly citing references, including authors, publication title, journal name, and digital object identifier (DOI). Statistical analysis was performed using a permutation test to compare the performance of the AI models.

RESULTS

 Among the 44 RCTs analyzed, Google Bard demonstrated the highest accuracy, correctly citing 13.6% of the requested RCTs, whereas ChatGPT and Chatsonic exhibited lower accuracy rates of 2.4 and 0%, respectively. Google Bard often substantially outperformed Chatsonic and ChatGPT in correctly citing the studied reference components. The majority of references from all AI models studied were noted to provide DOIs for unrelated studies or DOIs that do not exist.

CONCLUSION

 To ensure the reliability of scientific information being disseminated, authors must exercise caution when utilizing AI for scientific writing and literature search. However, despite their limitations, collaborative partnerships between AI systems and researchers have the potential to drive synergistic advancements, leading to improved patient care and outcomes.

KEY POINTS

· AI chatbots often cite scientific articles incorrectly.. · AI chatbots can create false references.. · Responsible AI use in research is vital..