Using ChatGPT to conduct a literature review

Using GPT-4 to write a scientific review article: a pilot evaluation study

GPT-4, as the most advanced version of OpenAI's large language models, has attracted widespread attention, rapidly becoming an indispensable AI tool across various areas. This includes its exploration by scientists for diverse applications. Our study focused on assessing GPT-4's capabilities in generating text, tables, and diagrams for biomedical review papers. We also assessed the consistency in text generation by GPT-4, along with potential plagiarism issues when employing this model for the composition of scientific review papers. Based on the results, we suggest the development of enhanced functionalities in ChatGPT, aiming to meet the needs of the scientific community more effectively. This includes enhancements in uploaded document processing for reference materials, a deeper grasp of intricate biomedical concepts, more precise and efficient information distillation for table generation, and a further refined model specifically tailored for scientific diagram creation.

Title and abstract screening for literature reviews using large language models: an exploratory study in the biomedical domain

BACKGROUND

Systematically screening published literature to determine the relevant publications to synthesize in a review is a time-consuming and difficult task. Large language models (LLMs) are an emerging technology with promising capabilities for the automation of language-related tasks that may be useful for such a purpose.

METHODS

LLMs were used as part of an automated system to evaluate the relevance of publications to a certain topic based on defined criteria and based on the title and abstract of each publication. A Python script was created to generate structured prompts consisting of text strings for instruction, title, abstract, and relevant criteria to be provided to an LLM. The relevance of a publication was evaluated by the LLM on a Likert scale (low relevance to high relevance). By specifying a threshold, different classifiers for inclusion/exclusion of publications could then be defined. The approach was used with four different openly available LLMs on ten published data sets of biomedical literature reviews and on a newly human-created data set for a hypothetical new systematic literature review.

RESULTS

The performance of the classifiers varied depending on the LLM being used and on the data set analyzed. Regarding sensitivity/specificity, the classifiers yielded 94.48%/31.78% for the FlanT5 model, 97.58%/19.12% for the OpenHermes-NeuralChat model, 81.93%/75.19% for the Mixtral model and 97.58%/38.34% for the Platypus 2 model on the ten published data sets. The same classifiers yielded 100% sensitivity at a specificity of 12.58%, 4.54%, 62.47%, and 24.74% on the newly created data set. Changing the standard settings of the approach (minor adaption of instruction prompt and/or changing the range of the Likert scale from 1-5 to 1-10) had a considerable impact on the performance.

CONCLUSIONS

LLMs can be used to evaluate the relevance of scientific publications to a certain review topic and classifiers based on such an approach show some promising results. To date, little is known about how well such systems would perform if used prospectively when conducting systematic literature reviews and what further implications this might have. However, it is likely that in the future researchers will increasingly use LLMs for evaluating and classifying scientific publications.

Utility of Artificial Intelligence in Orthopedic Surgery Literature Review: A Comparative Pilot Study

OBJECTIVE

Literature reviews are essential to the scientific process and allow clinician researchers to advance general knowledge. The purpose of this study was to evaluate if the artificial intelligence (AI) programs ChatGPT and Perplexity.AI can perform an orthopedic surgery literature review.

MATERIALS AND METHODS

Five different search topics of varying specificity within orthopedic surgery were chosen for each search arm to investigate. A consolidated list of unique articles for each search topic was recorded for the experimental AI search arms and compared with the results of the control arm of two independent reviewers. Articles in the experimental arms were examined by the two independent reviewers for relevancy and validity.

RESULTS

ChatGPT was able to identify a total of 61 unique articles. Four articles were not relevant to the search topic and 51 articles were deemed to be fraudulent, resulting in 6 valid articles. Perplexity.AI was able to identify a total of 43 unique articles. Nineteen were not relevant to the search topic but all articles were able to be verified, resulting in 24 valid articles. The control arm was able to identify 132 articles. Success rates for ChatGPT and Perplexity. AI were 4.6% (6 of 132) and 18.2% (24 of 132), respectively.

CONCLUSION

The current iteration of ChatGPT cannot perform a reliable literature review, and Perplexity.AI is only able to perform a limited review of the medical literature. Any utilization of these open AI programs should be done with caution and human quality assurance to promote responsible use and avoid the risk of using fabricated search results. [Orthopedics. 2024;47(3):e125-e130.].

The Application of ChatGPT in Medicine: A Scoping Review and Bibliometric Analysis

Purpose

ChatGPT has a wide range of applications in the medical field. Therefore, this review aims to define the key issues and provide a comprehensive view of the literature based on the application of ChatGPT in medicine.

Methods

This scope follows Arksey and O'Malley's five-stage framework. A comprehensive literature search of publications (30 November 2022 to 16 August 2023) was conducted. Six databases were searched and relevant references were systematically catalogued. Attention was focused on the general characteristics of the articles, their fields of application, and the advantages and disadvantages of using ChatGPT. Descriptive statistics and narrative synthesis methods were used for data analysis.

Results

Of the 3426 studies, 247 met the criteria for inclusion in this review. The majority of articles (31.17%) were from the United States. Editorials (43.32%) ranked first, followed by experimental studys (11.74%). The potential applications of ChatGPT in medicine are varied, with the largest number of studies (45.75%) exploring clinical practice, including assisting with clinical decision support and providing disease information and medical advice. This was followed by medical education (27.13%) and scientific research (16.19%). Particularly noteworthy in the discipline statistics were radiology, surgery and dentistry at the top of the list. However, ChatGPT in medicine also faces issues of data privacy, inaccuracy and plagiarism.

Conclusion

The application of ChatGPT in medicine focuses on different disciplines and general application scenarios. ChatGPT has a paradoxical nature: it offers significant advantages, but at the same time raises great concerns about its application in healthcare settings. Therefore, it is imperative to develop theoretical frameworks that not only address its widespread use in healthcare but also facilitate a comprehensive assessment. In addition, these frameworks should contribute to the development of strict and effective guidelines and regulatory measures.

Application of Artificial Intelligence to Patient-Targeted Health Information on Kidney Stone Disease

OBJECTIVE

The American Medical Association recommends health information to be written at a 6th grade level reading level. Our aim was to determine whether Artificial Intelligence can outperform the existing health information on kidney stone prevention and treatment.

METHODS

The top 50 search results for "Kidney Stone Prevention" and "Kidney Stone Treatment" on Google, Bing, and Yahoo were selected. Duplicate webpages, advertisements, pages intended for health professionals such as science articles, links to videos, paid subscription pages, and links nonrelated to kidney stone prevention and/or treatment were excluded. Included pages were categorized into academic, hospital-affiliated, commercial, nonprofit foundations, and other. Quality and readability of webpages were evaluated using validated tools, and the reading level was descriptively compared with ChatGPT generated health information on kidney stone prevention and treatment.

RESULTS

50 webpages on kidney stone prevention and 49 on stone treatment were included in this study. The reading level was determined to equate to that of a 10th to 12th grade student. Quality was measured as "fair" with no pages scoring "excellent" and only 20% receiving a "good" quality. There was no significant difference between pages from academic, hospital-affiliated, commercial, and nonprofit foundation publications. The text generated by ChatGPT was considerably easier to understand with readability levels measured as low as 5th grade.

CONCLUSIONS

The language used in existing information on kidney stone disease is of subpar quality and too complex to understand. Machine learning tools could aid in generating information that is comprehensible by the public.

A Retrospective Comparison of Medication Recommendations Between a Cardiologist and ChatGPT-4 for Hypertension Patients in a Rural Clinic

Background With ChatGPT demonstrating impressive abilities in solving clinical vignettes and medical questions, there is still a lack of studies assessing ChatGPT using real patient data. With real-world cases offering added complexity, ChatGPT's utility in treatment using such data must be tested to better assess its accuracy and dependability. In this study, we compared a rural cardiologist's medication recommendations to that of GPT-4 for patients with lab review appointments. Methodology We reviewed the lab review appointments of 40 hypertension patients, noting their age, sex, medical conditions, medications and dosage, and current and past lab values. The cardiologist's medication recommendations (decreasing dose, increasing dose, stopping, or adding medications) from the most recent lab visit, if any, were recorded for each patient. Data collected from each patient was inputted into GPT-4 using a set prompt and the resulting medication recommendations from the model were recorded. Results Out of the 40 patients, 95% had conflicting overall recommendations between the physician and GPT-4, with only 10.2% of the specific medication recommendations matching between the two. Cohen's kappa coefficient was -0.0127, indicating no agreement between the cardiologist and GPT-4 for providing medication changes overall for a patient. Possible reasons for this discrepancy can be differing optimal lab value ranges, lack of holistic analysis by GPT-4, and a need for providing further supplementary information to the model. Conclusions The study findings showed a significant difference between the cardiologist's medication recommendations and that of ChatGPT-4. Future research should continue to test GPT-4 in clinical settings to validate its abilities in the real world where more intricacies and challenges exist.

New evidence-based practice: Artificial intelligence as a barrier breaker

The concept of evidence-based practice has persisted over several years and remains a cornerstone in clinical practice, representing the gold standard for optimal patient care. However, despite widespread recognition of its significance, practical application faces various challenges and barriers, including a lack of skills in interpreting studies, limited resources, time constraints, linguistic competencies, and more. Recently, we have witnessed the emergence of a groundbreaking technological revolution known as artificial intelligence. Although artificial intelligence has become increasingly integrated into our daily lives, some reluctance persists among certain segments of the public. This article explores the potential of artificial intelligence as a solution to some of the main barriers encountered in the application of evidence-based practice. It highlights how artificial intelligence can assist in staying updated with the latest evidence, enhancing clinical decision-making, addressing patient misinformation, and mitigating time constraints in clinical practice. The integration of artificial intelligence into evidence-based practice has the potential to revolutionize healthcare, leading to more precise diagnoses, personalized treatment plans, and improved doctor-patient interactions. This proposed synergy between evidence-based practice and artificial intelligence may necessitate adjustments to its core concept, heralding a new era in healthcare.

Universal skepticism of ChatGPT: a review of early literature on chat generative pre-trained transformer

ChatGPT, a new language model developed by OpenAI, has garnered significant attention in various fields since its release. This literature review provides an overview of early ChatGPT literature across multiple disciplines, exploring its applications, limitations, and ethical considerations. The review encompasses Scopus-indexed publications from November 2022 to April 2023 and includes 156 articles related to ChatGPT. The findings reveal a predominance of negative sentiment across disciplines, though subject-specific attitudes must be considered. The review highlights the implications of ChatGPT in many fields including healthcare, raising concerns about employment opportunities and ethical considerations. While ChatGPT holds promise for improved communication, further research is needed to address its capabilities and limitations. This literature review provides insights into early research on ChatGPT, informing future investigations and practical applications of chatbot technology, as well as development and usage of generative AI.

The emergent role of artificial intelligence, natural learning processing, and large language models in higher education and research

Artificial Intelligence (AI) has revolutionized various domains, including education and research. Natural language processing (NLP) techniques and large language models (LLMs) such as GPT-4 and BARD have significantly advanced our comprehension and application of AI in these fields. This paper provides an in-depth introduction to AI, NLP, and LLMs, discussing their potential impact on education and research. By exploring the advantages, challenges, and innovative applications of these technologies, this review gives educators, researchers, students, and readers a comprehensive view of how AI could shape educational and research practices in the future, ultimately leading to improved outcomes. Key applications discussed in the field of research include text generation, data analysis and interpretation, literature review, formatting and editing, and peer review. AI applications in academics and education include educational support and constructive feedback, assessment, grading, tailored curricula, personalized career guidance, and mental health support. Addressing the challenges associated with these technologies, such as ethical concerns and algorithmic biases, is essential for maximizing their potential to improve education and research outcomes. Ultimately, the paper aims to contribute to the ongoing discussion about the role of AI in education and research and highlight its potential to lead to better outcomes for students, educators, and researchers.

Genital and Extragenital Lichen Sclerosus et Atrophicus: A Case Series Written Using ChatGPT

Background Lichen sclerosus et atrophicus (LSEA) is a chronic inflammatory dermatosis of genital and extragenital sites with a prevalence ranging from 9% in prepubertal patients to 50% in postmenopausal patients. Chat generative pre-trained transformer (ChatGPT) is an artificial intelligence tool designed to assist humans based on supervised and reinforcement techniques. In this study, we aimed to evaluate the characteristics of patients with LSEA using ChatGPT. Methods In this retrospective study, we included all patients who presented to the outpatient dermatology department during 2017-2022 at a tertiary care teaching hospital in South India. Information regarding demographic data, characteristics of LSEA, comorbidities, and associated autoimmune disorders was gathered using a medical chart review. Following data analysis and drafting of the manuscript, the utility of ChatGPT-3 and ChatGPT-4 in finalizing the draft was assessed. Results Of 20 patients diagnosed with LSEA, 16 (80%) and four (20%) patients were females and males, respectively. Of them, 50% of female patients had attained menopause. While 65% of patients had genital LSEA, 30% of patients had extragenital LSEA only, and 5% of patients had both genital and extragenital LSEA. Furthermore, four (20%) patients were prepubertal children. Of four male patients, two (50%) were younger than 18 years of age, and one patient was diagnosed with balanitis xerotica obliterans. The commonest associated features in LSEA included joint involvement (30%), hypertension (25%), and anemia (15%). Rare concomitant disorders included psoriasis, asthma, and basal cell carcinoma over the nose. Conclusions LSEA may be confused with other various dermatoses, such as morphea, vitiligo, and lichen planus. A high index of suspicion is required, especially in children, to diagnose it early and intervene to prevent further complications. Its relationship with autoimmune disorders and comorbidities warrants further large-scale studies. ChatGPT was unreliable in the literature search due to the provision of non-existent citations. ChatGPT-4 was better than ChatGPT-3 since it provided few true publications. ChatGPT was used in this study to summarize the articles identified by the authors during the literature search and to correct grammatical errors in the final draft of the manuscript.

Practical Applications of ChatGPT in Undergraduate Medical Education

ChatGPT is a chatbot developed by OpenAI that has garnered significant attention for achieving at or near a passing standard on the United States Medical Licensing Exam (USMLE). Currently, researchers and users are exploring ChatGPT's broad range of potential applications in academia, business, programming, and beyond. We attempt outline how ChatGPT may be applied to support undergraduate medical education during the preclinical and clinical years, and highlight possible concerns regarding its use which necessitates the creation of formal policies and training by medical schools.