Accuracy of GPT-4 in histopathological image detection and classification of colorectal adenomas

Exploring the Potential of Code-Free Custom GPTs in Ophthalmology: An Early Analysis of GPT Store and User-Creator Guidance

INTRODUCTION

OpenAI recently introduced the ability to create custom generative pre-trained transformers (cGPTs) using text-based instruction and/or external documents using retrieval-augmented generation (RAG) architecture without coding knowledge. This study aimed to analyze the features of ophthalmology-related cGPTs and explore their potential utilities.

METHODS

Data collection took place on January 20 and 21, 2024, and custom GPTs were found by entering ophthalmology keywords into the "Explore GPTS" section of the website. General and specific features of cGPTs were recorded, such as knowledge other than GPT-4 training data. The instruction and description sections were analyzed for compatibility using the Likert scale. We analyzed two custom GPTs with the highest Likert score in detail. We attempted to create a convincingly presented yet potentially harmful cGPT to test safety features.

RESULTS

We analyzed 22 ophthalmic cGPTs, of which 55% were for general use and the most common subspecialty was glaucoma (18%). Over half (55%) contained knowledge other than GPT-4 training data. The representation of the instructions through the description was between "Moderately representative" and "Very representative" with a median Likert score of 3.5 (IQR 3.0-4.0). The instruction word count was significantly associated with Likert scores (P = 0.03). Tested cGPTs demonstrated potential for specific conversational tone, information, retrieval and combining knowledge from an uploaded source. With these safety settings, creating a malicious GPT was possible.

CONCLUSIONS

This is the first study to our knowledge to examine the GPT store for a medical field. Our findings suggest that these cGPTs can be immediately implemented in practice and may offer more targeted and effective solutions compared to the standard GPT-4. However, further research is necessary to evaluate their capabilities and limitations comprehensively. The safety features currently appear to be rather limited. It may be helpful for the user to review the instruction section before using a cGPT.

A survey analysis of the adoption of large language models among pathologists

OBJECTIVES

We sought to investigate the adoption and perception of large language model (LLM) applications among pathologists.

METHODS

A cross-sectional survey was conducted, gathering data from pathologists on their usage and views concerning LLM tools. The survey, distributed globally through various digital platforms, included quantitative and qualitative questions. Patterns in the respondents' adoption and perspectives on these artificial intelligence tools were analyzed.

RESULTS

Of 215 respondents, 100 (46.5%) reported using LLMs, particularly ChatGPT (OpenAI), for professional purposes, predominantly for information retrieval, proofreading, academic writing, and drafting pathology reports, highlighting a significant time-saving benefit. Academic pathologists demonstrated a better level of understanding of LLMs than their peers. Although chatbots sometimes provided incorrect general domain information, they were considered moderately proficient concerning pathology-specific knowledge. The technology was mainly used for drafting educational materials and programming tasks. The most sought-after feature in LLMs was their image analysis capabilities. Participants expressed concerns about information accuracy, privacy, and the need for regulatory approval.

CONCLUSIONS

Large language model applications are gaining notable acceptance among pathologists, with nearly half of respondents indicating adoption less than a year after the tools' introduction to the market. They see the benefits but are also worried about these tools' reliability, ethical implications, and security.

ChatGPT-4 Consistency in Interpreting Laryngeal Clinical Images of Common Lesions and Disorders

OBJECTIVE

To investigate the consistency of Chatbot Generative Pretrained Transformer (ChatGPT)-4 in the analysis of clinical pictures of common laryngological conditions.

STUDY DESIGN

Prospective uncontrolled study.

SETTING

Multicenter study.

METHODS

Patient history and clinical videolaryngostroboscopic images were presented to ChatGPT-4 for differential diagnoses, management, and treatment(s). ChatGPT-4 responses were assessed by 3 blinded laryngologists with the artificial intelligence performance instrument (AIPI). The complexity of cases and the consistency between practitioners and ChatGPT-4 for interpreting clinical images were evaluated with a 5-point Likert Scale. The intraclass correlation coefficient (ICC) was used to measure the strength of interrater agreement.

RESULTS

Forty patients with a mean complexity score of 2.60 ± 1.15. were included. The mean consistency score for ChatGPT-4 image interpretation was 2.46 ± 1.42. ChatGPT-4 perfectly analyzed the clinical images in 6 cases (15%; 5/5), while the consistency between GPT-4 and judges was high in 5 cases (12.5%; 4/5). Judges reported an ICC of 0.965 for the consistency score (P = .001). ChatGPT-4 erroneously documented vocal fold irregularity (mass or lesion), glottic insufficiency, and vocal cord paralysis in 21 (52.5%), 2 (0.05%), and 5 (12.5%) cases, respectively. ChatGPT-4 and practitioners indicated 153 and 63 additional examinations, respectively (P = .001). The ChatGPT-4 primary diagnosis was correct in 20.0% to 25.0% of cases. The clinical image consistency score was significantly associated with the AIPI score (rs = 0.830; P = .001).

CONCLUSION

The ChatGPT-4 is more efficient in primary diagnosis, rather than in the image analysis, selecting the most adequate additional examinations and treatments.

Comparing customized ChatGPT and pathology residents in histopathologic description and diagnosis of common diseases

This study aimed to evaluate and analyze the performance of a customized Chat Generative Pre-Trained Transformer (ChatGPT), known as GPT, against pathology residents in providing microscopic descriptions and diagnosing diseases from histopathological images. A dataset of representative photomicrographs from 70 diseases across 14 organ systems was analyzed by a customized version of ChatGPT-4 (GPT-4) and pathology residents. Two pathologists independently evaluated the microscopic descriptions and diagnoses using a predefined scoring system (0-4 for microscopic descriptions and 0-2 for pathological diagnoses), with higher scores indicating greater accuracy. Microscopic descriptions that received perfect scores, which included all relevant keywords and findings, were then presented to the standard version of ChatGPT to assess its diagnostic capabilities based on these descriptions. GPT-4 showed consistency in microscopic description and diagnosis scores across five rounds, accomplishing median scores of 50 % and 48.6 %, respectively. However, its performance was still inferior to junior and senior pathology residents (73.9 % and 93.9 % description scores and 63.9 % and 87.9 % diagnosis scores, respectively). When analyzing classic ChatGPT's understanding of microscopic descriptions provided by residents, it correctly diagnosed 35 (87.5 %) of cases from junior residents and 44 (68.8 %) from senior residents, given that the initial descriptions consisted of keywords and relevant findings. While GPT-4 can accurately interpret some histopathological images, its overall performance is currently inferior to that of pathology residents. However, ChatGPT's ability to accurately interpret and diagnose diseases from the descriptions provided by residents suggests that this technology could serve as a valuable support tool in pathology diagnostics.

Applications of Large Language Models in Pathology

Large language models (LLMs) are transformer-based neural networks that can provide human-like responses to questions and instructions. LLMs can generate educational material, summarize text, extract structured data from free text, create reports, write programs, and potentially assist in case sign-out. LLMs combined with vision models can assist in interpreting histopathology images. LLMs have immense potential in transforming pathology practice and education, but these models are not infallible, so any artificial intelligence generated content must be verified with reputable sources. Caution must be exercised on how these models are integrated into clinical practice, as these models can produce hallucinations and incorrect results, and an over-reliance on artificial intelligence may lead to de-skilling and automation bias. This review paper provides a brief history of LLMs and highlights several use cases for LLMs in the field of pathology.