Generative pretrained transformer-4, an artificial intelligence text predictive model, has a high capability for passing novel written radiology exam questions

PURPOSE

AI-image interpretation, through convolutional neural networks, shows increasing capability within radiology. These models have achieved impressive performance in specific tasks within controlled settings, but possess inherent limitations, such as the inability to consider clinical context. We assess the ability of large language models (LLMs) within the context of radiology specialty exams to determine whether they can evaluate relevant clinical information.

METHODS

A database of questions was created with official sample, author written, and textbook questions based on the Royal College of Radiology (United Kingdom) FRCR 2A and American Board of Radiology (ABR) Certifying examinations. The questions were input into the Generative Pretrained Transformer (GPT) versions 3 and 4, with prompting to answer the questions.

RESULTS

One thousand seventy-two questions were evaluated by GPT-3 and GPT-4. 495 (46.2%) were for the FRCR 2A and 577 (53.8%) were for the ABR exam. There were 890 single best answers (SBA), and 182 true/false questions. GPT-4 was correct in 629/890 (70.7%) SBA and 151/182 (83.0%) true/false questions. There was no degradation on author written questions. GPT-4 performed significantly better than GPT-3 which selected the correct answer in 282/890 (31.7%) SBA and 111/182 (61.0%) true/false questions. Performance of GPT-4 was similar across both examinations for all categories of question.

CONCLUSION

The newest generation of LLMs, GPT-4, demonstrates high capability in answering radiology exam questions. It shows marked improvement from GPT-3, suggesting further improvements in accuracy are possible. Further research is needed to explore the clinical applicability of these AI models in real-world settings.

Educational treasures in radiology: a free online program for Radiology Boards preparation

PURPOSE

An objective tool is desired, which optimally prepares for Radiology boards examination. Such program should prepare examinees with pertinent radiological contents and simulations as expected in the real examination.

BACKGROUND

Many countries require written boards examinations for Radiology certification eligibility. No objective measure exists to tell if the examinee is ready to pass the exam or not. Time pressure and computer environment might be unfamiliar to examinees. Traditional preparation lectures don't simulate the "real" Radiology exam because they don't provide the special environment with multiple choice questions and timing.

MATERIALS AND METHODS

This online program consists of 4 parts. The entry section allows to create questions with additional fields for comprehensive information. Sections include Pediatrics/Mammography/GI/IR/Nucs/Thoracic/Musculoskeletal/GU/Neuro/Ultrasound/Cardiac/OB/GYN and Miscellaneous. Experienced radiologists and educators evaluate and release/delete these entries in the administrator section. In the exam section users can create (un)timed customized exams for individual needs and learning pace. Exams can either include all sections or only specific sections to gear learning towards areas with weaker performance. Comprehensive statistics unveil the user's strengths and weaknesses to help focussing on "weak" areas. In the search section a comprehensive search and review can be performed by searching the entire database for keywords/topics or only searching within specific sections.

CONCLUSION

www.RadiologyBoards.org is a new working concept of Radiology boards preparation to detect and improve the examinee's weaknesses and finally to increase the examinee's confidence level for the final exam. It is beneficial for Radiology residents and also board certified radiologists to refresh/maintain radiological knowledge.