Evaluation of the performance of GPT-3.5 and GPT-4 on the Polish Medical Final Examination

The study aimed to evaluate the performance of two Large Language Models (LLMs): ChatGPT (based on GPT-3.5) and GPT-4 with two temperature parameter values, on the Polish Medical Final Examination (MFE). The models were tested on three editions of the MFE from: Spring 2022, Autumn 2022, and Spring 2023 in two language versions-English and Polish. The accuracies of both models were compared and the relationships between the correctness of answers with the answer's metrics were investigated. The study demonstrated that GPT-4 outperformed GPT-3.5 in all three examinations regardless of the language used. GPT-4 achieved mean accuracies of 79.7% for both Polish and English versions, passing all MFE versions. GPT-3.5 had mean accuracies of 54.8% for Polish and 60.3% for English, passing none and 2 of 3 Polish versions for temperature parameter equal to 0 and 1 respectively while passing all English versions regardless of the temperature parameter value. GPT-4 score was mostly lower than the average score of a medical student. There was a statistically significant correlation between the correctness of the answers and the index of difficulty for both models. The overall accuracy of both models was still suboptimal and worse than the average for medical students. This emphasizes the need for further improvements in LLMs before they can be reliably deployed in medical settings. These findings suggest an increasing potential for the usage of LLMs in terms of medical education.

Supine sleep patterns as a part of phenotyping patients with sleep apnea-a pilot study

Purpose

Polysomnography (PSG) is considered the best objective study to diagnose and quantify sleep disorders. However, PSG involves multiple electrodes and is usually performed in a sleep laboratory that in itself may change the physiology of sleep. One of the parameters that can change during PSG is the sleep position, leading to more supine sleep. The aim of this study was to quantify the amount of supine sleep during PSG and compare it to consecutive nights of a home sleep apnea test (HSAT) in the same patients.

Methods

This prospective study evaluated 22 consecutive patients undergoing PSG followed by HSAT. Sleep position was analyzed during PSG and subsequently on 2 to 6 nights (mean 3.7 nights) at home, and the amount of supine sleep was recorded during each night.

Results

Of 22 patients, there were 12 men (55%). The median age was 60.0 years for women and 45.5 years for men. Median proportion of supine sleep during PSG and HSAT was 61% and 26% (p < 0.001), respectively. Four “phenotypes” were identified according to their sleep position during PSG and HSAT, with 5 patients sleeping mainly supine during all nights, 7 patients sleeping mainly non-supine during all nights, 3 patients sleeping in different positions during each night, and 7 patients sleeping supine during PSG but non-supine at home, during HSAT.

Conclusions

There is a higher proportion of supine sleep during PSG compared to home sleep. We identified a subgroup of patients who slept mainly supine during PSG and mainly non-supine during HSAT. PSG may overestimate OSA severity in a specific phenotype of patients.