Popular large language model chatbots' accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries

In light of growing interest in using emerging large language models (LLMs) for self-diagnosis, we systematically assessed the performance of ChatGPT-3.5, ChatGPT-4.0, and Google Bard in delivering proficient responses to 37 common inquiries regarding ocular symptoms. Responses were masked, randomly shuffled, and then graded by three consultant-level ophthalmologists for accuracy (poor, borderline, good) and comprehensiveness. Additionally, we evaluated the self-awareness capabilities (ability to self-check and self-correct) of the LLM-Chatbots. 89.2% of ChatGPT-4.0 responses were 'good'-rated, outperforming ChatGPT-3.5 (59.5%) and Google Bard (40.5%) significantly (all p < 0.001). All three LLM-Chatbots showed optimal mean comprehensiveness scores as well (ranging from 4.6 to 4.7 out of 5). However, they exhibited subpar to moderate self-awareness capabilities. Our study underscores the potential of ChatGPT-4.0 in delivering accurate and comprehensive responses to ocular symptom inquiries. Future rigorous validation of their performance is crucial to ensure their reliability and appropriateness for actual clinical use.

Minimally Invasive Glaucoma Surgery: Latest Developments and Future Challenges

The development of minimally invasive glaucoma surgeries (MIGSs) was intended to provide safe and modestly efficacious modalities for early intervention of mild-to-moderate glaucoma, with minimal trauma and rapid recovery. They were mainly ab interno procedures that reduce intraocular pressure by facilitating the aqueous outflow by bypassing the trabecular meshwork resistance, reinforcing the uveoscleral flow via the supraciliary space, and reducing aqueous production by the ciliary body. While the cumulating evidence helps shape the role of the available MIGS, the exponential new development and advancement in this field has expanded the territory of MIGS. Apart from developing subconjunctival MIGS filtration devices (Xen gel stent and PRESERFLO MicroShunt), there is a tendency to revisit the "traditional" MIGS for alternative use and to modify the procedures with consideration of the fundamental aqueous outflow physiology. Combined MIGS has also been suggested, based on the theory that their different mechanisms may provide additive or synergistic effects. The advancement of laser procedures is also promising and could supplement unmet needs along the glaucoma treatment algorithm. This review examines the broad array of MIGS, updates the recent findings, discusses their potential alternative applications, and explores future challenges.

Axial length elongation profiles from 3 to 6 years in an Asian paediatric population: the Growing Up in Singapore Towards Healthy Outcomes birth cohort study (GUSTO)

AIMS

To determine axial length (AL) elongation profiles in children aged 3-6 years in an Asian population.

METHODS

Eligible subjects were recruited from the Growing Up in Singapore Towards Healthy Outcomes birth cohort. AL measurement was performed using IOLMaster (Carl Zeiss Meditec, Jena, Germany) at 3 and 6 years. Anthropometric measurements at birth, cycloplegic refraction at 3 and 6 years, questionnaires on the children's behavioural habits at 2 years and parental spherical equivalent refraction were performed. Multivariable linear regression model with generalised estimating equation was performed to determine factors associated with AL elongation.

RESULTS

273 eyes of 194 children were included. The mean AL increased from 21.72±0.59 mm at 3 years to 22.52±0.66 mm at 6 years (p<0.001). Myopic eyes at 6 years had greater AL elongation (1.02±0.34 mm) compared with emmetropic eyes (0.85±0.25 mm, p=0.008) and hyperopic eyes (0.74±0.16 mm, p<0.001). The 95th percentile limit of AL elongation was 1.59 mm in myopes, 1.34 mm in emmetropes and 1.00 mm in hyperopes. Greater birth weight (per 100 g, β=0.010, p=0.02) was significantly associated with greater AL elongation from 3 to 6 years, while parental and other behavioural factors assessed at 2 years were not (all p≥0.08).

CONCLUSION

In this preschool cohort, AL elongates at an average length of 0.80 mm from 3 to 6 years, with myopes demonstrating the greatest elongation. The differences in 95th percentile limits for AL elongation between myopes, emmetropes and hyperopes can be valuable information in identifying myopia development in preschool children.

Benchmarking large language models' performances for myopia care: a comparative analysis of ChatGPT-3.5, ChatGPT-4.0, and Google Bard

BACKGROUND

Large language models (LLMs) are garnering wide interest due to their human-like and contextually relevant responses. However, LLMs' accuracy across specific medical domains has yet been thoroughly evaluated. Myopia is a frequent topic which patients and parents commonly seek information online. Our study evaluated the performance of three LLMs namely ChatGPT-3.5, ChatGPT-4.0, and Google Bard, in delivering accurate responses to common myopia-related queries.

METHODS

We curated thirty-one commonly asked myopia care-related questions, which were categorised into six domains-pathogenesis, risk factors, clinical presentation, diagnosis, treatment and prevention, and prognosis. Each question was posed to the LLMs, and their responses were independently graded by three consultant-level paediatric ophthalmologists on a three-point accuracy scale (poor, borderline, good). A majority consensus approach was used to determine the final rating for each response. 'Good' rated responses were further evaluated for comprehensiveness on a five-point scale. Conversely, 'poor' rated responses were further prompted for self-correction and then re-evaluated for accuracy.

FINDINGS

ChatGPT-4.0 demonstrated superior accuracy, with 80.6% of responses rated as 'good', compared to 61.3% in ChatGPT-3.5 and 54.8% in Google Bard (Pearson's chi-squared test, all p ≤ 0.009). All three LLM-Chatbots showed high mean comprehensiveness scores (Google Bard: 4.35; ChatGPT-4.0: 4.23; ChatGPT-3.5: 4.11, out of a maximum score of 5). All LLM-Chatbots also demonstrated substantial self-correction capabilities: 66.7% (2 in 3) of ChatGPT-4.0's, 40% (2 in 5) of ChatGPT-3.5's, and 60% (3 in 5) of Google Bard's responses improved after self-correction. The LLM-Chatbots performed consistently across domains, except for 'treatment and prevention'. However, ChatGPT-4.0 still performed superiorly in this domain, receiving 70% 'good' ratings, compared to 40% in ChatGPT-3.5 and 45% in Google Bard (Pearson's chi-squared test, all p ≤ 0.001).

INTERPRETATION

Our findings underscore the potential of LLMs, particularly ChatGPT-4.0, for delivering accurate and comprehensive responses to myopia-related queries. Continuous strategies and evaluations to improve LLMs' accuracy remain crucial.

FUNDING

Dr Yih-Chung Tham was supported by the National Medical Research Council of Singapore (NMRC/MOH/HCSAINV21nov-0001).

Variability in Grading Diabetic Retinopathy Using Retinal Photography and Its Comparison with an Automated Deep Learning Diabetic Retinopathy Screening Software

BACKGROUND

Diabetic retinopathy (DR) screening using colour retinal photographs is cost-effective and time-efficient. In real-world clinical settings, DR severity is frequently graded by individuals of different expertise levels. We aim to determine the agreement in DR severity grading between human graders of varying expertise and an automated deep learning DR screening software (ADLS).

METHODS

Using the International Clinical DR Disease Severity Scale, two hundred macula-centred fundus photographs were graded by retinal specialists, ophthalmology residents, family medicine physicians, medical students, and the ADLS. Based on referral urgency, referral grading was divided into no referral, non-urgent referral, and urgent referral to an ophthalmologist. Inter-observer and intra-group variations were analysed using Gwet's agreement coefficient, and the performance of ADLS was evaluated using sensitivity and specificity.

RESULTS

The agreement coefficient for inter-observer and intra-group variability ranged from fair to very good, and moderate to good, respectively. The ADLS showed a high area under curve of 0.879, 0.714, and 0.836 for non-referable DR, non-urgent referable DR, and urgent referable DR, respectively, with varying sensitivity and specificity values.

CONCLUSION

Inter-observer and intra-group agreements among human graders vary widely, but ADLS is a reliable and reasonably sensitive tool for mass screening to detect referable DR and urgent referable DR.

Applications of Extended Reality in Ophthalmology: Systematic Review

BACKGROUND

Virtual reality, augmented reality, and mixed reality make use of a variety of different software and hardware, but they share three main characteristics: immersion, presence, and interaction. The umbrella term for technologies with these characteristics is extended reality. The ability of extended reality to create environments that are otherwise impossible in the real world has practical implications in the medical discipline. In ophthalmology, virtual reality simulators have become increasingly popular as tools for surgical education. Recent developments have also explored diagnostic and therapeutic uses in ophthalmology.

OBJECTIVE

This systematic review aims to identify and investigate the utility of extended reality in ophthalmic education, diagnostics, and therapeutics.

METHODS

A literature search was conducted using PubMed, Embase, and Cochrane Register of Controlled Trials. Publications from January 1, 1956 to April 15, 2020 were included. Inclusion criteria were studies evaluating the use of extended reality in ophthalmic education, diagnostics, and therapeutics. Eligible studies were evaluated using the Oxford Centre for Evidence-Based Medicine levels of evidence. Relevant studies were also evaluated using a validity framework. Findings and relevant data from the studies were extracted, evaluated, and compared to determine the utility of extended reality in ophthalmology.

RESULTS

We identified 12,490 unique records in our literature search; 87 met final eligibility criteria, comprising studies that evaluated the use of extended reality in education (n=54), diagnostics (n=5), and therapeutics (n=28). Of these, 79 studies (91%) achieved evidence levels in the range 2b to 4, indicating poor quality. Only 2 (9%) out of 22 relevant studies addressed all 5 sources of validity evidence. In education, we found that ophthalmic surgical simulators demonstrated efficacy and validity in improving surgical performance and reducing complication rates. Ophthalmoscopy simulators demonstrated efficacy and validity evidence in improving ophthalmoscopy skills in the clinical setting. In diagnostics, studies demonstrated proof-of-concept in presenting ocular imaging data on extended reality platforms and validity in assessing the function of patients with ophthalmic diseases. In therapeutics, heads-up surgical systems had similar complication rates, procedural success rates, and outcomes in comparison with conventional ophthalmic surgery.

CONCLUSIONS

Extended reality has promising areas of application in ophthalmology, but additional high-quality comparative studies are needed to assess their roles among incumbent methods of ophthalmic education, diagnostics, and therapeutics.

Periorbital muscle atrophy associated with topical bimatoprost therapy

Topical Bimatoprost is a common and popular prostaglandin analog used as an ocular hypotensive agent in the treatment of glaucoma. Side effects include ocular hyperaemia, ocular pruritus, and periocular and iris pigmentary changes. Perioribital lipodystrophy is another well-documented outcome associated with chronic use of topical bimatoprost, which results in periorbital hallowing, upper eyelid sulcus deepening, eyelid retraction and enophthalmos. We report an unusual case of periocular muscle atrophy and weakness from unilateral topical bimatoprost use. Our patient had primary angle closure and experienced a right upper eyelid ptosis 2 months after she started to use topical bimatoprost in that eye. Clinical measurements of her eyelids clearly showed reduction in the function of her right levator muscle, suggesting that effects of topical bimatoprost may not be limited to periorbital fat. She was advised to stop topical bimatoprost and right ptosis correction surgery with levator muscle advancement was performed successfully. Ophthalmologists and patients should be aware of this potential rare side effect of topical bimatoprost, as it may be potentially disfiguring, especially with monocular use. However, its exact mechanism of action needs to be clarified further.

Survey on the management of orbital and intraocular tumors among oculofacial surgeons in the Asia-Pacific region

A web-based anonymous survey was performed to assess the common practices of oculofacial surgeons in the management of orbital and intraocular tumors in the Asia-Pacific region. The questionnaire comprised a series of questions covering controversial topics sent via email to 131 oculofacial surgeons across 14 countries in the Asia-Pacific region. A total response rate of 61.7 % was achieved from May to December 2012. The most common benign orbital tumor was cavernous hemangioma (39.6 %) and the most common malignant orbital tumor was lymphoma (85.7 %). 40 % of surgeons recommended orbital radiation, for which the most common indications were thyroid eye disease (70.0 %) and malignancy (30.0 %). The most common orbitotomy approach was lateral (79.2 %). Most surgeons (87.1 %) offered enucleation for retinoblastoma, but there was also a significant proportion that offered chemoreduction with transpupillary thermotherapy or cryotherapy (58.1 %). Fewer surgeons offered brachytherapy (16.1 %) and intra-arterial chemotherapy (6.5 %). When performing enucleation for retinoblastoma, 81.8 % of surgeons performed a primary orbital implant placement. The most preferred type of implant was silicone/acrylic (90.2, 90.2 and 87.8 % for elderly, adults and children, respectively). The majority of surgeons used donor sclera (57.5 %) or no wrapping material at all (32.5 %). Almost all surgeons (95.1 %) did not drill and peg the implant for motility. We report the results of the first survey of oculofacial surgeons in the Asia-Pacific region on the management of intraocular and orbital tumors. In comparison with previous surveys performed in the USA and the UK, we found the practice patterns of the Asia-Pacific surgeons to be comparable.