Artificial intelligence for teleophthalmology-based diabetic retinopathy screening in a national programme: an economic analysis modelling study

Comparison of 21 artificial intelligence algorithms in automated diabetic retinopathy screening using handheld fundus camera

BACKGROUND

Diabetic retinopathy (DR) is a common complication of diabetes and may lead to irreversible visual loss. Efficient screening and improved treatment of both diabetes and DR have amended visual prognosis for DR. The number of patients with diabetes is increasing and telemedicine, mobile handheld devices and automated solutions may alleviate the burden for healthcare. We compared the performance of 21 artificial intelligence (AI) algorithms for referable DR screening in datasets taken by handheld Optomed Aurora fundus camera in a real-world setting.

PATIENTS AND METHODS

Prospective study of 156 patients (312 eyes) attending DR screening and follow-up. Both papilla- and macula-centred 50° fundus images were taken from each eye. DR was graded by experienced ophthalmologists and 21 AI algorithms.

RESULTS

Most eyes, 183 out of 312 (58.7%), had no DR and mild NPDR was noted in 21 (6.7%) of the eyes. Moderate NPDR was detected in 66 (21.2%) of the eyes, severe NPDR in 1 (0.3%), and PDR in 41 (13.1%) composing a group of 34.6% of eyes with referable DR. The AI algorithms achieved a mean agreement of 79.4% for referable DR, but the results varied from 49.4% to 92.3%. The mean sensitivity for referable DR was 77.5% (95% CI 69.1-85.8) and specificity 80.6% (95% CI 72.1-89.2). The rate for images ungradable by AI varied from 0% to 28.2% (mean 1.9%). Nineteen out of 21 (90.5%) AI algorithms resulted in grading for DR at least in 98% of the images.

CONCLUSIONS

Fundus images captured with Optomed Aurora were suitable for DR screening. The performance of the AI algorithms varied considerably emphasizing the need for external validation of screening algorithms in real-world settings before their clinical application.

The use of artificial intelligence to assess diabetic eye disease among the Greenlandic population

Background: Retina fundus images conducted in Greenland are telemedically assessed for diabetic retinopathy by ophthalmological nurses in Denmark. Applying an AI grading solution, in a Greenlandic setting, could potentially improve the efficiency and cost-effectiveness of DR screening.Method: We developed an AI model using retina fundus photos, performed on persons registered with diabetes in Greenland and Denmark, using Optos® ultra wide-field scanning laser ophthalmoscope, graded according to ICDR.Using the ResNet50 network we compared the model's ability to distinguish between different images of ICDR severity levels in a confusion matrix.Results: Comparing images with ICDR level 0 to images of ICDR level 4 resulted in an accuracy of 0.9655, AUC of 0.9905, sensitivity and specificity of 96.6%.Comparing ICDR levels 0,1,2 with ICDR levels 3,4, we achieved a performance with an accuracy of 0.8077, an AUC of 0.8728, a sensitivity of 84.6% and a specificity of 78.8%. For the other comparisons, we achieved a modest performance.Conclusion: We developed an AI model using Greenlandic data, to automatically detect DR on Optos retina fundus images. The sensitivity and specificity were too low for our model to be applied directly in a clinical setting, thus optimising the model should be prioritised.

A systematic review of economic evaluation of artificial intelligence-based screening for eye diseases: From possibility to reality

Artificial Intelligence (AI) has become a focus of research in the rapidly evolving field of ophthalmology. Nevertheless, there is a lack of systematic studies on the health economics of AI in this field. We examine studies from the PubMed, Google Scholar, and Web of Science databases that employed quantitative analysis, retrieved up to July 2023. Most of the studies indicate that AI leads to cost savings and improved efficiency in ophthalmology. On the other hand, some studies suggest that using AI in healthcare may raise costs for patients, especially when taking into account factors such as labor costs, infrastructure, and patient adherence. Future research should cover a wider range of ophthalmic diseases beyond common eye conditions. Moreover, conducting extensive health economic research, designed to collect data relevant to its own context, is imperative.

Automated Identification of Different Severity Levels of Diabetic Retinopathy Using a Handheld Fundus Camera and Single-Image Protocol

Purpose

To evaluate the performance of artificial intelligence (AI) systems embedded in a mobile, handheld retinal camera, with a single retinal image protocol, in detecting both diabetic retinopathy (DR) and more-than-mild diabetic retinopathy (mtmDR).

Design

Multicenter cross-sectional diagnostic study, conducted at 3 diabetes care and eye care facilities.

Participants

A total of 327 individuals with diabetes mellitus (type 1 or type 2) underwent a retinal imaging protocol enabling expert reading and automated analysis.

Methods

Participants underwent fundus photographs using a portable retinal camera (Phelcom Eyer). The captured images were automatically analyzed by deep learning algorithms retinal alteration score (RAS) and diabetic retinopathy alteration score (DRAS), consisting of convolutional neural networks trained on EyePACS data sets and fine-tuned using data sets of portable device fundus images. The ground truth was the classification of DR corresponding to adjudicated expert reading, performed by 3 certified ophthalmologists.

Main Outcome Measures

Primary outcome measures included the sensitivity and specificity of the AI system in detecting DR and/or mtmDR using a single-field, macula-centered fundus photograph for each eye, compared with a rigorous clinical reference standard comprising the reading center grading of 2-field imaging protocol using the International Classification of Diabetic Retinopathy severity scale.

Results

Of 327 analyzed patients (mean age, 57.0 ± 16.8 years; mean diabetes duration, 16.3 ± 9.7 years), 307 completed the study protocol. Sensitivity and specificity of the AI system were high in detecting any DR with DRAS (sensitivity, 90.48% [95% confidence interval (CI), 84.99%-94.46%]; specificity, 90.65% [95% CI, 84.54%-94.93%]) and mtmDR with the combination of RAS and DRAS (sensitivity, 90.23% [95% CI, 83.87%-94.69%]; specificity, 85.06% [95% CI, 78.88%-90.00%]). The area under the receiver operating characteristic curve was 0.95 for any DR and 0.89 for mtmDR.

Conclusions

This study showed a high accuracy for the detection of DR in different levels of severity with a single retinal photo per eye in an all-in-one solution, composed of a portable retinal camera powered by AI. Such a strategy holds great potential for increasing coverage rates of screening programs, contributing to prevention of avoidable blindness.

Financial Disclosures

F.K.M. is a medical consultant for Phelcom Technologies. J.A.S. is Chief Executive Officer and proprietary of Phelcom Technologies. D.L. is Chief Technology Officer and proprietary of Phelcom Technologies. P.V.P. is an employee at Phelcom Technologies.

Advances in Structural and Functional Retinal Imaging and Biomarkers for Early Detection of Diabetic Retinopathy

Diabetic retinopathy (DR), a vision-threatening microvascular complication of diabetes mellitus (DM), is a leading cause of blindness worldwide that requires early detection and intervention. However, diagnosing DR early remains challenging due to the subtle nature of initial pathological changes. This review explores developments in multimodal imaging and functional tests for early DR detection. Where conventional color fundus photography is limited in the field of view and resolution, advanced quantitative analysis of retinal vessel traits such as retinal microvascular caliber, tortuosity, and fractal dimension (FD) can provide additional prognostic value. Optical coherence tomography (OCT) has also emerged as a reliable structural imaging tool for assessing retinal and choroidal neurodegenerative changes, which show potential as early DR biomarkers. Optical coherence tomography angiography (OCTA) enables the evaluation of vascular perfusion and the contours of the foveal avascular zone (FAZ), providing valuable insights into early retinal and choroidal vascular changes. Functional tests, including multifocal electroretinography (mfERG), visual evoked potential (VEP), multifocal pupillographic objective perimetry (mfPOP), microperimetry, and contrast sensitivity (CS), offer complementary data on early functional deficits in DR. More importantly, combining structural and functional imaging data may facilitate earlier detection of DR and targeted management strategies based on disease progression. Artificial intelligence (AI) techniques show promise for automated lesion detection, risk stratification, and biomarker discovery from various imaging data. Additionally, hematological parameters, such as neutrophil-lymphocyte ratio (NLR) and neutrophil extracellular traps (NETs), may be useful in predicting DR risk and progression. Although current methods can detect early DR, there is still a need for further research and development of reliable, cost-effective methods for large-scale screening and monitoring of individuals with DM.

Novel artificial intelligence algorithms for diabetic retinopathy and diabetic macular edema

BACKGROUND

Diabetic retinopathy (DR) and diabetic macular edema (DME) are major causes of visual impairment that challenge global vision health. New strategies are needed to tackle these growing global health problems, and the integration of artificial intelligence (AI) into ophthalmology has the potential to revolutionize DR and DME management to meet these challenges.

MAIN TEXT

This review discusses the latest AI-driven methodologies in the context of DR and DME in terms of disease identification, patient-specific disease profiling, and short-term and long-term management. This includes current screening and diagnostic systems and their real-world implementation, lesion detection and analysis, disease progression prediction, and treatment response models. It also highlights the technical advancements that have been made in these areas. Despite these advancements, there are obstacles to the widespread adoption of these technologies in clinical settings, including regulatory and privacy concerns, the need for extensive validation, and integration with existing healthcare systems. We also explore the disparity between the potential of AI models and their actual effectiveness in real-world applications.

CONCLUSION

AI has the potential to revolutionize the management of DR and DME, offering more efficient and precise tools for healthcare professionals. However, overcoming challenges in deployment, regulatory compliance, and patient privacy is essential for these technologies to realize their full potential. Future research should aim to bridge the gap between technological innovation and clinical application, ensuring AI tools integrate seamlessly into healthcare workflows to enhance patient outcomes.

Advancing healthcare: the role and impact of AI and foundation models

BACKGROUND

The integration of artificial intelligence (AI) into the healthcare domain is a monumental shift with profound implications for diagnostics, medical interventions, and the overall structure of healthcare systems.

PURPOSE

This study explores the transformative journey of foundation AI models in healthcare, shedding light on the challenges, ethical considerations, and vast potential they hold for improving patient outcome and system efficiency. Notably, in this investigation we observe a relatively slow adoption of AI within the public sector of healthcare. The evolution of AI in healthcare is un-paralleled, especially its prowess in revolutionizing diagnostic processes.

RESULTS

This research showcases how these foundational models can unravel hidden patterns within complex medical datasets. The impact of AI reverberates through medical interventions, encompassing pathology, imaging, genomics, and personalized healthcare, positioning AI as a cornerstone in the quest for precision medicine. The paper delves into the applications of generative AI models in critical facets of healthcare, including decision support, medical imaging, and the prediction of protein structures. The study meticulously evaluates various AI models, such as transfer learning, RNN, autoencoders, and their roles in the healthcare landscape. A pioneering concept introduced in this exploration is that of General Medical AI (GMAI), advocating for the development of reusable and flexible AI models.

CONCLUSION

The review article discusses how AI can revolutionize healthcare by stressing the significance of transparency, fairness and accountability, in AI applications regarding patient data privacy and biases. By tackling these issues and suggesting a governance structure the article adds to the conversation about AI integration in healthcare environments.

Artificial Intelligence Applications in Diabetic Retinopathy: What We Have Now and What to Expect in the Future

Diabetic retinopathy (DR) is a major complication of diabetes mellitus and is a leading cause of vision loss globally. A prompt and accurate diagnosis is crucial for ensuring favorable visual outcomes, highlighting the need for increased access to medical care. The recent remarkable advancements in artificial intelligence (AI) have raised high expectations for its role in disease diagnosis and prognosis prediction across various medical fields. In addition to achieving high precision comparable to that of ophthalmologists, AI-based diagnosis of DR has the potential to improve medical accessibility, especially through telemedicine. In this review paper, we aim to examine the current role of AI in the diagnosis of DR and explore future directions.

Assisting the implementation of screening for type 1 diabetes by using artificial intelligence on publicly available data

The type 1 diabetes community is coalescing around the benefits and advantages of early screening for disease risk. To be accepted by healthcare providers, regulatory authorities and payers, screening programmes need to show that the testing variables allow accurate risk prediction and that individualised risk-informed monitoring plans are established, as well as operational feasibility, cost-effectiveness and acceptance at population level. Artificial intelligence (AI) has the potential to contribute to solving these issues, starting with the identification and stratification of at-risk individuals. ASSET (AI for Sustainable Prevention of Autoimmunity in the Society; www.asset.healthcare ) is a public/private consortium that was established to contribute to research around screening for type 1 diabetes and particularly to how AI can drive the implementation of a precision medicine approach to disease prevention. ASSET will additionally focus on issues pertaining to operational implementation of screening. The authors of this article, researchers and clinicians active in the field of type 1 diabetes, met in an open forum to independently debate key issues around screening for type 1 diabetes and to advise ASSET. The potential use of AI in the analysis of longitudinal data from observational cohort studies to inform the design of improved, more individualised screening programmes was also discussed. A key issue was whether AI would allow the research community and industry to capitalise on large publicly available data repositories to design screening programmes that allow the early detection of individuals at high risk and enable clinical evaluation of preventive therapies. Overall, AI has the potential to revolutionise type 1 diabetes screening, in particular to help identify individuals who are at increased risk of disease and aid in the design of appropriate follow-up plans. We hope that this initiative will stimulate further research on this very timely topic.

A pilot cost-analysis study comparing AI-based EyeArt® and ophthalmologist assessment of diabetic retinopathy in minority women in Oslo, Norway

BACKGROUND

Diabetic retinopathy (DR) is the leading cause of adult blindness in the working age population worldwide, which can be prevented by early detection. Regular eye examinations are recommended and crucial for detecting sight-threatening DR. Use of artificial intelligence (AI) to lessen the burden on the healthcare system is needed.

PURPOSE

To perform a pilot cost-analysis study for detecting DR in a cohort of minority women with DM in Oslo, Norway, that have the highest prevalence of diabetes mellitus (DM) in the country, using both manual (ophthalmologist) and autonomous (AI) grading. This is the first study in Norway, as far as we know, that uses AI in DR- grading of retinal images.

METHODS

On Minority Women's Day, November 1, 2017, in Oslo, Norway, 33 patients (66 eyes) over 18 years of age diagnosed with DM (T1D and T2D) were screened. The Eidon - True Color Confocal Scanner (CenterVue, United States) was used for retinal imaging and graded for DR after screening had been completed, by an ophthalmologist and automatically, using EyeArt Automated DR Detection System, version 2.1.0 (EyeArt, EyeNuk, CA, USA). The gradings were based on the International Clinical Diabetic Retinopathy (ICDR) severity scale [1] detecting the presence or absence of referable DR. Cost-minimization analyses were performed for both grading methods.

RESULTS

33 women (64 eyes) were eligible for the analysis. A very good inter-rater agreement was found: 0.98 (P < 0.01), between the human and AI-based EyeArt grading system for detecting DR. The prevalence of DR was 18.6% (95% CI: 11.4-25.8%), and the sensitivity and specificity were 100% (95% CI: 100-100% and 95% CI: 100-100%), respectively. The cost difference for AI screening compared to human screening was $143 lower per patient (cost-saving) in favour of AI.

CONCLUSION

Our results indicate that The EyeArt AI system is both a reliable, cost-saving, and useful tool for DR grading in clinical practice.

Consolidated Health Economic Evaluation Reporting Standards for Interventions That Use Artificial Intelligence (CHEERS-AI)

OBJECTIVES

Economic evaluations (EEs) are commonly used by decision makers to understand the value of health interventions. The Consolidated Health Economic Evaluation Reporting Standards (CHEERS 2022) provide reporting guidelines for EEs. Healthcare systems will increasingly see new interventions that use artificial intelligence (AI) to perform their function. We developed Consolidated Health Economic Evaluation Reporting Standards for Interventions that use AI (CHEERS-AI) to ensure EEs of AI-based health interventions are reported in a transparent and reproducible manner.

METHODS

Potential CHEERS-AI reporting items were informed by 2 published systematic literature reviews of EEs and a contemporary update. A Delphi study was conducted using 3 survey rounds to elicit multidisciplinary expert views on 26 potential items, through a 9-point Likert rating scale and qualitative comments. An online consensus meeting was held to finalize outstanding reporting items. A digital health patient group reviewed the final checklist from a patient perspective.

RESULTS

A total of 58 participants responded to survey round 1, 42, and 31 of whom responded to rounds 2 and 3, respectively. Nine participants joined the consensus meeting. Ultimately, 38 reporting items were included in CHEERS-AI. They comprised the 28 original CHEERS 2022 items, plus 10 new AI-specific reporting items. Additionally, 8 of the original CHEERS 2022 items were elaborated on to ensure AI-specific nuance is reported.

CONCLUSIONS

CHEERS-AI should be used when reporting an EE of an intervention that uses AI to perform its function. CHEERS-AI will help decision makers and reviewers to understand important AI-specific details of an intervention, and any implications for the EE methods used and cost-effectiveness conclusions.

Cost-effectiveness and cost-utility of a digital technology-driven hierarchical healthcare screening pattern in China

Utilization of digital technologies for cataract screening in primary care is a potential solution for addressing the dilemma between the growing aging population and unequally distributed resources. Here, we propose a digital technology-driven hierarchical screening (DH screening) pattern implemented in China to promote the equity and accessibility of healthcare. It consists of home-based mobile artificial intelligence (AI) screening, community-based AI diagnosis, and referral to hospitals. We utilize decision-analytic Markov models to evaluate the cost-effectiveness and cost-utility of different cataract screening strategies (no screening, telescreening, AI screening and DH screening). A simulated cohort of 100,000 individuals from age 50 is built through a total of 30 1-year Markov cycles. The primary outcomes are incremental cost-effectiveness ratio and incremental cost-utility ratio. The results show that DH screening dominates no screening, telescreening and AI screening in urban and rural China. Annual DH screening emerges as the most economically effective strategy with 341 (338 to 344) and 1326 (1312 to 1340) years of blindness avoided compared with telescreening, and 37 (35 to 39) and 140 (131 to 148) years compared with AI screening in urban and rural settings, respectively. The findings remain robust across all sensitivity analyses conducted. Here, we report that DH screening is cost-effective in urban and rural China, and the annual screening proves to be the most cost-effective option, providing an economic rationale for policymakers promoting public eye health in low- and middle-income countries.

RETFound-enhanced community-based fundus disease screening: real-world evidence and decision curve analysis

Visual impairments and blindness are major public health concerns globally. Effective eye disease screening aided by artificial intelligence (AI) is a promising countermeasure, although it is challenged by practical constraints such as poor image quality in community screening. The recently developed ophthalmic foundation model RETFound has shown higher accuracy in retinal image recognition tasks. This study developed an RETFound-enhanced deep learning (DL) model for multiple-eye disease screening using real-world images from community screenings. Our results revealed that our DL model improved the sensitivity and specificity by over 15% compared with commercial models. Our model also shows better generalisation ability than AI models developed using traditional processes. Additionally, decision curve analysis underscores the higher net benefit of employing our model in both urban and rural settings in China. These findings indicate that the RETFound-enhanced DL model can achieve a higher net benefit in community-based screening, advocating its adoption in low- and middle-income countries to address global eye health challenges.

Service Quality and Residents' Preferences for Facilitated Self-Service Fundus Disease Screening: Cross-Sectional Study

BACKGROUND

Fundus photography is the most important examination in eye disease screening. A facilitated self-service eye screening pattern based on the fully automatic fundus camera was developed in 2022 in Shanghai, China; it may help solve the problem of insufficient human resources in primary health care institutions. However, the service quality and residents' preference for this new pattern are unclear.

OBJECTIVE

This study aimed to compare the service quality and residents' preferences between facilitated self-service eye screening and traditional manual screening and to explore the relationships between the screening service's quality and residents' preferences.

METHODS

We conducted a cross-sectional study in Shanghai, China. Residents who underwent facilitated self-service fundus disease screening at one of the screening sites were assigned to the exposure group; those who were screened with a traditional fundus camera operated by an optometrist at an adjacent site comprised the control group. The primary outcome was the screening service quality, including effectiveness (image quality and screening efficiency), physiological discomfort, safety, convenience, and trustworthiness. The secondary outcome was the participants' preferences. Differences in service quality and the participants' preferences between the 2 groups were compared using chi-square tests separately. Subgroup analyses for exploring the relationships between the screening service's quality and residents' preference were conducted using generalized logit models.

RESULTS

A total of 358 residents enrolled; among them, 176 (49.16%) were included in the exposure group and the remaining 182 (50.84%) in the control group. Residents' basic characteristics were balanced between the 2 groups. There was no significant difference in service quality between the 2 groups (image quality pass rate: P=.79; average screening time: P=.57; no physiological discomfort rate: P=.92; safety rate: P=.78; convenience rate: P=.95; trustworthiness rate: P=.20). However, the proportion of participants who were willing to use the same technology for their next screening was significantly lower in the exposure group than in the control group (P<.001). Subgroup analyses suggest that distrust in the facilitated self-service eye screening might increase the probability of refusal to undergo screening (P=.02).

CONCLUSIONS

This study confirms that the facilitated self-service fundus disease screening pattern could achieve good service quality. However, it was difficult to reverse residents' preferences for manual screening in a short period, especially when the original manual service was already excellent. Therefore, the digital transformation of health care must be cautious. We suggest that attention be paid to the residents' individual needs. More efficient man-machine collaboration and personalized health management solutions based on large language models are both needed.

Extending the diabetic retinopathy screening intervals in Singapore: methodology and preliminary findings of a cohort study

BACKGROUND

The Diabetic Retinopathy Extended Screening Study (DRESS) aims to develop and validate a new DR/diabetic macular edema (DME) risk stratification model in patients with Type 2 diabetes (DM) to identify low-risk groups who can be safely assigned to biennial or triennial screening intervals. We describe the study methodology, participants' baseline characteristics, and preliminary DR progression rates at the first annual follow-up.

METHODS

DRESS is a 3-year ongoing longitudinal study of patients with T2DM and no or mild non-proliferative DR (NPDR, non-referable) who underwent teleophthalmic screening under the Singapore integrated Diabetic Retinopathy Programme (SiDRP) at four SingHealth Polyclinics. Patients with referable DR/DME (> mild NPDR) or ungradable fundus images were excluded. Sociodemographic, lifestyle, medical and clinical information was obtained from medical records and interviewer-administered questionnaires at baseline. These data are extracted from medical records at 12, 24 and 36 months post-enrollment. Baseline descriptive characteristics stratified by DR severity at baseline and rates of progression to referable DR at 12-month follow-up were calculated.

RESULTS

Of 5,840 eligible patients, 78.3% (n = 4,570, median [interquartile range [IQR] age 61.0 [55-67] years; 54.7% male; 68.0% Chinese) completed the baseline assessment. At baseline, 97.4% and 2.6% had none and mild NPDR (worse eye), respectively. Most participants had hypertension (79.2%) and dyslipidemia (92.8%); and almost half were obese (43.4%, BMI ≥ 27.5 kg/m2). Participants without DR (vs mild DR) reported shorter DM duration, and had lower haemoglobin A1c, triglycerides and urine albumin/creatinine ratio (all p < 0.05). To date, we have extracted 41.8% (n = 1909) of the 12-month follow-up data. Of these, 99.7% (n = 1,904) did not progress to referable DR. Those who progressed to referable DR status (0.3%) had no DR at baseline.

CONCLUSIONS

In our prospective study of patients with T2DM and non-referable DR attending polyclinics, we found extremely low annual DR progression rates. These preliminary results suggest that extending screening intervals beyond 12 months may be viable and safe for most participants, although our 3-year follow up data are needed to substantiate this claim and develop the risk stratification model to identify low-risk patients with T2DM who can be assigned biennial or triennial screening intervals.

An Economic Analysis for the Use of Artificial Intelligence in Screening for Diabetic Retinopathy in Trinidad and Tobago

This is a systematic review of 25 publications on the topics of the prevalence and cost of diabetic retinopathy (DR) in Trinidad and Tobago, the cost of traditional methods of screening for DR, and the use and cost of artificial intelligence (AI) in screening for DR. Analysis of these publications was used to identify and make estimates for how resources allocated to ophthalmology in public health systems in Trinidad and Tobago can be more efficiently utilized by employing AI in diagnosing treatable DR. DR screening was found to be an effective method of detecting the disease. Screening was found to be a universally cost-effective method of disease prevention and for altering the natural history of the disease in the spectrum of low-middle to high-income economies, such as Rwanda, Thailand, China, South Korea, and Singapore. AI and deep learning systems were found to be clinically superior to, or as effective as, human graders in areas where they were deployed, indicating that the systems are clinically safe. They have been shown to improve access to diabetic retinal screening, improve compliance with screening appointments, and prove to be cost-effective, especially in rural areas. Trinidad and Tobago, which is estimated to be disproportionately more affected by the burden of DR when projected out to the mid-21st century, stands to save as much as US$60 million annually from the implementation of an AI-based system to screen for DR versus conventional manual grading.

Evaluating ChatGPT's Capabilities on Orthopedic Training Examinations: An Analysis of New Image Processing Features

Introduction The efficacy of integrating artificial intelligence (AI) models like ChatGPT into the medical field, specifically orthopedic surgery, has yet to be fully determined. The most recent adaptation of ChatGPT that has yet to be explored is its image analysis capabilities. This study assesses ChatGPT's performance in answering Orthopedic In-Training Examination (OITE) questions, including those that require image analysis. Methods Questions from the 2014, 2015, 2021, and 2022 AAOS OITE were screened for inclusion. All questions without images were entered into ChatGPT 3.5 and 4.0 twice. Questions that necessitated the use of images were only entered into ChatGPT 4.0 twice, as this is the only version of the system that can analyze images. The responses were recorded and compared to AAOS's correct answers, evaluating the AI's accuracy and precision. Results A total of 940 questions were included in the final analysis (457 questions with images and 483 questions without images). ChatGPT 4.0 performed significantly better on questions that did not require image analysis (67.81% vs 47.59%, p<0.001). Discussion While the use of AI in orthopedics is an intriguing possibility, this evaluation demonstrates how, even with the addition of image processing capabilities, ChatGPT still falls short in terms of its accuracy. As AI technology evolves, ongoing research is vital to harness AI's potential effectively, ensuring it complements rather than attempts to replace the nuanced skills of orthopedic surgeons.

Economic evaluation for medical artificial intelligence: accuracy vs. cost-effectiveness in a diabetic retinopathy screening case

Artificial intelligence (AI) models have shown great accuracy in health screening. However, for real-world implementation, high accuracy may not guarantee cost-effectiveness. Improving AI's sensitivity finds more high-risk patients but may raise medical costs while increasing specificity reduces unnecessary referrals but may weaken detection capability. To evaluate the trade-off between AI model performance and the long-running cost-effectiveness, we conducted a cost-effectiveness analysis in a nationwide diabetic retinopathy (DR) screening program in China, comprising 251,535 participants with diabetes over 30 years. We tested a validated AI model in 1100 different diagnostic performances (presented as sensitivity/specificity pairs) and modeled annual screening scenarios. The status quo was defined as the scenario with the most accurate AI performance. The incremental cost-effectiveness ratio (ICER) was calculated for other scenarios against the status quo as cost-effectiveness metrics. Compared to the status quo (sensitivity/specificity: 93.3%/87.7%), six scenarios were cost-saving and seven were cost-effective. To achieve cost-saving or cost-effective, the AI model should reach a minimum sensitivity of 88.2% and specificity of 80.4%. The most cost-effective AI model exhibited higher sensitivity (96.3%) and lower specificity (80.4%) than the status quo. In settings with higher DR prevalence and willingness-to-pay levels, the AI needed higher sensitivity for optimal cost-effectiveness. Urban regions and younger patient groups also required higher sensitivity in AI-based screening. In real-world DR screening, the most accurate AI model may not be the most cost-effective. Cost-effectiveness should be independently evaluated, which is most likely to be affected by the AI's sensitivity.

Validation of a deep learning system for the detection of diabetic retinopathy in Indigenous Australians

BACKGROUND/AIMS

Deep learning systems (DLSs) for diabetic retinopathy (DR) detection show promising results but can underperform in racial and ethnic minority groups, therefore external validation within these populations is critical for health equity. This study evaluates the performance of a DLS for DR detection among Indigenous Australians, an understudied ethnic group who suffer disproportionately from DR-related blindness.

METHODS

We performed a retrospective external validation study comparing the performance of a DLS against a retinal specialist for the detection of more-than-mild DR (mtmDR), vision-threatening DR (vtDR) and all-cause referable DR. The validation set consisted of 1682 consecutive, single-field, macula-centred retinal photographs from 864 patients with diabetes (mean age 54.9 years, 52.4% women) at an Indigenous primary care service in Perth, Australia. Three-person adjudication by a panel of specialists served as the reference standard.

RESULTS

For mtmDR detection, sensitivity of the DLS was superior to the retina specialist (98.0% (95% CI, 96.5 to 99.4) vs 87.1% (95% CI, 83.6 to 90.6), McNemar's test p<0.001) with a small reduction in specificity (95.1% (95% CI, 93.6 to 96.4) vs 97.0% (95% CI, 95.9 to 98.0), p=0.006). For vtDR, the DLS's sensitivity was again superior to the human grader (96.2% (95% CI, 93.4 to 98.6) vs 84.4% (95% CI, 79.7 to 89.2), p<0.001) with a slight drop in specificity (95.8% (95% CI, 94.6 to 96.9) vs 97.8% (95% CI, 96.9 to 98.6), p=0.002). For all-cause referable DR, there was a substantial increase in sensitivity (93.7% (95% CI, 91.8 to 95.5) vs 74.4% (95% CI, 71.1 to 77.5), p<0.001) and a smaller reduction in specificity (91.7% (95% CI, 90.0 to 93.3) vs 96.3% (95% CI, 95.2 to 97.4), p<0.001).

CONCLUSION

The DLS showed improved sensitivity and similar specificity compared with a retina specialist for DR detection. This demonstrates its potential to support DR screening among Indigenous Australians, an underserved population with a high burden of diabetic eye disease.

Artificial intelligence, ChatGPT, and other large language models for social determinants of health: Current state and future directions

This perspective highlights the importance of addressing social determinants of health (SDOH) in patient health outcomes and health inequity, a global problem exacerbated by the COVID-19 pandemic. We provide a broad discussion on current developments in digital health and artificial intelligence (AI), including large language models (LLMs), as transformative tools in addressing SDOH factors, offering new capabilities for disease surveillance and patient care. Simultaneously, we bring attention to challenges, such as data standardization, infrastructure limitations, digital literacy, and algorithmic bias, that could hinder equitable access to AI benefits. For LLMs, we highlight potential unique challenges and risks including environmental impact, unfair labor practices, inadvertent disinformation or "hallucinations," proliferation of bias, and infringement of copyrights. We propose the need for a multitiered approach to digital inclusion as an SDOH and the development of ethical and responsible AI practice frameworks globally and provide suggestions on bridging the gap from development to implementation of equitable AI technologies.

Population impact and cost-effectiveness of artificial intelligence-based diabetic retinopathy screening in people living with diabetes in Australia: a cost effectiveness analysis

Background

We aimed to evaluate the cost-effectiveness of an artificial intelligence-(AI) based diabetic retinopathy (DR) screening system in the primary care setting for both non-Indigenous and Indigenous people living with diabetes in Australia.

Methods

We performed a cost-effectiveness analysis between January 01, 2022 and August 01, 2023. A decision-analytic Markov model was constructed to simulate DR progression in a population of 1,197,818 non-Indigenous and 65,160 Indigenous Australians living with diabetes aged ≥20 years over 40 years. From a healthcare provider's perspective, we compared current practice to three primary care AI-based screening scenarios-(A) substitution of current manual grading, (B) scaling up to patient acceptance level, and (C) achieving universal screening. Study results were presented as incremental cost-effectiveness ratio (ICER), benefit-cost ratio (BCR), and net monetary benefits (NMB). A Willingness-to-pay (WTP) threshold of AU$50,000 per quality-adjusted life year (QALY) and a discount rate of 3.5% were adopted in this study.

Findings

With the status quo, the non-Indigenous diabetic population was projected to develop 96,269 blindness cases, resulting in AU$13,039.6 m spending on DR screening and treatment during 2020-2060. In comparison, all three intervention scenarios were effective and cost-saving. In particular, if a universal screening program was to be implemented (Scenario C), it would prevent 38,347 blindness cases, gain 172,090 QALYs and save AU$595.8 m, leading to a BCR of 3.96 and NMB of AU$9,200 m. Similar findings were also reported in the Indigenous population. With the status quo, 3,396 Indigenous individuals would develop blindness, which would cost the health system AU$796.0 m during 2020-2060. All three intervention scenarios were cost-saving for the Indigenous population. Notably, universal AI-based DR screening (Scenario C) would prevent 1,211 blindness cases and gain 9,800 QALYs in the Indigenous population, leading to a saving of AU$19.2 m with a BCR of 1.62 and NMB of AU$509 m.

Interpretation

Our findings suggest that implementing AI-based DR screening in primary care is highly effective and cost-saving in both Indigenous and non-Indigenous populations.

Funding

This project received grant funding from the Australian Government: the National Critical Research Infrastructure Initiative, Medical Research Future Fund (MRFAI00035) and the NHMRC Investigator Grant (APP1175405). The contents of the published material are solely the responsibility of the Administering Institution, a participating institution or individual authors and do not reflect the views of the NHMRC. This work was supported by the Global STEM Professorship Scheme (P0046113), the Fundamental Research Funds of the State Key Laboratory of Ophthalmology, Project of Investigation on Health Status of Employees in Financial Industry in Guangzhou, China (Z012014075). The Centre for Eye Research Australia receives Operational Infrastructure Support from the Victorian State Government. W.H. is supported by the Melbourne Research Scholarship established by the University of Melbourne. The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

AI-Human Hybrid Workflow Enhances Teleophthalmology for the Detection of Diabetic Retinopathy

Objective

Detection of diabetic retinopathy (DR) outside of specialized eye care settings is an important means of access to vision-preserving health maintenance. Remote interpretation of fundus photographs acquired in a primary care or other nonophthalmic setting in a store-and-forward manner is a predominant paradigm of teleophthalmology screening programs. Artificial intelligence (AI)-based image interpretation offers an alternative means of DR detection. IDx-DR (Digital Diagnostics Inc) is a Food and Drug Administration-authorized autonomous testing device for DR. We evaluated the diagnostic performance of IDx-DR compared with human-based teleophthalmology over 2 and a half years. Additionally, we evaluated an AI-human hybrid workflow that combines AI-system evaluation with human expert-based assessment for referable cases.

Design

Prospective cohort study and retrospective analysis.

Participants

Diabetic patients ≥ 18 years old without a prior DR diagnosis or DR examination in the past year presenting for routine DR screening in a primary care clinic.

Methods

Macula-centered and optic nerve-centered fundus photographs were evaluated by an AI algorithm followed by consensus-based overreading by retina specialists at the Stanford Ophthalmic Reading Center. Detection of more-than-mild diabetic retinopathy (MTMDR) was compared with in-person examination by a retina specialist.

Main Outcome Measures

Sensitivity, specificity, accuracy, positive predictive value, and gradability achieved by the AI algorithm and retina specialists.

Results

The AI algorithm had higher sensitivity (95.5% sensitivity; 95% confidence interval [CI], 86.7%-100%) but lower specificity (60.3% specificity; 95% CI, 47.7%-72.9%) for detection of MTMDR compared with remote image interpretation by retina specialists (69.5% sensitivity; 95% CI, 50.7%-88.3%; 96.9% specificity; 95% CI, 93.5%-100%). Gradability of encounters was also lower for the AI algorithm (62.5%) compared with retina specialists (93.1%). A 2-step AI-human hybrid workflow in which the AI algorithm initially rendered an assessment followed by overread by a retina specialist of MTMDR-positive encounters resulted in a sensitivity of 95.5% (95% CI, 86.7%-100%) and a specificity of 98.2% (95% CI, 94.6%-100%). Similarly, a 2-step overread by retina specialists of AI-ungradable encounters improved gradability from 63.5% to 95.6% of encounters.

Conclusions

Implementation of an AI-human hybrid teleophthalmology workflow may both decrease reliance on human specialist effort and improve diagnostic accuracy.

Financial Disclosures

Proprietary or commercial disclosure may be found after the references.

Undiagnosed diabetic retinopathy in Northeast China: prevalence and determinants

Objective

To report the prevalence and contributing factors of undiagnosed diabetic retinopathy (DR) in a population from Northeastern China.

Subjects/Methods

A total of 800 subjects from the Fushun Diabetic Retinopathy Cohort Study were enrolled. A questionnaire assessing incentives and barriers to diagnosis of DR was administered. Logistic regression was used to identify clinical and sociodemographic factors associated with undiagnosed DR. In a prespecified subgroup analysis, we divided patients into vision-threatening diabetic retinopathy (VTDR) and non-VTDR (NVTDR) subgroups.

Results

Among 800 participants with DR, 712 (89.0%) were undiagnosed. Among 601 with NVTDR, 566 (94.2%) were undiagnosed. Among 199 with VTDR, 146 (73.4%) were undiagnosed. The risk factors affecting the timely diagnosis of NVTDR and VTDR exhibit significant disparities. In multivariate models, factors associated with undiagnosed VTDR were age over 60 years (OR = 2.966; 95% CI = 1.205-7.299; P = 0.018), duration of diabetes over 10 years (OR = 0.299; 95% CI = 0.118-0753; P = 0.010), visual impairment or blindness (OR = 0.310; 95% CI = 0.117-0.820; P = 0.018), receiving a reminder to schedule an eye examination (OR = 0.380; 95% CI = 0.163-0.883; P = 0.025), and the belief that "people with diabetes are unlikely to develop an eye disease" (OR = 4.691; 95% CI = 1.116-19.724; P = 0.035). However, none of the factors were associated with undiagnosed NVTDR (all P ≥ 0.145).

Conclusion

Our research has uncovered a disconcerting trend of underdiagnosis in cases of DR within our population. Addressing determinants of undiagnosed DR may facilitate early detection.

Cost-effectiveness analysis of artificial intelligence-based diabetic retinopathy screening in rural China based on the Markov model

This study assessed the cost-effectiveness of different diabetic retinopathy (DR) screening strategies in rural regions in China by using a Markov model to make health economic evaluations. In this study, we determined the structure of a Markov model according to the research objectives, which required parameters collected through field investigation and literature retrieval. After perfecting the model with parameters and assumptions, we developed a Markov decision analytic model according to the natural history of DR in TreeAge Pro 2011. For this model, we performed Markov cohort and cost-effectiveness analyses to simulate the probabilistic distributions of different developments in DR and the cumulative cost-effectiveness of artificial intelligence (AI)-based screening and ophthalmologist screening for DR in the rural population with diabetes mellitus (DM) in China. Additionally, a model-based health economic evaluation was performed by using quality-adjusted life years (QALYs) and incremental cost-effectiveness ratios. Last, one-way and probabilistic sensitivity analyses were performed to assess the stability of the results. From the perspective of the health system, compared with no screening, AI-based screening cost more (the incremental cost was 37,257.76 RMB (approximately 5,211.31 US dollars)), but the effect was better (the incremental utility was 0.33). Compared with AI-based screening, the cost of ophthalmologist screening was higher (the incremental cost was 14,886.76 RMB (approximately 2,070.19 US dollars)), and the effect was worse (the incremental utility was -0.31). Compared with no screening, the incremental cost-effectiveness ratio (ICER) of AI-based DR screening was 112,146.99 RMB (15,595.47 US dollars)/QALY, which was less than the threshold for the ICER (< 3 times the per capita gross domestic product (GDP), 217,341.00 RMB (30,224.03 US dollars)). Therefore, AI-based screening was cost-effective, which meant that the increased cost for each additional quality-adjusted life year was merited. Compared with no screening and ophthalmologist screening for DR, AI-based screening was the most cost-effective, which not only saved costs but also improved the quality of life of diabetes patients. Popularizing AI-based DR screening strategies in rural areas would be economically effective and feasible and can provide a scientific basis for the further formulation of early screening programs for diabetic retinopathy.

Artificial intelligence in retinal disease: clinical application, challenges, and future directions

Retinal diseases are a leading cause of blindness in developed countries, accounting for the largest share of visually impaired children, working-age adults (inherited retinal disease), and elderly individuals (age-related macular degeneration). These conditions need specialised clinicians to interpret multimodal retinal imaging, with diagnosis and intervention potentially delayed. With an increasing and ageing population, this is becoming a global health priority. One solution is the development of artificial intelligence (AI) software to facilitate rapid data processing. Herein, we review research offering decision support for the diagnosis, classification, monitoring, and treatment of retinal disease using AI. We have prioritised diabetic retinopathy, age-related macular degeneration, inherited retinal disease, and retinopathy of prematurity. There is cautious optimism that these algorithms will be integrated into routine clinical practice to facilitate access to vision-saving treatments, improve efficiency of healthcare systems, and assist clinicians in processing the ever-increasing volume of multimodal data, thereby also liberating time for doctor-patient interaction and co-development of personalised management plans.

Uncertainty-inspired open set learning for retinal anomaly identification

Failure to recognize samples from the classes unseen during training is a major limitation of artificial intelligence in the real-world implementation for recognition and classification of retinal anomalies. We establish an uncertainty-inspired open set (UIOS) model, which is trained with fundus images of 9 retinal conditions. Besides assessing the probability of each category, UIOS also calculates an uncertainty score to express its confidence. Our UIOS model with thresholding strategy achieves an F1 score of 99.55%, 97.01% and 91.91% for the internal testing set, external target categories (TC)-JSIEC dataset and TC-unseen testing set, respectively, compared to the F1 score of 92.20%, 80.69% and 64.74% by the standard AI model. Furthermore, UIOS correctly predicts high uncertainty scores, which would prompt the need for a manual check in the datasets of non-target categories retinal diseases, low-quality fundus images, and non-fundus images. UIOS provides a robust method for real-world screening of retinal anomalies.

Artificial intelligence in diabetes management: Advancements, opportunities, and challenges

The increasing prevalence of diabetes, high avoidable morbidity and mortality due to diabetes and diabetic complications, and related substantial economic burden make diabetes a significant health challenge worldwide. A shortage of diabetes specialists, uneven distribution of medical resources, low adherence to medications, and improper self-management contribute to poor glycemic control in patients with diabetes. Recent advancements in digital health technologies, especially artificial intelligence (AI), provide a significant opportunity to achieve better efficiency in diabetes care, which may diminish the increase in diabetes-related health-care expenditures. Here, we review the recent progress in the application of AI in the management of diabetes and then discuss the opportunities and challenges of AI application in clinical practice. Furthermore, we explore the possibility of combining and expanding upon existing digital health technologies to develop an AI-assisted digital health-care ecosystem that includes the prevention and management of diabetes.

Democratizing Artificial Intelligence Imaging Analysis With Automated Machine Learning: Tutorial

Deep learning-based clinical imaging analysis underlies diagnostic artificial intelligence (AI) models, which can match or even exceed the performance of clinical experts, having the potential to revolutionize clinical practice. A wide variety of automated machine learning (autoML) platforms lower the technical barrier to entry to deep learning, extending AI capabilities to clinicians with limited technical expertise, and even autonomous foundation models such as multimodal large language models. Here, we provide a technical overview of autoML with descriptions of how autoML may be applied in education, research, and clinical practice. Each stage of the process of conducting an autoML project is outlined, with an emphasis on ethical and technical best practices. Specifically, data acquisition, data partitioning, model training, model validation, analysis, and model deployment are considered. The strengths and limitations of available code-free, code-minimal, and code-intensive autoML platforms are considered. AutoML has great potential to democratize AI in medicine, improving AI literacy by enabling "hands-on" education. AutoML may serve as a useful adjunct in research by facilitating rapid testing and benchmarking before significant computational resources are committed. AutoML may also be applied in clinical contexts, provided regulatory requirements are met. The abstraction by autoML of arduous aspects of AI engineering promotes prioritization of data set curation, supporting the transition from conventional model-driven approaches to data-centric development. To fulfill its potential, clinicians must be educated on how to apply these technologies ethically, rigorously, and effectively; this tutorial represents a comprehensive summary of relevant considerations.

Artificial Intelligence and Diabetic Retinopathy: AI Framework, Prospective Studies, Head-to-head Validation, and Cost-effectiveness

Current guidelines recommend that individuals with diabetes receive yearly eye exams for detection of referable diabetic retinopathy (DR), one of the leading causes of new-onset blindness. For addressing the immense screening burden, artificial intelligence (AI) algorithms have been developed to autonomously screen for DR from fundus photography without human input. Over the last 10 years, many AI algorithms have achieved good sensitivity and specificity (>85%) for detection of referable DR compared with human graders; however, many questions still remain. In this narrative review on AI in DR screening, we discuss key concepts in AI algorithm development as a background for understanding the algorithms. We present the AI algorithms that have been prospectively validated against human graders and demonstrate the variability of reference standards and cohort demographics. We review the limited head-to-head validation studies where investigators attempt to directly compare the available algorithms. Next, we discuss the literature regarding cost-effectiveness, equity and bias, and medicolegal considerations, all of which play a role in the implementation of these AI algorithms in clinical practice. Lastly, we highlight ongoing efforts to bridge gaps in AI model data sets to pursue equitable development and delivery.

Feasibility Analysis of Teleophthalmology Diagnosis and Referral in Remote Areas of Taiwan

Background: Remote areas of Taiwan lack routine and specialized ophthalmology services. This study aimed to analyze feasibility of teleophthalmology service for diseases diagnosis and referral in remote areas of Taiwan. Methods: A retrospective study of medical records from 11 remote teleophthalmology clinics in the Taitung area of Taiwan was conducted from May 2020 to December 2021. Vision and intraocular pressure were checked. Ophthalmic imaging was performed by local trained nurses using a hand-held ophthalmoscope and slit lamp biomicroscope. The images were transmitted by telemedicine system to a medical center. Consultation was conducted via face-to-face real-time video calls. Ophthalmologists in the medical center provided diagnosis and treatment advice based on the real-time images and interactive history taking via the telemedicine system. All the images and data were collected and well-reviewed by ophthalmologists in the medical center, and disease prevalence and referral were analyzed for the program. A small-scale satisfaction questionnaire survey was conducted for efficacy evaluation of the program. Results: A total of 1,401 medical records from 1,094 patients were collected and screened. Patients' ages ranged from 9 months to 94 years, with a mean age of 57.27 (standard deviation ±20.47) years. The most frequent ophthalmologic diagnosis was dry eye disease (20.2%), followed by conjunctivitis (12.4%). Among 322 patients with underlying diseases of diabetes mellitus, 59 patients (18.3%) were diagnosed with diabetic retinopathy. Major diagnosis was made in 102 patients (7.3%) and referral to hospital for further management was suggested. This program had high overall satisfaction score of 89% (mean 4.43 ± 0.52 points) in satisfaction questionnaire survey. Conclusion: Teleophthalmology provides an alternative tool for ocular disease diagnosis and screening for patients in remote areas, especially during the COVID-19 pandemic. This service helps to detect major but undiagnosed diseases and promotes health care accessibility and availability in remote areas that lack specialists.

Artificial intelligence for telemedicine diabetic retinopathy screening: a review

PURPOSE

This study aims to compare artificial intelligence (AI) systems applied in diabetic retinopathy (DR) teleophthalmology screening, currently deployed systems, fairness initiatives and the challenges for implementation.

METHODS

The review included articles retrieved from PubMed/Medline/EMBASE literature search strategy regarding telemedicine, DR and AI. The screening criteria included human articles in English, Portuguese or Spanish and related to telemedicine and AI for DR screening. The author's affiliations and the study's population income group were classified according to the World Bank Country and Lending Groups.

RESULTS

The literature search yielded a total of 132 articles, and nine were included after full-text assessment. The selected articles were published between 2004 and 2020 and were grouped as telemedicine systems, algorithms, economic analysis and image quality assessment. Four telemedicine systems that perform a quality assessment, image preprocessing and pathological screening were reviewed. A data and post-deployment bias assessment are not performed in any of the algorithms, and none of the studies evaluate the social impact implementations. There is a lack of representativeness in the reviewed articles, with most authors and target populations from high-income countries and no low-income country representation.

CONCLUSIONS

Telemedicine and AI hold great promise for augmenting decision-making in medical care, expanding patient access and enhancing cost-effectiveness. Economic studies and social science analysis are crucial to support the implementation of AI in teleophthalmology screening programs. Promoting fairness and generalizability in automated systems combined with telemedicine screening programs is not straightforward. Improving data representativeness, reducing biases and promoting equity in deployment and post-deployment studies are all critical steps in model development.

Artificial intelligence and digital health in global eye health: opportunities and challenges

Global eye health is defined as the degree to which vision, ocular health, and function are maximised worldwide, thereby optimising overall wellbeing and quality of life. Improving eye health is a global priority as a key to unlocking human potential by reducing the morbidity burden of disease, increasing productivity, and supporting access to education. Although extraordinary progress fuelled by global eye health initiatives has been made over the last decade, there remain substantial challenges impeding further progress. The accelerated development of digital health and artificial intelligence (AI) applications provides an opportunity to transform eye health, from facilitating and increasing access to eye care to supporting clinical decision making with an objective, data-driven approach. Here, we explore the opportunities and challenges presented by digital health and AI in global eye health and describe how these technologies could be leveraged to improve global eye health. AI, telehealth, and emerging technologies have great potential, but require specific work to overcome barriers to implementation. We suggest that a global digital eye health task force could facilitate coordination of funding, infrastructural development, and democratisation of AI and digital health to drive progress forwards in this domain.

Recent evidence of economic evaluation of artificial intelligence in ophthalmology

PURPOSE OF REVIEW

Health economic evaluation (HEE) is essential for assessing value of health interventions, including artificial intelligence. Recent approaches, current challenges, and future directions of HEE of artificial intelligence in ophthalmology are reviewed.

RECENT FINDINGS

Majority of recent HEEs of artificial intelligence in ophthalmology were for diabetic retinopathy screening. Two models, one conducted in the rural USA (5-year period) and another in China (35-year period), found artificial intelligence to be more cost-effective than without screening for diabetic retinopathy. Two additional models, which compared artificial intelligence with human screeners in Brazil and Thailand for the lifetime of patients, found artificial intelligence to be more expensive from a healthcare system perspective. In the Thailand analysis, however, artificial intelligence was less expensive when opportunity loss from blindness was included. An artificial intelligence model for screening retinopathy of prematurity was cost-effective in the USA. A model for screening age-related macular degeneration in Japan and another for primary angle close in China did not find artificial intelligence to be cost-effective, compared with no screening. The costs of artificial intelligence varied widely in these models.

SUMMARY

Like other medical fields, there is limited evidence in assessing the value of artificial intelligence in ophthalmology and more appropriate HEE models are needed.

Is Artificial Intelligence the Cost-Saving Lens to Diabetic Retinopathy Screening in Low- and Middle-Income Countries?

Diabetes is a rapidly growing global health crisis disproportionately affecting low- and middle-income countries (LMICs). The emergence of diabetes as a global pandemic is one of the major challenges to human health, as long-term microvascular complications such as diabetic retinopathy (DR) can lead to irreversible blindness. Leveraging artificial intelligence (AI) technology may improve the diagnostic accuracy, efficiency, and accessibility of DR screenings across LMICs. However, there is a gap between the potential of AI technology and its implementation in clinical practice. The main objective of this systematic review is to summarize the currently available literature on the health economic assessments of AI implementation for DR screening in LMICs. The review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines. We conducted an extensive systematic search of PubMed/MEDLINE, Scopus, and the Web of Science on July 15, 2023. Our review included full-text English-language articles from any publication year. The Joanna Briggs Institute's (JBI) critical appraisal checklist for economic evaluations was used to rate the quality and rigor of the selected articles. The initial search generated 1,423 records and was narrowed to five full-text articles through comprehensive inclusion and exclusion criteria. Of the five articles included in our systematic review, two used a cost-effectiveness analysis, two used a cost-utility analysis, and one used both a cost-effectiveness analysis and a cost-utility analysis. Across the five articles, LMICs such as China, Thailand, and Brazil were represented in the economic evaluations and models. Overall, three out of the five articles concluded that AI-based DR screening was more cost-effective in comparison to standard-of-care screening methods. Our systematic review highlights the need for more primary health economic analyses that carefully evaluate the economic implications of adopting AI technology for DR screening in LMICs. We hope this systematic review will offer valuable guidance to healthcare providers, scientists, and legislators to support appropriate decision-making regarding the implementation of AI algorithms for DR screening in healthcare workflows.

Single retinal image for diabetic retinopathy screening: performance of a handheld device with embedded artificial intelligence

BACKGROUND

Diabetic retinopathy (DR) is a leading cause of blindness. Our objective was to evaluate the performance of an artificial intelligence (AI) system integrated into a handheld smartphone-based retinal camera for DR screening using a single retinal image per eye.

METHODS

Images were obtained from individuals with diabetes during a mass screening program for DR in Blumenau, Southern Brazil, conducted by trained operators. Automatic analysis was conducted using an AI system (EyerMaps™, Phelcom Technologies LLC, Boston, USA) with one macula-centered, 45-degree field of view retinal image per eye. The results were compared to the assessment by a retinal specialist, considered as the ground truth, using two images per eye. Patients with ungradable images were excluded from the analysis.

RESULTS

A total of 686 individuals (average age 59.2 ± 13.3 years, 56.7% women, diabetes duration 12.1 ± 9.4 years) were included in the analysis. The rates of insulin use, daily glycemic monitoring, and systemic hypertension treatment were 68.4%, 70.2%, and 70.2%, respectively. Although 97.3% of patients were aware of the risk of blindness associated with diabetes, more than half of them underwent their first retinal examination during the event. The majority (82.5%) relied exclusively on the public health system. Approximately 43.4% of individuals were either illiterate or had not completed elementary school. DR classification based on the ground truth was as follows: absent or nonproliferative mild DR 86.9%, more than mild (mtm) DR 13.1%. The AI system achieved sensitivity, specificity, positive predictive value, and negative predictive value percentages (95% CI) for mtmDR as follows: 93.6% (87.8-97.2), 71.7% (67.8-75.4), 42.7% (39.3-46.2), and 98.0% (96.2-98.9), respectively. The area under the ROC curve was 86.4%.

CONCLUSION

The portable retinal camera combined with AI demonstrated high sensitivity for DR screening using only one image per eye, offering a simpler protocol compared to the traditional approach of two images per eye. Simplifying the DR screening process could enhance adherence rates and overall program coverage.

Are current clinical studies on artificial intelligence-based medical devices comprehensive enough to support a full health technology assessment? A systematic review

INTRODUCTION

Artificial Intelligence-based Medical Devices (AI-based MDs) are experiencing exponential growth in healthcare. This study aimed to investigate whether current studies assessing AI contain the information required for health technology assessment (HTA) by HTA bodies.

METHODS

We conducted a systematic literature review based on the Preferred Reporting Items for Systematic Reviews and Meta-Analyses methodology to extract articles published between 2016 and 2021 related to the assessment of AI-based MDs. Data extraction focused on study characteristics, technology, algorithms, comparators, and results. AI quality assessment and HTA scores were calculated to evaluate whether the items present in the included studies were concordant with the HTA requirements. We performed a linear regression for the HTA and AI scores with the explanatory variables of the impact factor, publication date, and medical specialty. We conducted a univariate analysis of the HTA score and a multivariate analysis of the AI score with an alpha risk of 5 %.

RESULTS

Of 5578 retrieved records, 56 were included. The mean AI quality assessment score was 67 %; 32 % of articles had an AI quality score ≥ 70 %, 50 % had a score between 50 % and 70 %, and 18 % had a score under 50 %. The highest quality scores were observed for the study design (82 %) and optimisation (69 %) categories, whereas the scores were lowest in the clinical practice category (23 %). The mean HTA score was 52 % for all seven domains. 100 % of the studies assessed clinical effectiveness, whereas only 9 % evaluated safety, and 20 % evaluated economic issues. There was a statistically significant relationship between the impact factor and the HTA and AI scores (both p = 0.046).

DISCUSSION

Clinical studies on AI-based MDs have limitations and often lack adapted, robust, and complete evidence. High-quality datasets are also required because the output data can only be trusted if the inputs are reliable. The existing assessment frameworks are not specifically designed to assess AI-based MDs. From the perspective of regulatory authorities, we suggest that these frameworks should be adapted to assess the interpretability, explainability, cybersecurity, and safety of ongoing updates. From the perspective of HTA agencies, we highlight that transparency, professional and patient acceptance, ethical issues, and organizational changes are required for the implementation of these devices. Economic assessments of AI should rely on a robust methodology (business impact or health economic models) to provide decision-makers with more reliable evidence.

CONCLUSION

Currently, AI studies are insufficient to cover HTA prerequisites. HTA processes also need to be adapted because they do not consider the important specificities of AI-based MDs. Specific HTA workflows and accurate assessment tools should be designed to standardise evaluations, generate reliable evidence, and create confidence.

A multi-centre prospective evaluation of THEIA™ to detect diabetic retinopathy (DR) and diabetic macular oedema (DMO) in the New Zealand screening program

PURPOSE

To validate the potential application of THEIA™ as clinical decision making assistant in a national screening program.

METHODS

A total of 900 patients were recruited from either an urban large eye hospital, or a semi-rural optometrist led screening provider, as they were attending their appointment as part of New Zealand Diabetic Eye Screening Programme. The de-identified images were independently graded by three senior specialists, and final results were aggregated using New Zealand grading scheme, which was then converted to referable/non-referable and Healthy/mild/more than mild/sight threatening categories.

RESULTS

THEIA™ managed to grade all images obtained during the study. Comparing the adjudicated images from the specialist grading team, "ground truth", with the grading by the AI platform in detecting "sight threatening" disease, at the patient level THEIA™ achieved 100% imageability, 100% [98.49-100.00%] sensitivity and [97.02-99.16%] specificity, and negative predictive value of 100%. In other words, THEIA™ did not miss any patients with "more than mild" or "sight threatening" disease. The level of agreement between the clinicians and the aggregated results was (k value: 0.9881, 0.9557, and 0.9175), and the level of agreement between THEIA™ and the aggregated labels was (k value: 0.9515).

CONCLUSION

This multi-centre prospective trial showed that THEIA™ did not miss referable disease when screening for diabetic retinopathy and maculopathy. It also had a very high level of granularity in reporting the disease level. As THEIA™ has been tested on a variety of cameras, operating in a range of clinics (rural/urban, ophthalmologist-led\optometrist-led), we believe that it will be a suitable addition to a public diabetic screening program.

Artificial intelligence innovation in healthcare: Relevance of reporting guidelines for clinical translation from bench to bedside

Artificial intelligence (AI) and digital innovation are transforming healthcare. Technologies such as machine learning in image analysis, natural language processing in medical chatbots and electronic medical record extraction have the potential to improve screening, diagnostics and prognostication, leading to precision medicine and preventive health. However, it is crucial to ensure that AI research is conducted with scientific rigour to facilitate clinical implementation. Therefore, reporting guidelines have been developed to standardise and streamline the development and validation of AI technologies in health. This commentary proposes a structured approach to utilise these reporting guidelines for the translation of promising AI techniques from research and development into clinical translation, and eventual widespread implementation from bench to bedside.

Cost-Utility Analysis of Deep Learning and Trained Human Graders for Diabetic Retinopathy Screening in a Nationwide Program

INTRODUCTION

Deep learning (DL) for screening diabetic retinopathy (DR) has the potential to address limited healthcare resources by enabling expanded access to healthcare. However, there is still limited health economic evaluation, particularly in low- and middle-income countries, on this subject to aid decision-making for DL adoption.

METHODS

In the context of a middle-income country (MIC), using Thailand as a model, we constructed a decision tree-Markov hybrid model to estimate lifetime costs and outcomes of Thailand's national DR screening program via DL and trained human graders (HG). We calculated the incremental cost-effectiveness ratio (ICER) between the two strategies. Sensitivity analyses were performed to probe the influence of modeling parameters.

RESULTS

From a societal perspective, screening with DL was associated with a reduction in costs of ~ US$ 2.70, similar quality-adjusted life-years (QALY) of + 0.0043, and an incremental net monetary benefit of ~ US$ 24.10 in the base case. In sensitivity analysis, DL remained cost-effective even with a price increase from US$ 1.00 to US$ 4.00 per patient at a Thai willingness-to-pay threshold of ~ US$ 4.997 per QALY gained. When further incorporating recent findings suggesting improved compliance to treatment referral with DL, our analysis models effectiveness benefits of ~ US$ 20 to US$ 50 depending on compliance.

CONCLUSION

DR screening using DL in an MIC using Thailand as a model may result in societal cost-savings and similar health outcomes compared with HG. This study may provide an economic rationale to expand DL-based DR screening in MICs as an alternative solution for limited availability of skilled human resources for primary screening, particularly in MICs with similar prevalence of diabetes and low compliance to referrals for treatment.

Effectiveness of artificial intelligence screening in preventing vision loss from diabetes: a policy model

The effectiveness of using artificial intelligence (AI) systems to perform diabetic retinal exams ('screening') on preventing vision loss is not known. We designed the Care Process for Preventing Vision Loss from Diabetes (CAREVL), as a Markov model to compare the effectiveness of point-of-care autonomous AI-based screening with in-office clinical exam by an eye care provider (ECP), on preventing vision loss among patients with diabetes. The estimated incidence of vision loss at 5 years was 1535 per 100,000 in the AI-screened group compared to 1625 per 100,000 in the ECP group, leading to a modelled risk difference of 90 per 100,000. The base-case CAREVL model estimated that an autonomous AI-based screening strategy would result in 27,000 fewer Americans with vision loss at 5 years compared with ECP. Vision loss at 5 years remained lower in the AI-screened group compared to the ECP group, in a wide range of parameters including optimistic estimates biased toward ECP. Real-world modifiable factors associated with processes of care could further increase its effectiveness. Of these factors, increased adherence with treatment was estimated to have the greatest impact.

Artificial intelligence-assisted smartphone-based sensing for bioanalytical applications: A review

Artificial intelligence (AI) has received great attention since the concept was proposed, and it has developed rapidly in recent years with applications in many fields. Meanwhile, newer iterations of smartphone hardware technologies which have excellent data processing capabilities have leveraged on AI capabilities. Based on the desirability for portable detection, researchers have been investigating intelligent analysis by combining smartphones with AI algorithms. Various examples of the application of AI algorithm-based smartphone detection and analysis have been developed. In this review, we give an overview of this field, with a particular focus on bioanalytical detection applications. The applications are presented in terms of hardware design, software algorithms, and specific application areas. We also discuss the existing limitations of AI-based smartphone detection and analytical approaches, and their future prospects. The take-home message of our review is that the application of AI in the field of detection analysis is restricted by the limitations of the smartphone's hardware as well as the model building of AI for detection targets with insufficient data. Nevertheless, at this juncture, while bioanalytical diagnostics and health monitoring have set the pace for AI-based smartphone applicability, the future should see the technology making greater inroads into other fields. In relation to the latter, it is likely that the ordinary or average person will play a greater participatory role.

Artificial Intelligence for Diabetic Retinopathy Screening Using Color Retinal Photographs: From Development to Deployment

Diabetic retinopathy (DR), a leading cause of preventable blindness, is expected to remain a growing health burden worldwide. Screening to detect early sight-threatening lesions of DR can reduce the burden of vision loss; nevertheless, the process requires intensive manual labor and extensive resources to accommodate the increasing number of patients with diabetes. Artificial intelligence (AI) has been shown to be an effective tool which can potentially lower the burden of screening DR and vision loss. In this article, we review the use of AI for DR screening on color retinal photographs in different phases of application, ranging from development to deployment. Early studies of machine learning (ML)-based algorithms using feature extraction to detect DR achieved a high sensitivity but relatively lower specificity. Robust sensitivity and specificity were achieved with the application of deep learning (DL), although ML is still used in some tasks. Public datasets were utilized in retrospective validations of the developmental phases in most algorithms, which require a large number of photographs. Large prospective clinical validation studies led to the approval of DL for autonomous screening of DR although the semi-autonomous approach may be preferable in some real-world settings. There have been few reports on real-world implementations of DL for DR screening. It is possible that AI may improve some real-world indicators for eye care in DR, such as increased screening uptake and referral adherence, but this has not been proven. The challenges in deployment may include workflow issues, such as mydriasis to lower ungradable cases; technical issues, such as integration into electronic health record systems and integration into existing camera systems; ethical issues, such as data privacy and security; acceptance of personnel and patients; and health-economic issues, such as the need to conduct health economic evaluations of using AI in the context of the country. The deployment of AI for DR screening should follow the governance model for AI in healthcare which outlines four main components: fairness, transparency, trustworthiness, and accountability.

Economic evaluation of combined population-based screening for multiple blindness-causing eye diseases in China: a cost-effectiveness analysis

BACKGROUND

More than 90% of vision impairment is avoidable. However, in China, a routine screening programme is currently unavailable in primary health care. With the dearth of economic evidence on screening programmes for multiple blindness-causing eye diseases, delivery options, and screening frequencies, we aimed to evaluate the costs and benefits of a population-based screening programme for multiple eye diseases in China.

METHODS

We developed a decision-analytic Markov model for a cohort of individuals aged 50 years and older with a total of 30 1-year cycles. We calculated the cost-effectiveness and cost-utility of screening programmes for multiple major blindness-causing eye diseases in China, including age-related macular degeneration, glaucoma, diabetic retinopathy, cataracts, and pathological myopia, from a societal perspective (including direct and indirect costs). We analysed rural and urban settings separately by different screening delivery options (non-telemedicine [ie, face-to-face] screening, artificial intelligence [AI] telemedicine screening, and non-AI telemedicine screening) and frequencies. We calculated incremental cost-utility ratios (ICURs) using quality-adjusted life-years and incremental cost-effectiveness ratios (ICERs) in terms of the cost per blindness year avoided. One-way deterministic and simulated probabilistic sensitivity analyses were used to assess the robustness of the main outcomes.

FINDINGS

Compared with no screening, non-telemedicine combined screening of multiple eye diseases satisfied the criterion for a highly cost-effective health intervention, with an ICUR of US$2494 (95% CI 1130 to 2716) and an ICER of $12 487 (8773 to 18 791) in rural settings. In urban areas, the ICUR was $624 (395 to 907), and the ICER was $7251 (4238 to 13 501). Non-AI telemedicine screening could result in fewer costs and greater gains in health benefits (ICUR $2326 [1064 to 2538] and ICER $11 766 [8200 to 18 000] in rural settings; ICUR $581 [368 to 864] and ICER $6920 [3926 to 13 231] in urban settings). AI telemedicine screening dominated no screening in rural settings, and in urban settings the ICUR was $244 (-315 to 1073) and the ICER was $2567 (-4111 to 15 389). Sensitivity analyses showed all results to be robust. By further comparison, annual AI telemedicine screening was the most cost-effective strategy in both rural and urban areas.

INTERPRETATION

Combined screening of multiple eye diseases is cost-effective in both rural and urban China. AI coupled with teleophthalmology presents an opportunity to promote equity in eye health.

FUNDING

National Natural Science Foundation of China.

Correlation Between Retinal Vessel Diameters and Uveitis Activity

Purpose

To assess retinal vein and artery diameters during active and inactive intraocular inflammation in eyes with uveitis.

Methods

Color fundus photographs and clinical data of eyes with uveitis collected during two visits (active disease [i.e., T0] and inactive stage [i.e., T1]) were reviewed. The images were semi-automatically analyzed to obtain the central retina vein equivalent (CRVE) and central retina artery equivalent (CRAE). Changes of CRVE and CRAE from T0 to T1 were calculated, and their possible correlation with clinical data, including age, gender, ethnicity, uveitis etiology, and visual acuity, were investigated.

Results

Eighty-nine eyes were enrolled in the study. Both CRVE and CRAE reduced from T0 to T1 (P < 0.0001 and P = 0.01, respectively), with active inflammation being able to influence the CRVE and CRAE (P < 0.0001 and P = 0.0004, respectively) after accounting for all other variables. The degree of venular (∆V) and arteriolar (∆A) dilation was influenced only by time (P = 0.03 and P = 0.04, respectively). Best-corrected visual acuity was influenced by time and ethnicity (P = 0.003 and P = 0.0006).

Conclusions

CRVE and CRAE are increased in eyes with active intraocular inflammation regardless of the type of uveitis, and they decrease when the inflammation wears off.

Artificial Intelligence in Community-Based Diabetic Retinopathy Telemedicine Screening in Urban China: Cost-effectiveness and Cost-Utility Analyses With Real-world Data

BACKGROUND

Community-based telemedicine screening for diabetic retinopathy (DR) has been highly recommended worldwide. However, evidence from low- and middle-income countries (LMICs) on the choice between artificial intelligence (AI)-based and manual grading-based telemedicine screening is inadequate for policy making.

OBJECTIVE

The aim of this study was to test whether the AI model is more worthwhile than manual grading in community-based telemedicine screening for DR in the context of labor costs in urban China.

METHODS

We conducted cost-effectiveness and cost-utility analyses by using decision-analytic Markov models with 30 one-year cycles from a societal perspective to compare the cost, effectiveness, and utility of 2 scenarios in telemedicine screening for DR: manual grading and an AI model. Sensitivity analyses were performed. Real-world data were obtained mainly from the Shanghai Digital Eye Disease Screening Program. The main outcomes were the incremental cost-effectiveness ratio (ICER) and the incremental cost-utility ratio (ICUR). The ICUR thresholds were set as 1 and 3 times the local gross domestic product per capita.

RESULTS

The total expected costs for a 65-year-old resident were US $3182.50 and US $3265.40, while the total expected years without blindness were 9.80 years and 9.83 years, and the utilities were 6.748 quality-adjusted life years (QALYs) and 6.753 QALYs in the AI model and manual grading, respectively. The ICER for the AI-assisted model was US $2553.39 per year without blindness, and the ICUR was US $15,216.96 per QALY, which indicated that AI-assisted model was not cost-effective. The sensitivity analysis suggested that if there is an increase in compliance with referrals after the adoption of AI by 7.5%, an increase in on-site screening costs in manual grading by 50%, or a decrease in on-site screening costs in the AI model by 50%, then the AI model could be the dominant strategy.

CONCLUSIONS

Our study may provide a reference for policy making in planning community-based telemedicine screening for DR in LMICs. Our findings indicate that unless the referral compliance of patients with suspected DR increases, the adoption of the AI model may not improve the value of telemedicine screening compared to that of manual grading in LMICs. The main reason is that in the context of the low labor costs in LMICs, the direct health care costs saved by replacing manual grading with AI are less, and the screening effectiveness (QALYs and years without blindness) decreases. Our study suggests that the magnitude of the value generated by this technology replacement depends primarily on 2 aspects. The first is the extent of direct health care costs reduced by AI, and the second is the change in health care service utilization caused by AI. Therefore, our research can also provide analytical ideas for other health care sectors in their decision to use AI.

Artificial Intelligence Software for Diabetic Eye Screening: Diagnostic Performance and Impact of Stratification

AIM

To evaluate the MONA.health artificial intelligence screening software for detecting referable diabetic retinopathy (DR) and diabetic macular edema (DME), including subgroup analysis.

METHODS

The algorithm's threshold value was fixed at the 90% sensitivity operating point on the receiver operating curve to perform the disease classification. Diagnostic performance was appraised on a private test set and publicly available datasets. Stratification analysis was executed on the private test set considering age, ethnicity, sex, insulin dependency, year of examination, camera type, image quality, and dilatation status.

RESULTS

The software displayed an area under the curve (AUC) of 97.28% for DR and 98.08% for DME on the private test set. The specificity and sensitivity for combined DR and DME predictions were 94.24 and 90.91%, respectively. The AUC ranged from 96.91 to 97.99% on the publicly available datasets for DR. AUC values were above 95% in all subgroups, with lower predictive values found for individuals above the age of 65 (82.51% sensitivity) and Caucasians (84.03% sensitivity).

CONCLUSION

We report good overall performance of the MONA.health screening software for DR and DME. The software performance remains stable with no significant deterioration of the deep learning models in any studied strata.