Longitudinal Screening for Diabetic Retinopathy in a Nationwide Screening Program: Comparing Deep Learning and Human Graders

Economic evaluation for medical artificial intelligence: accuracy vs. cost-effectiveness in a diabetic retinopathy screening case

Artificial intelligence (AI) models have shown great accuracy in health screening. However, for real-world implementation, high accuracy may not guarantee cost-effectiveness. Improving AI's sensitivity finds more high-risk patients but may raise medical costs while increasing specificity reduces unnecessary referrals but may weaken detection capability. To evaluate the trade-off between AI model performance and the long-running cost-effectiveness, we conducted a cost-effectiveness analysis in a nationwide diabetic retinopathy (DR) screening program in China, comprising 251,535 participants with diabetes over 30 years. We tested a validated AI model in 1100 different diagnostic performances (presented as sensitivity/specificity pairs) and modeled annual screening scenarios. The status quo was defined as the scenario with the most accurate AI performance. The incremental cost-effectiveness ratio (ICER) was calculated for other scenarios against the status quo as cost-effectiveness metrics. Compared to the status quo (sensitivity/specificity: 93.3%/87.7%), six scenarios were cost-saving and seven were cost-effective. To achieve cost-saving or cost-effective, the AI model should reach a minimum sensitivity of 88.2% and specificity of 80.4%. The most cost-effective AI model exhibited higher sensitivity (96.3%) and lower specificity (80.4%) than the status quo. In settings with higher DR prevalence and willingness-to-pay levels, the AI needed higher sensitivity for optimal cost-effectiveness. Urban regions and younger patient groups also required higher sensitivity in AI-based screening. In real-world DR screening, the most accurate AI model may not be the most cost-effective. Cost-effectiveness should be independently evaluated, which is most likely to be affected by the AI's sensitivity.

Artificial intelligence for diabetic retinopathy in low-income and middle-income countries: a scoping review

Diabetic retinopathy (DR) is a leading cause of blindness globally. There is growing evidence to support the use of artificial intelligence (AI) in diabetic eye care, particularly for screening populations at risk of sight loss from DR in low-income and middle-income countries (LMICs) where resources are most stretched. However, implementation into clinical practice remains limited. We conducted a scoping review to identify what AI tools have been used for DR in LMICs and to report their performance and relevant characteristics. 81 articles were included. The reported sensitivities and specificities were generally high providing evidence to support use in clinical practice. However, the majority of studies focused on sensitivity and specificity only and there was limited information on cost, regulatory approvals and whether the use of AI improved health outcomes. Further research that goes beyond reporting sensitivities and specificities is needed prior to wider implementation.

Variability in Grading Diabetic Retinopathy Using Retinal Photography and Its Comparison with an Automated Deep Learning Diabetic Retinopathy Screening Software

BACKGROUND

Diabetic retinopathy (DR) screening using colour retinal photographs is cost-effective and time-efficient. In real-world clinical settings, DR severity is frequently graded by individuals of different expertise levels. We aim to determine the agreement in DR severity grading between human graders of varying expertise and an automated deep learning DR screening software (ADLS).

METHODS

Using the International Clinical DR Disease Severity Scale, two hundred macula-centred fundus photographs were graded by retinal specialists, ophthalmology residents, family medicine physicians, medical students, and the ADLS. Based on referral urgency, referral grading was divided into no referral, non-urgent referral, and urgent referral to an ophthalmologist. Inter-observer and intra-group variations were analysed using Gwet's agreement coefficient, and the performance of ADLS was evaluated using sensitivity and specificity.

RESULTS

The agreement coefficient for inter-observer and intra-group variability ranged from fair to very good, and moderate to good, respectively. The ADLS showed a high area under curve of 0.879, 0.714, and 0.836 for non-referable DR, non-urgent referable DR, and urgent referable DR, respectively, with varying sensitivity and specificity values.

CONCLUSION

Inter-observer and intra-group agreements among human graders vary widely, but ADLS is a reliable and reasonably sensitive tool for mass screening to detect referable DR and urgent referable DR.

Cost-Utility Analysis of Deep Learning and Trained Human Graders for Diabetic Retinopathy Screening in a Nationwide Program

INTRODUCTION

Deep learning (DL) for screening diabetic retinopathy (DR) has the potential to address limited healthcare resources by enabling expanded access to healthcare. However, there is still limited health economic evaluation, particularly in low- and middle-income countries, on this subject to aid decision-making for DL adoption.

METHODS

In the context of a middle-income country (MIC), using Thailand as a model, we constructed a decision tree-Markov hybrid model to estimate lifetime costs and outcomes of Thailand's national DR screening program via DL and trained human graders (HG). We calculated the incremental cost-effectiveness ratio (ICER) between the two strategies. Sensitivity analyses were performed to probe the influence of modeling parameters.

RESULTS

From a societal perspective, screening with DL was associated with a reduction in costs of ~ US$ 2.70, similar quality-adjusted life-years (QALY) of + 0.0043, and an incremental net monetary benefit of ~ US$ 24.10 in the base case. In sensitivity analysis, DL remained cost-effective even with a price increase from US$ 1.00 to US$ 4.00 per patient at a Thai willingness-to-pay threshold of ~ US$ 4.997 per QALY gained. When further incorporating recent findings suggesting improved compliance to treatment referral with DL, our analysis models effectiveness benefits of ~ US$ 20 to US$ 50 depending on compliance.

CONCLUSION

DR screening using DL in an MIC using Thailand as a model may result in societal cost-savings and similar health outcomes compared with HG. This study may provide an economic rationale to expand DL-based DR screening in MICs as an alternative solution for limited availability of skilled human resources for primary screening, particularly in MICs with similar prevalence of diabetes and low compliance to referrals for treatment.

Calibrating a Transformer-Based Model's Confidence on Community-Engaged Research Studies: Decision Support Evaluation Study

BACKGROUND

Deep learning offers great benefits in classification tasks such as medical imaging diagnostics or stock trading, especially when compared with human-level performances, and can be a viable option for classifying distinct levels within community-engaged research (CEnR). CEnR is a collaborative approach between academics and community partners with the aim of conducting research that is relevant to community needs while incorporating diverse forms of expertise. In the field of deep learning and artificial intelligence (AI), training multiple models to obtain the highest validation accuracy is common practice; however, it can overfit toward that specific data set and not generalize well to a real-world population, which creates issues of bias and potentially dangerous algorithmic decisions. Consequently, if we plan on automating human decision-making, there is a need for creating techniques and exhaustive evaluative processes for these powerful unexplainable models to ensure that we do not incorporate and blindly trust poor AI models to make real-world decisions.

OBJECTIVE

We aimed to conduct an evaluation study to see whether our most accurate transformer-based models derived from previous studies could emulate our own classification spectrum for tracking CEnR studies as well as whether the use of calibrated confidence scores was meaningful.

METHODS

We compared the results from 3 domain experts, who classified a sample of 45 studies derived from our university's institutional review board database, with those from 3 previously trained transformer-based models, as well as investigated whether calibrated confidence scores can be a viable technique for using AI in a support role for complex decision-making systems.

RESULTS

Our findings reveal that certain models exhibit an overestimation of their performance through high confidence scores, despite not achieving the highest validation accuracy.

CONCLUSIONS

Future studies should be conducted with larger sample sizes to generalize the results more effectively. Although our study addresses the concerns of bias and overfitting in deep learning models, there is a need to further explore methods that allow domain experts to trust our models more. The use of a calibrated confidence score can be a misleading metric when determining our AI model's level of competency.

Artificial intelligence at the national eye institute

PURPOSE OF REVIEW

This review highlights the artificial intelligence, machine learning, and deep learning initiatives supported by the National Institutes of Health (NIH) and the National Eye Institute (NEI) and calls attention to activities and goals defined in the NEI Strategic Plan as well as opportunities for future activities and breakthroughs in ophthalmology.

RECENT FINDINGS

Ophthalmology is at the forefront of artificial intelligence-based innovations in biomedical research that may lead to improvement in early detection and surveillance of ocular disease, prediction of progression, and improved quality of life. Technological advances have ushered in an era where unprecedented amounts of information can be linked that enable scientific discovery. However, there remains an unmet need to collect, harmonize, and share data in a machine actionable manner. Similarly, there is a need to ensure that efforts promote health and research equity by expanding diversity in the data and workforce.

SUMMARY

The NIH/NEI has supported the development artificial intelligence-based innovations to advance biomedical research. The NIH/NEI has defined activities to achieve these goals in the NIH Strategic Plan for Data Science and the NEI Strategic Plan and have spearheaded initiatives to facilitate research in these areas.

An Effective Method for Detecting and Classifying Diabetic Retinopathy Lesions Based on Deep Learning

Diabetic retinopathy occurs as a result of the harmful effects of diabetes on the eyes. Diabetic retinopathy is also a disease that should be diagnosed early. If not treated early, vision loss may occur. It is estimated that one third of more than half a million diabetic patients will have diabetic retinopathy by the 22nd century. Many effective methods have been proposed for disease detection with deep learning. In this study, unlike other studies, a deep learning-based method has been proposed in which diabetic retinopathy lesions are detected automatically and independently of datasets, and the detected lesions are classified. In the first stage of the proposed method, a data pool is created by collecting diabetic retinopathy data from different datasets. With Faster RCNN, lesions are detected, and the region of interests are marked. The images obtained in the second stage are classified using the transfer learning and attention mechanism. The method tested in Kaggle and MESSIDOR datasets reached 99.1% and 100% ACC and 99.9% and 100% AUC, respectively. When the obtained results are compared with other results in the literature, it is seen that more successful results are obtained.

DsbA-L Ameliorates Renal Injury Through the AMPK/NLRP3 Inflammasome Signaling Pathway in Diabetic Nephropathy

NLRP3-mediated inflammation is closely related to the pathological progression of diabetic nephropathy (DN). DsbA-L, an antioxidant enzyme, plays a protective role in a variety of diseases by inhibiting ER stress and regulating metabolism. However, the relationship of DsbA-L with inflammation, especially the NLRP3 inflammasome, has not been examined. In this study, we note that activation of the NLRP3 inflammasome and exacerbated fibrosis were observed in the kidneys of diabetic DsbA-L-knockout mice and were accompanied by decreased phosphorylation of AMP-activated protein kinase (AMPK). Moreover, correlation analysis shows that the phosphorylation of AMPK was negatively correlated with NLRP3 expression and tubular damage. In addition, the decreased AMPK phosphorylation and NLRP3 activation induced by high glucose (HG) in HK-2 cells could be alleviated by the overexpression of DsbA-L. Interestingly, the protective effect of DsbA-L was eliminated after treatment with compound C, a well-known AMPK inhibitor. Our findings suggest that DsbA-L inhibits NLRP3 inflammasome activation by promoting the phosphorylation of AMPK.