A pilot study of autonomous artificial intelligence‐based diabetic retinopathy screening in Poland

To date, there have been a number of diabetic retinopathy (DR) screening initiatives introduced all over the world (Lim et al. 2008; Pieczynski & Grzybowski 2015), including a few organized at a national level (most notably in the UK and Singapore) (Abràmoff et al. 2000; Lim et al. 2008). However, more often than not, these projects have been more localized mostly as a result of the associated cost, both in financial and human resources terms. With the recent advent of commercially available automatic DR screening systems, we decided to attempt a pilot programme using an autonomous artificial intelligence (AI)-based DR system (IDx Technologies Inc., Coralville, IA, USA) for screening purposes in Poland. The programme began in May 2018 on a local scale in order to assess its effectiveness and deployment method and to locate and mitigate any issues that might arise prior to attempting a larger-scale operation. We chose to set-up our pilot programme involving the at-risk population of a local, large diabetic clinic, as patients already visit the clinic in question for both routine and non-routine appointments. Additionally, such patients may constitute a population with more advanced stages of diabetes and as a result a higher likelihood for DR. Patients who visited the clinic for their appointments were offered DR screening based on non-mydriatic retinal images (TRC-NW400; Topcon, Tokyo, Japan) on an opt-in basis. Images were captured by diabetic nurses already employed within the clinic according to a standardized imaging protocol with one disc- and one fovea-centred 45° image per eye. These images were then submitted to the AI system for automated image quality evaluation. If the AI system detected sufficient quality, its output results in the form of a ruling of either more than mild (mtm) DR not detected, mtm DR detected or vision-threatening DR detected. Based on the AI system output of negative [i.e. no DR or only mild DR and no diabetic macular oedema (DME)], mtm DR or vision-threatening retinopathy (severe non-proliferative DR, proliferative DR and/or DME), the patient was recommended to undergo another screening after 1 year or routine or urgent ophthalmology referral, respectively. Patients whose image quality was deemed insufficient by IDx-DR system after reimaging were advised to visit an ophthalmologist. Patients known to have DR were excluded from the screening and reminded of the need for regular ophthalmology follow-up. For legal purposes, only patients above the age of 18 years were allowed to participate in the programme. The images were later reviewed by an ophthalmologist, and if incongruity with the IDx-DR results was identified, patients were notified at their next visit (if false-positive) or by phone (if false-negative). In reviewing the results given by the automatic system, we decided to take a pragmatic approach. The analysis software had previously received a class IIa certification in Europe as well as United States Food and Drug Administration (FDA) authorization and may be used as an autonomous screening device, meaning that the results may be relied on for a referral decision without review by a clinician. Additionally, when starting the programme, we did not have any certainty regarding its future beyond the initial year. As we could not guarantee that patients would be screened again the following year using the same system, in reviewing the images, we chose to adopt a lower threshold for referral (and therefore a positive result) than that of the IDx-DR specified cut-off of ‘more than mild DR’. Furthermore, we chose to adopt the following criteria for grading the images: threshold for DR was defined as more than three microaneurysms or any of either intraretinal microvascular abnormalities, venous beading, intraretinal haemorrhage or cotton wool spots. Additionally, when retinal changes unrelated to DR but requiring a fundus examination by an ophthalmologist were marked positive, we did not discount these as false-positives for the purpose of this screening. Some examples of such include extensive macular drusen or pigmented choroidal lesions (suspected choroidal nevi). While the screening programme is still ongoing at this time, we have gathered and preliminarily analysed the data from 450 screening episodes so far. The IDx-DR system was able to analyse 78% of screening episodes without pharmacological dilation, judging them to be of sufficient quality, in comparison with the achievement of 82% using a human grader. Out of 278 patients judged as having no DR by the IDx-DR system, only four were subsequently found to have DR, with none having more than moderate DR or DME. Separately, out of the 76 patients for whom the screening results were deemed positive by the AI, 14 were false-positives according to the clinician review. Overall, according to the single clinician reference standard, the sensitivity and specificity of the system were 94% and 95%, respectively, and the positive predictive and negative predictive values were 82% and 99%, respectively. Our IDx-DR accuracy results are higher than those previously reported in literature, particularly for specificity: for example, the IDx-DR system was found to have a specificity of 90.7% in the FDA pivotal clinical trial as compared with the true reference standard, Early treatment of diabetic retinopathy study (ETDRS) level grading based on four wide-field stereo fundus photographs and optical coherence tomography (Abràmoff et al. 2016; van der Heijden et al. 2018). Of note, the DR screening process using the IDx-DR system for the image analysis was easy to implement and use. Within 1 min, results were available, which were then automatically saved. Possible considerations for future include a cost-effectiveness analysis, additional data acquisition regarding patients’ DR risk factors and expanding the programme beyond the high-risk population of a single diabetes clinic and into a general practice-based or standalone screening programme. Overall, we are very pleased with the results so far and look forward to expanding and improving upon the current programme at a larger scale.