- Open Access
Diabetic retinopathy classification for supervised machine learning algorithms
International Journal of Retina and Vitreous volume 8, Article number: 1 (2022)
Artificial intelligence and automated technology were first reported more than 70 years ago and nowadays provide unprecedented diagnostic accuracy, screening capacity, risk stratification, and workflow optimization.
Diabetic retinopathy is an important cause of preventable blindness worldwide, and artificial intelligence technology provides precocious diagnosis, monitoring, and guide treatment. High-quality exams are fundamental in supervised artificial intelligence algorithms, but the lack of ground truth standards in retinal exams datasets is a problem.
In this article, ETDRS, NHS, ICDR, SDGS diabetic retinopathy grading, and manual annotation are described and compared in publicly available datasets. The various DR labeling systems generate a fundamental problem for AI datasets. Possible solutions are standardization of DR classification and direct retinal-finding identifications.
Reliable labeling methods also need to be considered in datasets with more trustworthy labeling.
Computers executing automated functions were first described in 1950, with the first publication in 1943. Since then, Artificial Intelligence capacity has evolved into deep learning and neural networks, technologies that could simulate interconnected neurons and provide outputs after multiple information layers [1, 2].
Automated technology provides unprecedented diagnostic accuracy, screening capacity, risk stratification, and workflow optimization with accuracy equivalent to healthcare professionals  and more cost-effective diseases screening .
Diabetic retinopathy (DR) is the leading cause of preventable blindness in working-age adults worldwide [7, 8], responsible for more than 24,000 annual cases of blindness  and the main focus in Ophthalmological AI screening algorithms . There is an increased blindness risk in patients with chronic diabetes mellitus, especially those with poor clinical control .
Telemedicine and automated screening programs could diagnose, monitor, and guide treatment. Precocious diagnosis and therapy could avoid severe vision loss in 90% of cases, but only 60% of diabetic patients have recommended yearly examinations .
There are many Diabetic Retinopathy classifications applied in distinct countries and screening programs, with the International Council of Ophthalmology Diabetic Retinopathy (ICDR) classification as the most applied in open-access ophthalmological datasets .
High-quality retinal exams are fundamental in the development of AI algorithms, but also standards in labeling protocols, classifications, and quality control. This article describes and compares the most commonly diabetic retinopathy classifications, referencing criteria, and their applications in datasets.
This study compared the most often-applied DR classification scales: Scottish Diabetic Retinopathy Grading , Early Treatment Diabetic Retinopathy Grading , International Clinic Diabetic Retinopathy , National Health Service Diabetic Retinopathy Classification grading , Modified Davis Retinopathy staging , and direct findings identification.
The Early Treatment Diabetic Retinopathy Study
At an international consortium of ophthalmologists at Airlie House in 1968, internists and neurosurgeons standardized a diabetic retinopathy classification applied in the landmark Early Treatment Diabetic Retinopathy Study , designed to generate a more precise staging for DR and macular edema. The study screened for the presence of microaneurysms (MA), retinal hemorrhages, cotton-wool spots, intraretinal microvascular abnormalities (IRMA), venous beading, and neovessels in 35-mm photographs. The consortium provided standard photos of microaneurysms, hemorrhages, and neovessels.
The ETDRS defined microaneurysms as red spots of less than 125 microns in its longest dimension with well-delimited margins and defined hemorrhage as a red spot with irregular margins with more than 125 microns. Punctate lesions, blots, linear hemorrhages, and microaneurysms were classified as red spots when they were not distinguished in ETDRS charts .
ETDRS defined clinically significant macular edema as retinal edema seen in retinal stereo photographs at or within 500 microns of the center of the macula or hard exudates at or within 500 microns of the foveal center and retina thickening or retinal thickening larger than one disc diameter area within one disc diameter of the center of the macula. In 2006, Rudnisky compared modified ETDRS protocols with one or two fields and 16:1 JPEG images and showed good reproducibility compared to standard ETDRS stereoscopic photos . (Table 1).
National Health Service diabetic retinopathy classification
The National Health Service (NHS) was a diabetic retinopathy classification system applied In England, Scotland, Wales, and Northern Ireland between 2002 and 2007. It applied an ETDRS modified diabetic retinopathy scale classified in four severity stages [17, 21]. This program evaluated and classified DR using macula-centered and optic disc-centered images . The NHS screening program provided guidelines for grading and lesions classifications .
This DR classification considered macular exudates sign of macular edema because the images were non-stereoscopic; it also added a photocoagulation classification (Table 1).
International Clinic Diabetic Retinopathy
The International Clinic Diabetic Retinopathy (ICDR) classification was published in 2003 after a consensus of 31 retina specialists, endocrinologists, and epidemiologists from 16 countries and sponsored by the American Academy of Ophthalmology . The ICDR classified DR on a five-stage severity scale and classified diabetic macular edema as apparently absent or present. The classification was created to simplify the ETDR and Wisconsin Epidemiologic Study scale and make it more applicable in daily practice studies .
The Scottish Diabetic Retinopathy Grading Scheme, 2004
In 2003, the National Scotland Eye Screening for Diabetic Retinopathy Program was created . This grading system classified DR in all patients aged 12 years and older. Retinal digital photos were analyzed, and the re-screening period or ophthalmologist referral was established. The Scottish diabetic retinopathy grade (SDRG) is divided into four DR severities in a single fovea-centered image with at least two disc diameters temporal to the fovea and one disc diameter nasal to the disc  (Table 1).
Modified Davis retinopathy staging
The ICDR score simplifies DR in three stages: simple diabetic retinopathy, pre-proliferative retinopathy, and proliferative retinopathy using 45-degree photographs of the posterior pole applied in the Jichi DR dataset  (Table 1).
Direct findings identification
In AI datasets, findings such as microaneurysms, hemorrhages, hard exudates, and retinal detachment could be identified through direct identification. Applications such as SuperAnnotate , VGG Image annotation Tool , Supervise.ly , Labelbox , and Visual Object Tagging Tool  are available as labeling tools.
Referencing criteria comparison
The NHS, ICDR, and SDRGS establish referencing criteria. In NHS and SDRGS, the criteria are similar, with multiple retinal hemorrhages, intraretinal microvascular anomalies, or venous beading. In the ICDR, should be referenced patients with more than just microaneurysm, a criterion with greater sensitivity [14, 16, 17].
Considering macular edema, the NHS, SDRGS, and ICDR recommend referencing patients with exudates or apparent thickening in the macular area. The NHS recommends exudates distance within half-disc diameter from the fovea and ICDR and SDRGS within one disc diameter [14, 16, 17] (Table 1).
Artificial intelligence and automated technology were first reported more than 70 years ago and nowadays provide unprecedented diagnostic accuracy, screening, risk stratification, and workflow optimization .
Reliable datasets are fundamental in supervised Machine Learning development; however, labeling process standardization, quality control, and homogenization remain challenging .
In diabetic retinopathy, there are distinct DR classifications, with different numbers of DR gradings and methods such as the Scottish Diabetic Retinopathy Grading , Early Treatment Diabetic Retinopathy Grading , ICDR , NHS Diabetic Retinopathy Classification grading , and Modified Davis Retinopathy staging  that are described in this review. Still, direct retinal findings annotation is valuable in neural networks training.
The Scottish Diabetic Retinopathy Grading is a valuable classification through retinal photographs due to a single macular centered retinal evaluation and is more sensitive for grading moderate and severe cases than ICDR classification.
When choosing the classification method applied in the dataset, the image field of view and the number of images must be considered. Classical ETDRS and ICDR classifications tend to underestimate DR classification in retinal photographic images due to limited image view areas compared to retinal fundus examinations.
The various DR labeling systems generate a fundamental problem for AI datasets, and it is fundamental to standardize DR grading in datasets to develop algorithms and ensure proper patient referral. Reliable labeling methods also need to be considered in datasets with more trustworthy labeling.
Availability of data and materials
Early Treatment Diabetic Retinopathy Grading
National Health Service Diabetic Retinopathy Classification grading
International Council of Ophthalmology Diabetic Retinopathy
Scottish Diabetic Retinopathy Grading
Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc. 2020;92:807–12. https://doi.org/10.1016/j.gie.2020.06.040.
Muthukrishnan N, Maleki F, Ovens K, Reinhold C, Forghani B, Forghani R. Brief history of artificial intelligence. Neuroimaging Clin N Am. 2020;30:393–9. https://doi.org/10.1016/j.nic.2020.07.004.
Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digital Health. 2019;1:e271–97. https://doi.org/10.1016/s2589-7500(19)30123-2.
Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018;1:39. https://doi.org/10.1038/s41746-018-0040-6.
Khan SM, Liu X, Nath S, Korot E, Faes L, Wagner SK, et al. A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability. Lancet Digital Health. 2020. https://doi.org/10.1016/S2589-7500(20)30240-5.
Lin W-C, Chen JS, Chiang MF, Hribar MR. Applications of artificial intelligence to electronic health record data in ophthalmology. Transl Vis Sci Technol. 2020;9:13. https://doi.org/10.1167/tvst.9.2.13.
Ophthalmology IC. Updated 2017 ICO guidelines for diabetic eye care. 2016.
Sadda SR. Assessing the severity of diabetic retinopathy: early treatment diabetic retinopathy study report number 10. Ophthalmology. 2020;127:S97–8. https://doi.org/10.1016/j.ophtha.2019.11.028.
Md A, Abramoff Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. Yearb Paediatr Endocrinol. 2019. https://doi.org/10.1530/ey.16.12.1.
Kras A, Celi LA, Miller JB. Accelerating ophthalmic artificial intelligence research: the role of an open access data repository. Curr Opin Ophthalmol. 2020;31:337–50. https://doi.org/10.1097/ICU.0000000000000678.
Antonetti DA, Klein R, Gardner TW. Diabetic retinopathy. N Engl J Med. 2012;366:1227–39. https://doi.org/10.1056/NEJMra1005073.
Flaxel CJ, Adelman RA, Bailey ST, Fawzi A, Lim JI, Vemulakonda GA, et al. Diabetic retinopathy preferred practice pattern®. Ophthalmology. 2020;127:P66-145. https://doi.org/10.1016/j.ophtha.2019.09.025.
Nakayama LF, Gonçalves MB, Ferraz DA, Santos HNV, Malerbi FK, Morales PH, et al. The Challenge of Diabetic Retinopathy Standardization in an Ophthalmological Dataset. J Diabetes Sci Technol. 2021. https://doi.org/10.1177/19322968211029943.
Zachariah S, Wykes W, Yorston D. Grading diabetic retinopathy (DR) using the Scottish grading protocol. Community Eye Health. 2015;28:72–3. https://www.ncbi.nlm.nih.gov/pubmed/27418727. Accessed 6 Sept 2021.
Solomon SD, Goldberg MF. ETDRS grading of diabetic retinopathy: still the gold standard? Ophthalmic Res. 2019;62:190–5. https://doi.org/10.1159/000501372.
Wilkinson CP, Ferris FL 3rd, Klein RE, Lee PP, Agardh CD, Davis M, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110:1677–82. https://doi.org/10.1016/S0161-6420(03)00475-5.
Scanlon PH. The english national screening programme for diabetic retinopathy 2003–2016. Acta Diabetol. 2017;54:515–25. https://doi.org/10.1007/s00592-017-0974-1.
Takahashi H, Tampo H, Arai Y, Inoue Y, Kawashima H. Applying artificial intelligence to disease staging: deep learning for improved staging of diabetic retinopathy. PLoS ONE. 2017;12: e0179790. https://doi.org/10.1371/journal.pone.0179790.
ETDRSR Group. Grading diabetic retinopathy from stereoscopic color fundus photographs—an extension of the modified airlie house classification: ETDRS report number 10. Ophthalmology. 1991;98(5):786–806. https://doi.org/10.1016/S0161-6420(13)38012-9.
Rudnisky CJ, Tennant MTS, Weis E, Ting A, Hinz BJ, Greve MDJ. Web-based grading of compressed stereoscopic digital photography versus standard slide film photography for the diagnosis of diabetic retinopathy. Ophthalmology. 2007;114:1748–54. https://doi.org/10.1016/j.ophtha.2006.12.010.
Peate I. The NHS diabetic eye screening programme. Br J Healthc Assist. 2019;13:596–9. https://doi.org/10.12968/bjha.2019.13.12.596.
Diabetic eye screening: guidance when adequate images cannot be taken. 2021. https://www.gov.uk/government/publications/diabetic-eye-screening-pathway-for-images-and-where-images-cannot-be-taken/diabetic-eye-screening-guidance-when-adequate-images-cannot-be-taken. Accessed 6 Dec 2021.
England PH. NHS Diabetic Eye Screening Programme grading definitions for referable disease. 2017. https://www.gov.uk/government/publications/diabetic-eye-screening-retinal-image-grading-criteria/nhs-diabetic-eye-screening-programme-grading-definitions-for-referable-disease. Accessed 6 Sept 2021.
Korot E, Guan Z, Ferraz D, Wagner SK, Zhang G, Liu X, et al. Code-free deep learning for multi-modality medical image classification. Nat Mach Intell. 2021;3:288–98. https://doi.org/10.1038/s42256-021-00305-2.
Khalifa NEM, Loey M, Taha MHN, Mohamed HNET. Deep transfer learning models for medical diabetic retinopathy detection. Acta Inform Med. 2019;27:327–32. https://doi.org/10.5455/aim.2019.27.327-332.
Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, et al. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data. 2018;3:25. https://doi.org/10.3390/data3030025.
Decencièreb E, Zhang X, Cazuguel G, Laÿ B, Cochener B, Trone C, et al. Feedback on a publicly distributed image database: the Messidor Database. Image Anal Stereol. 2014. https://doi.org/10.5566/ias.1155.
Diabetic retinopathy screening standards. 2021. https://www.healthcareimprovementscotland.org/our_work/standards_and_guidelines/stnds/diabetic_retinopathy_screening.aspx. Accessed 6 Sep 2021.
SuperAnnotate. 2020. https://superannotate.com/. Accessed 15 Jun 2021.
Visual geometry group—University of oxford. 2021. https://www.robots.ox.ac.uk/~vgg/software/via/. Accessed 15 Jun 2021.
Supervisely—Web platform for computer vision. Annotation, training and deploy. 2021. https://supervise.ly/. Accessed 15 Jun 2021.
Labelbox: The leading training data platform for data labeling. 2021. https://labelbox.com/. Accessed 15 Jun 2021.
Visual object tagging tool (VoTT) v22.214.171.1241. https://vott.z22.web.core.windows.net/. Accessed 15 Jun 2021.
Islam MT, Imran SA, Arefeen A, Hasan M, Shahnaz C. Source and camera independent ophthalmic disease recognition from fundus image using neural network. In: 2019 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON). 2019. Doi:https://doi.org/10.1109/spicscon48833.2019.9065162.
Kauppi T, Kalesnykiene V, Kamarainen J-K, Lensu L, Sorri I, Raninen A, et al. The DIARETDB1 diabetic retinopathy database and evaluation protocol. In: Procedings of the British Machine Vision Conference 2007. 2007. doi:https://doi.org/10.5244/c.21.15.
Pires R, Jelinek HF, Wainer J, Valle E, Rocha A. Advancing bag-of-visual-words representations for lesion classification in retinal images. 2014. PLoS ONE. https://doi.org/10.6084/m9.figshare.953671.v1.
Decencière E, Cazuguel G, Zhang X, Thibault G, Klein J-C, Meyer F, et al. TeleOphta: machine learning and image processing methods for teleophthalmology. IRBM. 2013;34:196–203. https://doi.org/10.1016/j.irbm.2013.01.010.
Giancardo L, Meriaudeau F, Karnowski TP, Li Y, Garg S, Tobin KW Jr, et al. Exudate-based diabetic macular edema detection in fundus images using publicly available datasets. Med Image Anal. 2012;16:216–26. https://doi.org/10.1016/j.media.2011.07.004.
Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103:167–75. https://doi.org/10.1136/bjophthalmol-2018-313173.
We would like to thank Ophthalmology sectors from Escola Paulista de Medicina/ São Paulo federal University and Instituto Paulista de Estudos e Pesquisas em Oftalmologia, IPEPO, Vision Institute, São Paulo (SP), Brazil.
M. B. Gonçalves is a researcher supported by Lemann Foundation, Instituto da Visão-IPEPO, São Paulo, Brazil and CAPES Foundation, Ministry of Education of Brazil, Brasília, DF, Brazil.
Ethics approval and consent to participate
UNIFESP Ethics Institutional Review Board Number: CAAE 33842220.7.0000.5505/n:0698/2020.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Nakayama, L.F., Ribeiro, L.Z., Gonçalves, M.B. et al. Diabetic retinopathy classification for supervised machine learning algorithms. Int J Retin Vitr 8, 1 (2022). https://doi.org/10.1186/s40942-021-00352-2
- Diabetic retinopathy classifications
- Artificial intelligence