Skip to main content

Diabetic retinopathy classification for supervised machine learning algorithms

Abstract

Background

Artificial intelligence and automated technology were first reported more than 70 years ago and nowadays provide unprecedented diagnostic accuracy, screening capacity, risk stratification, and workflow optimization.

Diabetic retinopathy is an important cause of preventable blindness worldwide, and artificial intelligence technology provides precocious diagnosis, monitoring, and guide treatment. High-quality exams are fundamental in supervised artificial intelligence algorithms, but the lack of ground truth standards in retinal exams datasets is a problem.

Main body

In this article, ETDRS, NHS, ICDR, SDGS diabetic retinopathy grading, and manual annotation are described and compared in publicly available datasets. The various DR labeling systems generate a fundamental problem for AI datasets. Possible solutions are standardization of DR classification and direct retinal-finding identifications.

Conclusion

Reliable labeling methods also need to be considered in datasets with more trustworthy labeling.

Background

Computers executing automated functions were first described in 1950, with the first publication in 1943. Since then, Artificial Intelligence capacity has evolved into deep learning and neural networks, technologies that could simulate interconnected neurons and provide outputs after multiple information layers [1, 2].

Automated technology provides unprecedented diagnostic accuracy, screening capacity, risk stratification, and workflow optimization with accuracy equivalent to healthcare professionals [3] and more cost-effective diseases screening [4].

In Machine Learning, supervised learning is the most applied method in disease screening and classification algorithms, corroborating the importance of data labeling quality [5, 6].

Diabetic retinopathy (DR) is the leading cause of preventable blindness in working-age adults worldwide [7, 8], responsible for more than 24,000 annual cases of blindness [9] and the main focus in Ophthalmological AI screening algorithms [10]. There is an increased blindness risk in patients with chronic diabetes mellitus, especially those with poor clinical control [11].

Telemedicine and automated screening programs could diagnose, monitor, and guide treatment. Precocious diagnosis and therapy could avoid severe vision loss in 90% of cases, but only 60% of diabetic patients have recommended yearly examinations [12].

There are many Diabetic Retinopathy classifications applied in distinct countries and screening programs, with the International Council of Ophthalmology Diabetic Retinopathy (ICDR) classification as the most applied in open-access ophthalmological datasets [13].

High-quality retinal exams are fundamental in the development of AI algorithms, but also standards in labeling protocols, classifications, and quality control. This article describes and compares the most commonly diabetic retinopathy classifications, referencing criteria, and their applications in datasets.

Main text

This study compared the most often-applied DR classification scales: Scottish Diabetic Retinopathy Grading [14], Early Treatment Diabetic Retinopathy Grading [15], International Clinic Diabetic Retinopathy [16], National Health Service Diabetic Retinopathy Classification grading [17], Modified Davis Retinopathy staging [18], and direct findings identification.

The Early Treatment Diabetic Retinopathy Study

At an international consortium of ophthalmologists at Airlie House in 1968, internists and neurosurgeons standardized a diabetic retinopathy classification applied in the landmark Early Treatment Diabetic Retinopathy Study [15], designed to generate a more precise staging for DR and macular edema. The study screened for the presence of microaneurysms (MA), retinal hemorrhages, cotton-wool spots, intraretinal microvascular abnormalities (IRMA), venous beading, and neovessels in 35-mm photographs. The consortium provided standard photos of microaneurysms, hemorrhages, and neovessels.

The ETDRS defined microaneurysms as red spots of less than 125 microns in its longest dimension with well-delimited margins and defined hemorrhage as a red spot with irregular margins with more than 125 microns. Punctate lesions, blots, linear hemorrhages, and microaneurysms were classified as red spots when they were not distinguished in ETDRS charts [19].

ETDRS defined clinically significant macular edema as retinal edema seen in retinal stereo photographs at or within 500 microns of the center of the macula or hard exudates at or within 500 microns of the foveal center and retina thickening or retinal thickening larger than one disc diameter area within one disc diameter of the center of the macula. In 2006, Rudnisky compared modified ETDRS protocols with one or two fields and 16:1 JPEG images and showed good reproducibility compared to standard ETDRS stereoscopic photos [20]. (Table 1).

Table 1 Comparison of ETDRS, NHS, ICDR, SDRGS, Modified Davis diabetic retinopathy scales

National Health Service diabetic retinopathy classification

The National Health Service (NHS) was a diabetic retinopathy classification system applied In England, Scotland, Wales, and Northern Ireland between 2002 and 2007. It applied an ETDRS modified diabetic retinopathy scale classified in four severity stages [17, 21]. This program evaluated and classified DR using macula-centered and optic disc-centered images [22]. The NHS screening program provided guidelines for grading and lesions classifications [23].

This DR classification considered macular exudates sign of macular edema because the images were non-stereoscopic; it also added a photocoagulation classification (Table 1).

International Clinic Diabetic Retinopathy

The International Clinic Diabetic Retinopathy (ICDR) classification was published in 2003 after a consensus of 31 retina specialists, endocrinologists, and epidemiologists from 16 countries and sponsored by the American Academy of Ophthalmology [16]. The ICDR classified DR on a five-stage severity scale and classified diabetic macular edema as apparently absent or present. The classification was created to simplify the ETDR and Wisconsin Epidemiologic Study scale and make it more applicable in daily practice studies [16].

ICDR is applied in the EYEPACS dataset [24], Asian Pacific Tele-Ophthalmology Society dataset [25], Indian Diabetic Retinopathy Image Dataset [26], Messidor 1 and 2 datasets [27] (Table 1).

The Scottish Diabetic Retinopathy Grading Scheme, 2004

In 2003, the National Scotland Eye Screening for Diabetic Retinopathy Program was created [28]. This grading system classified DR in all patients aged 12 years and older. Retinal digital photos were analyzed, and the re-screening period or ophthalmologist referral was established. The Scottish diabetic retinopathy grade (SDRG) is divided into four DR severities in a single fovea-centered image with at least two disc diameters temporal to the fovea and one disc diameter nasal to the disc [14] (Table 1).

Modified Davis retinopathy staging

The ICDR score simplifies DR in three stages: simple diabetic retinopathy, pre-proliferative retinopathy, and proliferative retinopathy using 45-degree photographs of the posterior pole applied in the Jichi DR dataset [18] (Table 1).

Direct findings identification

In AI datasets, findings such as microaneurysms, hemorrhages, hard exudates, and retinal detachment could be identified through direct identification. Applications such as SuperAnnotate [29], VGG Image annotation Tool [30], Supervise.ly [31], Labelbox [32], and Visual Object Tagging Tool [33] are available as labeling tools.

In ODIR [34], DIARETDB 0 and 1 [35], DR 1 and 2 [36], E-Ophtha [37], and HEI-MED [38], retinal findings are manually annotated (Fig. 1).

Fig. 1
figure 1

Direct retinal findings manual annotation example, in Labelbox software

Referencing criteria comparison

The NHS, ICDR, and SDRGS establish referencing criteria. In NHS and SDRGS, the criteria are similar, with multiple retinal hemorrhages, intraretinal microvascular anomalies, or venous beading. In the ICDR, should be referenced patients with more than just microaneurysm, a criterion with greater sensitivity [14, 16, 17].

Considering macular edema, the NHS, SDRGS, and ICDR recommend referencing patients with exudates or apparent thickening in the macular area. The NHS recommends exudates distance within half-disc diameter from the fovea and ICDR and SDRGS within one disc diameter [14, 16, 17] (Table 1).

Conclusions

Artificial intelligence and automated technology were first reported more than 70 years ago and nowadays provide unprecedented diagnostic accuracy, screening, risk stratification, and workflow optimization [3].

Reliable datasets are fundamental in supervised Machine Learning development; however, labeling process standardization, quality control, and homogenization remain challenging [39].

In diabetic retinopathy, there are distinct DR classifications, with different numbers of DR gradings and methods such as the Scottish Diabetic Retinopathy Grading [14], Early Treatment Diabetic Retinopathy Grading [15], ICDR [16], NHS Diabetic Retinopathy Classification grading [17], and Modified Davis Retinopathy staging [18] that are described in this review. Still, direct retinal findings annotation is valuable in neural networks training.

The Scottish Diabetic Retinopathy Grading is a valuable classification through retinal photographs due to a single macular centered retinal evaluation and is more sensitive for grading moderate and severe cases than ICDR classification.

When choosing the classification method applied in the dataset, the image field of view and the number of images must be considered. Classical ETDRS and ICDR classifications tend to underestimate DR classification in retinal photographic images due to limited image view areas compared to retinal fundus examinations.

The various DR labeling systems generate a fundamental problem for AI datasets, and it is fundamental to standardize DR grading in datasets to develop algorithms and ensure proper patient referral. Reliable labeling methods also need to be considered in datasets with more trustworthy labeling.

Availability of data and materials

Not applicable.

Abbreviations

ETDRS:

Early Treatment Diabetic Retinopathy Grading

NHS:

National Health Service Diabetic Retinopathy Classification grading

ICDR:

International Council of Ophthalmology Diabetic Retinopathy

SDRGS:

Scottish Diabetic Retinopathy Grading

AI:

Artificial intelligence

References

  1. 1.

    Kaul V, Enslin S, Gross SA. History of artificial intelligence in medicine. Gastrointest Endosc. 2020;92:807–12. https://doi.org/10.1016/j.gie.2020.06.040.

    Article  PubMed  Google Scholar 

  2. 2.

    Muthukrishnan N, Maleki F, Ovens K, Reinhold C, Forghani B, Forghani R. Brief history of artificial intelligence. Neuroimaging Clin N Am. 2020;30:393–9. https://doi.org/10.1016/j.nic.2020.07.004.

    Article  PubMed  Google Scholar 

  3. 3.

    Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digital Health. 2019;1:e271–97. https://doi.org/10.1016/s2589-7500(19)30123-2.

    Article  PubMed  Google Scholar 

  4. 4.

    Abràmoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. NPJ Digit Med. 2018;1:39. https://doi.org/10.1038/s41746-018-0040-6.

    Article  PubMed  PubMed Central  Google Scholar 

  5. 5.

    Khan SM, Liu X, Nath S, Korot E, Faes L, Wagner SK, et al. A global review of publicly available datasets for ophthalmological imaging: barriers to access, usability, and generalisability. Lancet Digital Health. 2020. https://doi.org/10.1016/S2589-7500(20)30240-5.

    Article  PubMed  Google Scholar 

  6. 6.

    Lin W-C, Chen JS, Chiang MF, Hribar MR. Applications of artificial intelligence to electronic health record data in ophthalmology. Transl Vis Sci Technol. 2020;9:13. https://doi.org/10.1167/tvst.9.2.13.

    Article  PubMed  PubMed Central  Google Scholar 

  7. 7.

    Ophthalmology IC. Updated 2017 ICO guidelines for diabetic eye care. 2016.

  8. 8.

    Sadda SR. Assessing the severity of diabetic retinopathy: early treatment diabetic retinopathy study report number 10. Ophthalmology. 2020;127:S97–8. https://doi.org/10.1016/j.ophtha.2019.11.028.

    Article  PubMed  Google Scholar 

  9. 9.

    Md A, Abramoff Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. Yearb Paediatr Endocrinol. 2019. https://doi.org/10.1530/ey.16.12.1.

    Article  Google Scholar 

  10. 10.

    Kras A, Celi LA, Miller JB. Accelerating ophthalmic artificial intelligence research: the role of an open access data repository. Curr Opin Ophthalmol. 2020;31:337–50. https://doi.org/10.1097/ICU.0000000000000678.

    Article  PubMed  PubMed Central  Google Scholar 

  11. 11.

    Antonetti DA, Klein R, Gardner TW. Diabetic retinopathy. N Engl J Med. 2012;366:1227–39. https://doi.org/10.1056/NEJMra1005073.

    CAS  Article  PubMed  Google Scholar 

  12. 12.

    Flaxel CJ, Adelman RA, Bailey ST, Fawzi A, Lim JI, Vemulakonda GA, et al. Diabetic retinopathy preferred practice pattern®. Ophthalmology. 2020;127:P66-145. https://doi.org/10.1016/j.ophtha.2019.09.025.

    Article  PubMed  Google Scholar 

  13. 13.

    Nakayama LF, Gonçalves MB, Ferraz DA, Santos HNV, Malerbi FK, Morales PH, et al. The Challenge of Diabetic Retinopathy Standardization in an Ophthalmological Dataset. J Diabetes Sci Technol. 2021. https://doi.org/10.1177/19322968211029943.

    Article  PubMed  Google Scholar 

  14. 14.

    Zachariah S, Wykes W, Yorston D. Grading diabetic retinopathy (DR) using the Scottish grading protocol. Community Eye Health. 2015;28:72–3. https://www.ncbi.nlm.nih.gov/pubmed/27418727. Accessed 6 Sept 2021.

  15. 15.

    Solomon SD, Goldberg MF. ETDRS grading of diabetic retinopathy: still the gold standard? Ophthalmic Res. 2019;62:190–5. https://doi.org/10.1159/000501372.

    Article  PubMed  Google Scholar 

  16. 16.

    Wilkinson CP, Ferris FL 3rd, Klein RE, Lee PP, Agardh CD, Davis M, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology. 2003;110:1677–82. https://doi.org/10.1016/S0161-6420(03)00475-5.

    CAS  Article  PubMed  Google Scholar 

  17. 17.

    Scanlon PH. The english national screening programme for diabetic retinopathy 2003–2016. Acta Diabetol. 2017;54:515–25. https://doi.org/10.1007/s00592-017-0974-1.

    Article  PubMed  PubMed Central  Google Scholar 

  18. 18.

    Takahashi H, Tampo H, Arai Y, Inoue Y, Kawashima H. Applying artificial intelligence to disease staging: deep learning for improved staging of diabetic retinopathy. PLoS ONE. 2017;12: e0179790. https://doi.org/10.1371/journal.pone.0179790.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  19. 19

    ETDRSR Group. Grading diabetic retinopathy from stereoscopic color fundus photographs—an extension of the modified airlie house classification: ETDRS report number 10. Ophthalmology. 1991;98(5):786–806. https://doi.org/10.1016/S0161-6420(13)38012-9.

    Article  Google Scholar 

  20. 20.

    Rudnisky CJ, Tennant MTS, Weis E, Ting A, Hinz BJ, Greve MDJ. Web-based grading of compressed stereoscopic digital photography versus standard slide film photography for the diagnosis of diabetic retinopathy. Ophthalmology. 2007;114:1748–54. https://doi.org/10.1016/j.ophtha.2006.12.010.

    Article  PubMed  Google Scholar 

  21. 21.

    Peate I. The NHS diabetic eye screening programme. Br J Healthc Assist. 2019;13:596–9. https://doi.org/10.12968/bjha.2019.13.12.596.

    Article  Google Scholar 

  22. 22.

    Diabetic eye screening: guidance when adequate images cannot be taken. 2021. https://www.gov.uk/government/publications/diabetic-eye-screening-pathway-for-images-and-where-images-cannot-be-taken/diabetic-eye-screening-guidance-when-adequate-images-cannot-be-taken. Accessed 6 Dec 2021.

  23. 23.

    England PH. NHS Diabetic Eye Screening Programme grading definitions for referable disease. 2017. https://www.gov.uk/government/publications/diabetic-eye-screening-retinal-image-grading-criteria/nhs-diabetic-eye-screening-programme-grading-definitions-for-referable-disease. Accessed 6 Sept 2021.

  24. 24.

    Korot E, Guan Z, Ferraz D, Wagner SK, Zhang G, Liu X, et al. Code-free deep learning for multi-modality medical image classification. Nat Mach Intell. 2021;3:288–98. https://doi.org/10.1038/s42256-021-00305-2.

    Article  Google Scholar 

  25. 25.

    Khalifa NEM, Loey M, Taha MHN, Mohamed HNET. Deep transfer learning models for medical diabetic retinopathy detection. Acta Inform Med. 2019;27:327–32. https://doi.org/10.5455/aim.2019.27.327-332.

    Article  PubMed  PubMed Central  Google Scholar 

  26. 26.

    Porwal P, Pachade S, Kamble R, Kokare M, Deshmukh G, Sahasrabuddhe V, et al. Indian diabetic retinopathy image dataset (idrid): a database for diabetic retinopathy screening research. Data. 2018;3:25. https://doi.org/10.3390/data3030025.

    Article  Google Scholar 

  27. 27.

    Decencièreb E, Zhang X, Cazuguel G, Laÿ B, Cochener B, Trone C, et al. Feedback on a publicly distributed image database: the Messidor Database. Image Anal Stereol. 2014. https://doi.org/10.5566/ias.1155.

    Article  Google Scholar 

  28. 28.

    Diabetic retinopathy screening standards. 2021. https://www.healthcareimprovementscotland.org/our_work/standards_and_guidelines/stnds/diabetic_retinopathy_screening.aspx. Accessed 6 Sep 2021.

  29. 29.

    SuperAnnotate. 2020. https://superannotate.com/. Accessed 15 Jun 2021.

  30. 30.

    Visual geometry group—University of oxford. 2021. https://www.robots.ox.ac.uk/~vgg/software/via/. Accessed 15 Jun 2021.

  31. 31.

    Supervisely—Web platform for computer vision. Annotation, training and deploy. 2021. https://supervise.ly/. Accessed 15 Jun 2021.

  32. 32.

    Labelbox: The leading training data platform for data labeling. 2021. https://labelbox.com/. Accessed 15 Jun 2021.

  33. 33.

    Visual object tagging tool (VoTT) v2.2.0.2021. https://vott.z22.web.core.windows.net/. Accessed 15 Jun 2021.

  34. 34.

    Islam MT, Imran SA, Arefeen A, Hasan M, Shahnaz C. Source and camera independent ophthalmic disease recognition from fundus image using neural network. In: 2019 IEEE International Conference on Signal Processing, Information, Communication and Systems (SPICSCON). 2019. Doi:https://doi.org/10.1109/spicscon48833.2019.9065162.

  35. 35.

    Kauppi T, Kalesnykiene V, Kamarainen J-K, Lensu L, Sorri I, Raninen A, et al. The DIARETDB1 diabetic retinopathy database and evaluation protocol. In: Procedings of the British Machine Vision Conference 2007. 2007. doi:https://doi.org/10.5244/c.21.15.

  36. 36.

    Pires R, Jelinek HF, Wainer J, Valle E, Rocha A. Advancing bag-of-visual-words representations for lesion classification in retinal images. 2014. PLoS ONE. https://doi.org/10.6084/m9.figshare.953671.v1.

  37. 37.

    Decencière E, Cazuguel G, Zhang X, Thibault G, Klein J-C, Meyer F, et al. TeleOphta: machine learning and image processing methods for teleophthalmology. IRBM. 2013;34:196–203. https://doi.org/10.1016/j.irbm.2013.01.010.

    Article  Google Scholar 

  38. 38.

    Giancardo L, Meriaudeau F, Karnowski TP, Li Y, Garg S, Tobin KW Jr, et al. Exudate-based diabetic macular edema detection in fundus images using publicly available datasets. Med Image Anal. 2012;16:216–26. https://doi.org/10.1016/j.media.2011.07.004.

    Article  PubMed  Google Scholar 

  39. 39.

    Ting DSW, Pasquale LR, Peng L, Campbell JP, Lee AY, Raman R, et al. Artificial intelligence and deep learning in ophthalmology. Br J Ophthalmol. 2019;103:167–75. https://doi.org/10.1136/bjophthalmol-2018-313173.

    Article  PubMed  Google Scholar 

Download references

Acknowledgements

We would like to thank Ophthalmology sectors from Escola Paulista de Medicina/ São Paulo federal University and Instituto Paulista de Estudos e Pesquisas em Oftalmologia, IPEPO, Vision Institute, São Paulo (SP), Brazil.

Funding

M. B. Gonçalves is a researcher supported by Lemann Foundation, Instituto da Visão-IPEPO, São Paulo, Brazil and CAPES Foundation, Ministry of Education of Brazil, Brasília, DF, Brazil.

Author information

Affiliations

Authors

Contributions

LFN: Conceptualization, data curation, formal analysis, investigation, methodology, Project administration, writing-original draft, writing—review and editing. LZR: Conceptualization, formal analysis, writing—original draft, writing-review and editing. MBG: Conceptualization, methodology, writing—original draft, writing—review and editing. DF, HS, FKM, PHM, MM, CVSR: Conceptualization, methodology, writing-review and editing. RBJ: conceptualization, methodology, supervision, writing—review and editing. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Luis Filipe Nakayama.

Ethics declarations

Ethics approval and consent to participate

UNIFESP Ethics Institutional Review Board Number: CAAE 33842220.7.0000.5505/n:0698/2020.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Nakayama, L.F., Ribeiro, L.Z., Gonçalves, M.B. et al. Diabetic retinopathy classification for supervised machine learning algorithms. Int J Retin Vitr 8, 1 (2022). https://doi.org/10.1186/s40942-021-00352-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s40942-021-00352-2

Keywords

  • Diabetic retinopathy classifications
  • Artificial intelligence
  • Datasets