Explainable Multimodal Deep Neural Networks for Identifying Ocular Fundus Diseases and Report Generation

The ocular fundus is the only part of the human body that can directly see the blood vessel microcirculation and nerve tissue. Through various imaging tests, including Color Fundus Photograph (CFP), Optical Coherence Tomography (OCT), Fluorescein Fundus Angiography (FFA) and Indocyanine Green Angiography (ICGA), etc., it is possible to statically overview or dynamically observe the retina and choroid, the condition of blood vessels and nerves, and comprehensive diagnosis of the disease. The screening, interpreting and accurate diagnosis of ocular fundus diseases are crucial for disease prevention, control and precise treatment. However, due to the variety of fundus examination methods, and the complexity and professionalism of the examination, there is a lack of fundus specialists who have sufficient clinical experience and knowledge to interpret fundus examinations. With the continuous development of artificial intelligence (AI) in diagnosing fundus diseases, various modalities of imaging examination methods are gradually applied to the development of fundus disease diagnosis systems. Moreover, medical images often come with corresponding reports, which are mostly generated by clinicians' or radiologists' experience.

Here, we are establishing a fundus disease diagnosis and report-generating system based on cross-modal ocular fundus imaging examinations, and fundus lesions were visualized at the same time. Multi-center data verification will also be conducted. The results of the research will assist in fundus lesions diagnosis and imaging reports generation. We hope this could popularize more complex fundus imaging examination methods to society, and help improve the early diagnosis and treatment of fundus lesions that cause blindness.