Development and Validation of an Automatic Ultrawide-Field Fundus Imaging Enhancement System for Facilitating Clinical Diagnosis: A Cross-Sectional Multicenter Study
Qiaoling Wei
,
Zhuoyao Gu
,
Weimin Tan
,
Hongyu Kong
,
Hao Fu
,
Qin Jiang
,
Wenjuan Zhuang
,
Shaochi Zhang
,
Lixia Feng
,
Yong Liu
,
Suyan Li
,
Bing Qin
,
Peirong Lu
,
Jiangyue Zhao
,
Zhigang Li
,
Songtao Yuan
,
Hong Yan
,
Shujie Zhang
,
Xiangjia Zhu
,
Jiaxu Hong
,
Chen Zhao
,
Bo Yan
Development and Validation of an Automatic Ultrawide-Field Fundus Imaging Enhancement System for Facilitating Clinical Diagnosis: A Cross-Sectional Multicenter Study
a Department of Ophthalmology, Eye & ENT Hospital of Fudan University, Shanghai 200231, China
b School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai 200433, China
c The Affiliated Eye Hospital of Nanjing Medical University, Nanjing 210029, China
d Ningxia Eye Hospital, People’s Hospital of Ningxia Hui Autonomous Region, Third Clinical Medical College of Ningxia Medical University, Yinchuan 750002, China
e Department of Ophthalmology, The First Affiliated Hospital of Anhui Medical University, Hefei 230022, China
f Southwest Hospital/Southwest Eye Hospital, Third Military Medical University (Army Medical University), Chongqing 400038, China
g Key Lab of Visual Damage and Regeneration Restoration of Chongqing, Chongqing 400038, China
h Department of Ophthalmology, The Affiliated Xuzhou Municipal Hospital of Xuzhou Medical University, Xuzhou 221000, China
i Department of Ophthalmology, Suqian First Hospital of Jiangsu Province Hospital, Suqian 223800, China
j Department of Ophthalmology, The First Affiliated Hospital of Soochow University, Suzhou 215006, China
k Department of Ophthalmology, The Fourth Affiliated Hospital of China Medical University, Eye Hospital of China Medical University, Shenyang 110000, China
l Department of Ophthalmology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou 450052, China
m Department of Ophthalmology, Jiangsu Province Hospital, Nanjing 210029, China
n Shaanxi Eye Hospital, Xi’an People’s Hospital (Xi’an Fourth Hospital), The Affiliated People’s Hospital of Northwest University, Xi’an 710004, China
In ophthalmology, the quality of fundus images is critical for accurate diagnosis, both in clinical practice and in artificial intelligence (AI)-assisted diagnostics. Despite the broad view provided by ultrawide-field (UWF) imaging, pseudocolor images may conceal critical lesions necessary for precise diagnosis. To address this, we introduce UWF-Net, a sophisticated image enhancement algorithm that takes disease characteristics into consideration. Using the Fudan University ultra-wide-field image (FDUWI) dataset, which includes 11 294 Optos pseudocolor and 2415 Zeiss true-color UWF images, each of which is rigorously annotated, UWF-Net combines global style modeling with feature-level lesion enhancement. Pathological consistency loss is also applied to maintain fundus feature integrity, significantly improving image quality. Quantitative and qualitative evaluations demonstrated that UWF-Net outperforms existing methods such as contrast limited adaptive histogram equalization (CLAHE) and structure and illumination constrained generative adversarial network (StillGAN), delivering superior retinal image quality, higher quality scores, and preserved feature details after enhancement. In disease classification tasks, images enhanced by UWF-Net showed notable improvements when processed with existing classification systems over those enhanced by StillGAN, demonstrating a 4.62% increase in sensitivity (SEN) and a 3.97% increase in accuracy (ACC). In a multicenter clinical setting, UWF-Net-enhanced images were preferred by ophthalmologic technicians and doctors, and yielded a significant reduction in diagnostic time ((13.17 ± 8.40) s for UWF-Net enhanced images vs (19.54 ± 12.40) s for original images) and an increase in diagnostic accuracy (87.71% for UWF-Net enhanced images vs 80.40% for original images). Our research verifies that UWF-Net markedly improves the quality of UWF imaging, facilitating better clinical outcomes and more reliable AI-assisted disease classification. The clinical integration of UWF-Net holds great promise for enhancing diagnostic processes and patient care in ophthalmology.
Fundus photography (FP) has emerged as a pivotal element in the realm of healthcare artificial intelligence (AI), offering critical data for algorithms that diagnose and manage both retinal and systemic diseases with remarkable precision and speed. FP-based AI systems herald a new era of cost-effective, wide-ranging disease screening, contributing significantly to the democratization of healthcare through accessible, rapid diagnostic capabilities. However, a significant challenge in real-world clinical settings is highlighted by a report indicating that approximately 21% of fundus photographs are undetected or misclassified by AI [1]. This limitation not only affects AI model performance but also hinders the practical implementation of AI in clinical practice, as this limitation impacts the reliability of AI models in real-world diagnostics. As AI has evolved, the need for high-quality fundus imagery has become increasingly pronounced, underscoring the indispensability of such imagery for the successful application of AI solutions in real-world clinical environments [1], [2].
Traditional narrow-angle FP, which captures a limited 45 degree view of the retina, was the earliest and most widely used resource for AI development in ophthalmology. This imaging modality, while foundational, has notable limitations, particularly in its potential to leave out indicators of peripheral retinal pathologies, thus limiting the thoroughness of ophthalmological disease analysis. In contrast, ultrawide-field (UWF) imaging, notably when used with advanced devices such as the Optos Daytona P200T, offers substantial progress by providing a 200 degree panoramic retinal view in a single image. Wild-field imaging can be performed without the need for dilation, when adverse conditions, such as dense cataracts or vitreous opacity, are a factor due to the enhanced penetration capability of this type of imaging. Nevertheless, the inherent pseudocolor nature of UWF imaging may present a significant risk of critical diagnostic details being concealed, as indicated in recent studies [3], [4]. The advent of enhanced UWF imaging, which incorporates true-color fidelity with improved lesion visibility, is anticipated to substantially improve fundus lesion detection in clinical practice. The integration of enhanced UWF imaging into AI systems is poised to significantly transform ophthalmology, facilitating a broader and more precise approach to disease detection and management.
The exploration of automatic enhancement techniques for UWF fundus images remains a nascent field, hindered primarily by a paucity of comprehensive datasets. Notably, existing databases such as deep diabetic retinopathy (DeepDR), with its limited offering of 256 images, prove inadequate for developing robust enhancement algorithms. Research has predominantly focused on improving narrow-angle fundus photos through methods such as wavelet transform [5], contrast limited adaptive histogram equalization (CLAHE) [6], and nonsubsampled contourlet transform (NSCT) [7], [8]. However, these techniques may not adequately capture the subtleties of critical lesion areas, a challenge amplified by the extensive field of view of UWF imaging. Unsupervised deep learning methods, including cycle-consistent generative adversarial network (CycleGAN) [9], [10], [11], eliminate the need for paired images; however, they also come with the risk of detail loss during the translation process. Even sophisticated algorithms such as structure and illumination constrained generative adversarial network (StillGAN) [12], while preserving detail, fall short in addressing the unique UWF imaging challenges, underscoring the need for specialized research focused on UWF fundus image enhancement.
In clinical ophthalmology, the challenge of obtaining high-quality image pairs from Optos UWF systems often leads to a preference for unsupervised enhancement techniques. Ideal UWF enhancement techniques should rectify pseudocolor discrepancies and artifacts while enhancing lesion signals. The Zeiss UWF system stands out for its true-color, high-resolution imaging across an UWF, enabling detailed posterior pole imaging. Nevertheless, its limitations, including a narrower field of view and reduced effectiveness in dense ocular media, along with peripheral distortion, hinder its ability to fully supplant Optos UWF images [3], [13]. Fig. 1 illustrates the comparative characteristics of these retinal imaging systems. To tackle these challenges, we compiled a comprehensive UWF dataset from both the Optos and Zeiss systems, with detailed annotations by retinal experts. Utilizing this dataset, we developed UWF-Net, an advanced algorithm designed to enhance Optos images. Our comparative analyses show that UWF-Net not only conserves critical disease details but also surpasses alternatives such as StillGAN in terms of image quality and clinical applicability. The clinical efficacy of UWF-Net, particularly in AI-assisted diagnostics, has been corroborated through extensive evaluations and validations by ophthalmologists.
2. Methods
2.1. Data preparation
Optos and Zeiss UWF images were acquired from the outpatient ophthalmology clinic at the Eye & ENT (EENT) Hospital of Fudan University, China. The study protocol was approved by the Ethics Committee of the EENT Hospital of Fudan University (Approval No. 2023427) and conformed to the tenets of the Declaration of Helsinki. Patient confidentiality was rigorously protected through comprehensive anonymization protocols. These images were annotated by four ophthalmologists who cross-referenced the clinical records to ensure annotation accuracy. Any discrepancies in annotation were resolved by a senior specialist, resulting in the formation of the Fudan University ultra-wide-field image (FDUWI) dataset. The FDUWI-1 segment includes 11 294 Optos images, each with a resolution of 3070 × 3900 pixels (Daytona; Optos®, UK). The FDUWI-2 segment comprises 2415 Zeiss images with resolutions ranging from 4000 to 6700 pixels (Clarus 500; Carl Zeiss Meditec AG, Germany), encompassing both single and composite images. The dataset assembly and annotation processes are illustrated in the workflow presented in Fig. 2(a).
2.2. UWF-Net architecture
We based UWF-Net’s generators on the model proposed by Zhu et al. [9]. The architecture includes a sequence of three convolutional layers, nine residual blocks [14], and upsampling layers, culminating in a red-green-blue (RGB) mapping layer. Each convolution is followed by instance normalization [15], and the residual blocks incorporate Convolution-InstanceNorm-rectified linear unit (ReLU) sequences with shortcut connections for improved performance. The discriminator architecture is adapted from patch-based generative adversarial network (PatchGAN) [16], which is ideal for processing high-resolution UWF images.
UWF-Net was developed using the PyTorch framework on an Nvidia GTX 1080Ti graphics processing unit (GPU). Its training utilized the Adam optimizer, starting with a learning rate of 2 × 10−2 and a batch size of 2. The weighted parameters were set to α = 5.0 and β = 40. In our investigation, we employed random cropping on UWF retinal images with a 3070 × 3900 pixels resolution to generate subimages of 512 × 512 pixels. These subimages were stored and retrieved for training, to reduce the computational load and expedite the training process. Prior to inputting the image data into the network, data augmentation techniques that included horizontal flipping and random cropping were applied to further reduce the image resolution, resulting in a consistent training resolution of 256 × 256 pixels. During the inference phase, full original resolution images were directly fed into the network to ensure the desired high-resolution output with preserved finer details, which are essential for precise analysis.
As shown in Fig. 2(b), UWF-Net combines global style modeling with feature-level lesion refinement, drawing on insights from our UWF disease classification network. This network is designed to preserve pathological features, including small lesions and intricate anatomical structures, improving the image enhancement processes.
Fig. 2(b) also illustrates the network’s backbone, which includes two encoders and a pathology-aware attention module. This module uses a pretrained convolutional neural network (CNN), specifically residual network-101 (ResNet-101) [14], and the convolutional block attention module (CBAM) [17], to extract and encode critical lesion data into feature maps , where Fe is pathology-aware disease feature. We further strengthened the network’s performance using asymmetric loss (ASL) [18], addressing the challenge of imbalanced disease annotations. The network’s structure involves two generators and two discriminators, conceptualizing UWF image enhancement as a domain translation from lower-quality Optos images (domain A) to higher-quality Zeiss images (domain B). For training, the network uses samples from both domains, where and N is the size of the dataset from domain A, and where , and M is the size of the dataset from domain B. The network learns mappings: and , with discriminators and distinguishing between real and generated images.
The network utilizes identity mapping loss to regularize two generators, constraining them to output images that are as close as possible to the input ones when they are already actual samples of the target domain. This mapping loss Lidt is defined as: where E denotes the expectation, quantifying the average magnitude of the identity mapping error across all samples in domains A and B, respectively.
The translation loss combines the adversarial loss [19] with the identity mapping loss, where the adversarial loss Ladv is applied to both mappings. It is defined as:
The overall translation loss Ltrans is the sum of these losses:where parameter α is experimentally set as α = 5.0 to control the relative importance of the two objectives.
Pathological consistency loss is critical for maintaining the fidelity of medical images. This ensures that the reconstructed images accurately reflect the original pathological features. For each image , the network generates a reconstructed image , and likewise for each image , the network guarantees the preservation of pathological features in the reconstructed image . The pathological feature consistency loss, Lpc, is thus defined as:where represents the dimensions of the feature map, and denotes the pathology-aware disease feature map , extracted by the pretrained disease classification network (DCN).
The composite objective function for UWF-Net, which balances translation and pathological consistency loss, is given by:where the parameter is empirically set to 40 to balance the relative significance of the two objectives in the model.
To bridge the current gap in UWF image quality resources, distinct from existing narrow-angle fundus datasets such as EyeQ [20], HRF [21], and DR2 [22], we developed a unique dataset for UWF image quality assessment derived from the FDUWI-2 collection. This dataset encompasses 1000 UWF retinal images acquired using Zeiss instruments. Twenty-three machine learning engineers were enlisted to conduct quality evaluations based on the following criteria: Images displaying uniform brightness and clarity were assigned a score of 3, those with minor issues in brightness or contrast that were still sufficient for analysis received a score of 2, and images with significant problems—such as pronounced brightness variability, low contrast, or extensive blurring—that affected their analytical utility were scored as 1. Each engineer independently evaluated 1000 images. The Kruskal-Wallis chi-squared test was applied to ascertain interrater reliability, yielding a P value of 1, which, exceeding the threshold of 0.05, indicating no statistically significant discrepancy in ratings among different engineers. The mean score for each image was subsequently calculated to classify image quality: An average score of 2.5 or higher was designated “Good,” scores between 1.5 and 2.5 were labeled “Usable,” and scores of 1.5 or lower were designated “Reject” [23]. The resultant dataset, comprising 177 “Good,” 590 “Usable,” and 233 “Reject” images, established a comprehensive benchmark for assessing the quality of UWF fundus images. The methodology of this assessment is shown in Fig. 3(a).
2.4. Statistical analysis
To assess the impact of UWF-Net on disease classification in UWF fundus images, we utilized statistical measures such as the mean average precision (mAP), sensitivity (SEN), area under the curve (AUC), accuracy (ACC), and F1 score. These metrics facilitated a detailed evaluation of each enhancement method’s performance compared to that of the original images.
For clinical validation, we carried out a diagnostic efficiency study involving ophthalmologists across multiple centers. We used Welch’s t test to compare the time spent on diagnosis and the overall accuracy between the UWF-Net-enhanced and original datasets, including an analysis of the diagnostic time across various diseases. Additionally, Pearson’s chi-squared test with Yates’ continuity correction was applied to investigate the rates of misdiagnosis of specific diseases, with the aim of discerning the benefits of UWF-Net enhancement in particular diagnostic scenarios. All the statistical analyses were performed with R software (version 4.2.3), and the figures were created using GraphPad Prism 6.0 and Photoshop, with a P value less than 0.05 considered to indicate statistical significance.
3. Results
In our evaluation, UWF-Net’s capabilities were benchmarked against established and emerging enhancement techniques, including CLAHE [6], dark channel prior (DCP) [23], low-light image enhancement (LIME) [24], StillGAN [12], zero-reference deep curve estimation (Zero-DCE) [25], and enlighten generative adversarial network (EnlightenGAN) [26]. Parameters for UWF images were optimized and data preprocessing was standardized for deep learning algorithms. The evaluation metrics included fundus image quality assessment (FIQA) scores and disease classification accuracy.
FIQA scores for enhanced UWF fundus images were determined using two distinct networks. Initially, multiple color-space fusion network (MCF-Net) [20], designed predominantly for narrow-angle FIQA, was trained on the EyeQ dataset [20] to quantitatively assess the enhanced UWF images. To further tailor our quality assessment to the specifics of UWF imagery, we developed a specialized UWF-FIQA using the densely connected convolutional network (DenseNet-121) architecture [27] informed by our meticulously annotated quality dataset. The UWF-FIQA achieved an 83.0% accuracy in predicting the FIQA scores. These FIQA scores, as determined by the MCF-Net and DenseNet-121 systems, were based on the proportion of “Good” images within the total assessed. A higher score indicated more successful enhancement. The comprehensive results of our assessments, encompassing both classification outcomes and quality scores, are compiled in Table 1. UWF-Net emerged as a superior enhancement method, achieving 9.2% and 18.8% quality scores in the MCF-Net and DenseNet-121 systems, respectively. Remarkably, UWF-Net upgraded a significant number of images categorized as “Reject” to “Good” status—52 in the MCF-Net system and 106 in the DenseNet-121 system. The visual results presented in Fig. 3(b) highlight UWF-Net’s ability to preserve critical optic disc details and authentic color rendition, surpassing the performance of other methods that faced issues with color fidelity and parameter sensitivity. Enhancements by UWF-Net yielded images with enhanced realism and improved quality, signifying a substantial advancement for both ophthalmological AI applications and clinical diagnostics.
To evaluate the impact of improved image quality on disease classification efficacy, a ResNet-101 [14] classifier trained on both original images and those enhanced by UWF-Net was initially employed. The resulting confusion matrix, as depicted in Fig. 3(c), indicates marked improvements in the identification of proliferative diabetic retinopathy (PDR), epiretinal membrane (ERM), and pathologic myopia (PM) from using enhanced images. Notably, these enhancements helped maintain robust detection capabilities for maculopathy. Comparative analyses highlight that using enhanced images leads to better SEN and AUC metrics than methods such as StillGAN, with increases of 4.62% in SEN and 3.97% in ACC when employing the ResNet-101 as the backbone architecture, as shown in Table 2. While techniques such as DCP offered classification benefits, they often compromised image authenticity, especially in color representation and optic disc visibility—challenges adeptly mitigated by the visual fidelity improvements of UWF-Net.
To further assess the impact of improved image quality on classification efficacy with advanced models, we incorporated additional classifiers such as ViT_base_patch16_224 [28] and DenseNet-121 [27], and a cutting-edge foundation model for retinal images (RETFound) model using a ViT_large_patch16_224 backbone [29]. The comprehensive results of these expanded experiments are detailed in Table 2. Using the ViT_base_patch16_224 backbone, the enhanced images achieved the highest mAP, ACC, and F1 scores. The DenseNet-121 backbone showed similar peaks in these metrics after UWF-Net enhancement. The application of RETFound model also significantly improved the ACC, SEN, and F1 score, indicating that UWF-Net achieves effective classification.
Our research incorporates the pathological consistency loss within the UWF-Net framework, ensuring the retention of medical image fidelity after enhancement. To ascertain the impact this might have on postenhancement retinal quality and disease classification efficacy, we conducted ablation studies using the CycleGAN framework [9], complemented by U-Net [30] for vessel segmentation derived from the digital retinal images for vessel extraction (DRIVE) dataset. Alongside cycle-consistency loss, we employed segmentation loss (Lseg) to synchronize with the vessel segmentation network and structural similarity loss (Lssim) inspired by StillGAN’s method [31]. Table 3 shows that the pathological consistency loss notably enhances disease classification metrics, including the mAP, SEN, and AUC. Although Lssim produced high FIQA scores when evaluated by MCF-Net, it did not retain critical pathological features well, adversely impacting disease classification accuracy. In contrast, the incorporation of pathological consistency loss resulted in superior classification performance. Fig. 3(d) displays the results of enhancing both normal and blurry fundus images. Our method focuses on the optic disc and macular regions for improved clarity, addressing the baseline method’s limitations in preserving vital details, the segmentation method’s vessel-centric focus, and the structural similarity’s tendency to oversmooth. Our methodology leverages pathological consistency loss to not only augment clarity in crucial regions but also preserve indispensable pathological features crucial for accurate disease diagnosis. Fig. S1 in Appendix A shows pairs of original and enhanced UWF images, processed by UWF-Net and baseline enhancement without pathological consistency loss to illustrate the advantages of integrating pathological consistency loss.
To assess the clinical effectiveness of UWF-Net, a preference study was first carried out with 23 eye specialists at EENT Hospital. They evaluated 50 sets of UWF fundus images; each set included an original and three variants improved by UWF-Net, Lssim, and StillGAN, chosen for their notable performance in prior retinal quality evaluations. The participants selected images based on the visual quality of the fundus, focusing on resemblance to real fundus color, which aids in diagnosis. The images spanned various retinal conditions: 9 normal, 5 with macular degeneration, 1 with acute retinal necrosis (ARN), 2 with PDR, 2 with retinitis pigmentosa (RP), 2 with retinal vasculitis, 3 with drusen, 10 with peripheral retinal degeneration, 6 with PM, 6 with optic disk abnormality, and 4 with ERM, totaling 200 images per participant. This detailed method, as shown in the procedural flow presented in Fig. 4(a), ensured a thorough evaluation across diverse pathologies.
A specially designed mobile application (app) supported the blind review process, presenting each expert with a randomized series of four images—one original and three modifications—to guarantee impartial feedback. The survey results, outlined in Table 4, strongly underscored the fidelity of the enhanced UWF-Net images to real fundus color, confirming its excellence and clinical significance in diagnosis.
To further evaluate the diagnostic performance of UWF-Net and explore its benefits for specific diseases, a comprehensive multicenter study was undertaken at twelve eye centers spanning eight provinces in China, as illustrated in Fig. 4(b). In this study, ophthalmologists used a mobile app to assess 30 images covering ten different conditions, with a distribution of UWF-Net enhanced and original images as follows: age-related macular degeneration (AMD; 2 enhanced, 1 original), ARN (1 enhanced, 2 original), branch retinal vein occlusion (BRVO; 1 enhanced, 2 original), central serous chorioretinopathy (CSC; 2 enhanced, 1 original), diabetic retinopathy (DR; 2 enhanced, 1 original), ERM (1 enhanced, 2 original), macular hole (MH; 1 enhanced, 2 original), retinal hole (RH; 2 each in both sets), RP (1 enhanced, 2 original), and Vogt-Koyanagi-Harada (VKH) disease (1 each in both sets). Both sets of images were carefully matched for pathological characteristics and lesion locations. The diagnoses were made using standard abbreviations, with results recorded for diagnostic time, accuracy, and misdiagnosis details.
Analysis of data from 182 physicians indicated that the accuracy of the UWF-Net-enhanced images reached 87.71%, which was notably higher than the 80.40% observed for the original images. Significant improvements in accuracy were observed in conditions such as DR, VKH disease, RP, ARN, BRVO, CSC, ERM, and MH, with particularly notable gains in the last five, as shown in Figs. 5(a) and (b). Furthermore, UWF-Net significantly reduced the diagnostic time to an average of (13.17 ± 8.40) s per image, compared to (19.54 ± 12.40) s for the original images. This reduction was consistent across most conditions, barring DR, as detailed in Figs. 5(c) and (d). These findings not only confirm the enhanced diagnostic efficiency of UWF-Net but also highlight its potential for improving clinical outcomes for specific retinal diseases, thus reinforcing its value for clinical application.
4. Discussion
In China, the acute shortage of ophthalmologists, especially in rural and remote areas, leads to a significant gap in eye care for patients and results in preventable blindness. The advent of ophthalmology AI promises to democratize access to eye care and mitigate healthcare disparities. Although numerous AI systems for retinal disease diagnosis using FP have shown remarkable accuracy and speed, their real-world implementation is limited. This limitation is partly due to the discrepancy between the quality of fundus images used in AI training and those typically obtained in clinical practice. AI models are often trained on selective, publicly available photographs that fail to capture the diversity and complexity of cases in clinical settings. Furthermore, these datasets usually focus on single diseases, not encompassing the variety required for comprehensive clinical practice. Another significant challenge is the reliance of these AI models on traditional narrow-angle FP, which limits their ability to detect peripheral retinal pathologies that are crucial for diagnosing emergencies such as retinal breaks or detachments.
Our research is dedicated to creating an automatic UWF enhancement system to improve UWF imaging accuracy and disease detection, thus aligning AI’s capabilities with the practical demands of real-world ophthalmic practice, especially in underserved areas. The primary challenge in creating an enhanced UWF system lies in the limited availability of public UWF image datasets. Currently, DeepDR is the only known online UWF dataset, containing 256 images specifically for DR diagnosis. In contrast, other well-known public datasets, such as eye picture archive communication system (EyePACS), methods to evaluate segmentation and indexing techniques in the field of retinal ophthalmology (MESSIDOR2), the Indian diabetic retinopathy dataset (IDRiD), age-related eye disease study (AREDS), and the retinopathy of prematurity (ROP) dataset, are focused on traditional narrow-angle FPs and typically target single diseases such as DR, AMD, or retinopathy of prematurity.
Diverging from these existing datasets, we amassed a comprehensive collection of UWF fundus images representing a wide array of diseases and manifestations, including images of varying quality, to effectively address the complexities of clinical settings. These images were rigorously analyzed and annotated by retinal experts. As a result, we compiled two substantial datasets: one with 11 294 Opto UWF photos and the other with 2415 true color UWF photos. Moreover, because most existing assessments of retinal image quality are tailored for traditional narrow-angle FP, we have taken an additional significant step forward. We established a UWF image dataset featuring Zeiss UWF images, categorized into three quality levels: “Reject, ” “Usable,” and “Good.” Building upon this dataset, we developed a UWF image assessment system using DenseNet-121. To our knowledge, this endeavor represents the first comprehensive, annotated UWF datasets. The release of these datasets, coupled with the UWF image quality assessment system—the first such compilation known to us—is set to markedly propel the progress and enhancement of AI systems, particularly those utilizing UWF imaging, in the field of ophthalmology.
Leveraging our extensive datasets, we engineered UWF-Net, an advanced framework tailored for enhancing UWF fundus images. This innovative system adeptly eliminates the need for for paired high/low-quality images, harmonizing the capabilities of Optos and Zeiss imaging systems. The results of the FIQA score experiments, as presented in Table 1, indicate the quantitative superiority of UWF-Net over other existing enhancement methods. Visually, as depicted in Fig. 3(b), UWF-Net also demonstrates qualitative excellence in retinal quality assessments, surpassing prevalent techniques. While existing methods such as CLAHE and LIME effectively preserve retinal structures, they often produce images with unrealistic coloration. Techniques such as DCP, although capable of adjusting image brightness, are overly sensitive to parameter settings, introducing a risk of overexposure. Manual parameter adjustment for each image is not feasible. In contrast, techniques such as EnlightenGAN and StillGAN frequently result in images mapped to suboptimal color and illumination spaces.
A key innovation in UWF-Net is the integration of a novel pathological feature consistency loss, implemented via a pretrained classification network. An ablation study to evaluate the impact of this feature consistently demonstrated the superior performance of UWF-Net in terms of FIQA, disease classification tasks (as detailed in Table 3), and visual comparison (illustrated in Fig. 3(d)). This marked a significant leap in fundus image enhancement technology, potentially setting a new standard in retinal imaging. The ability of UWF-Net to maintain clear details and accurately map images to true biological colors, with optimal contrast and illumination, is especially notable.
Within the scope of our disease classification task, we focused on ERM, PDR, PM, and a range of maculopathies due to their distinctive pathological characteristics that are critical for evaluating in UWF images. This selection was based on the need to address diseases with pathologies that are not primarily located in the peripheral retina—a region where Optos UWF imaging already performs well—such as conditions exemplified by the ARN and RP. By concentrating on diseases predominantly affecting the posterior pole, we aimed to assess the efficacy of UWF-Net in areas where Optos UWF imaging may encounter challenges in detail and clarity. Consequently, UWF-Net was developed to enhance the visualization of the posterior pole in Optos UWF images, striving to achieve a level of detail comparable to that provided by Zeiss UWF imaging systems.
In this study, we employed a suite of state-of-the-art neural network backbones for disease classification, including ResNet-101, ViT_base_patch16_224, DenseNet-121, and the newly introduced RETFound model utilizing the ViT_large_patch16_224 backbone. While images enhanced by UWF-Net demonstrated potential improvements in the efficacy of subsequent classification by various algorithms, the performance metrics across all employed methods were observed to be tightly clustered, with the highest values showing only slight improvements over those ranked second, or modest advancements over the baseline. This phenomenon could be attributed to the high baseline accuracy afforded by the carefully curated dataset. To evaluate the indirect benefits of UWF-Net-enhanced images more comprehensively for disease classification tasks, future research should employ broader datasets encompassing a more diverse range of real-world conditions. Such studies would be instrumental in confirming the full potential of UWF-Net in supporting disease classification processes in the domain of ophthalmological AI.
In clinical ophthalmology, the efficacy of UWF-Net as an image enhancement tool was rigorously assessed through a two-pronged approach. A preference study with 23 ophthalmic professionals unanimously endorsed UWF-Net-enhanced images over conventional methods, validating their superior visual quality and diagnostic utility. Subsequently, a comprehensive multicenter survey involving 182 physicians from 12 institutions evaluated the impact of using UWF-Net-enhanced images in real-world applications. The survey results showed a significant reduction in diagnostic time for various retinal diseases, except for DR, and enhanced diagnostic accuracy for conditions such as ARN, BRVO, CSC, ERM, and MH. These findings underscore the potential of UWF-Net to improve ophthalmological diagnostic processes significantly, suggesting its substantial benefit in patient care within real-world clinical settings.
This study represents a significant advancement in retinal imaging enhancement. However, it is crucial to acknowledge certain limitations that highlight avenues for future research. A primary limitation of our multicenter study was the decision to avoid exact one-to-one matching of images between the enhanced and original datasets. This strategy, which is designed to prevent bias from repeated exposure to identical images, may impact the reliability of our comparative analysis of diagnostic performance. Despite efforts to select nonidentical images with similar lesions and conditions, this method could influence the study's outcomes. Another important aspect to consider is the dataset’s focus on the Chinese Han population, raising questions about the universal applicability of the UWF-Net system across different ethnic and racial groups. This highlights the need for further research to enhance the system’s global relevance. Additionally, while images enhanced by UWF-Net have shown promising results in augmenting AI-based diagnosis for DR, the effectiveness of UWF-Net in enhancing images for a broader range of retinal diseases warrants further investigation. A comprehensive evaluation of UWF-Net across various retinal conditions is essential to fully ascertain its utility in diverse pathological contexts.
5. Conclusions
In this study, we introduced UWF-Net, an innovative framework developed for the enhancement of UWF fundus images. This framework uniquely improves image quality across various clinical imaging devices without relying on paired image comparisons. Its key feature is the incorporation of pathological feature consistency loss, supported by a pretrained classification network, which plays a crucial role in preserving essential retinal details. Additionally, we presented the FDUWI dataset, an extensive collection of UWF images with detailed annotations, marking a significant first for the field and offering a valuable resource for ongoing research. Central to our findings is the capability of UWF-Net to enhance the quality of images that are subsequently used for AI-based diagnosis, potentially improving accuracy and utility in clinical settings. Through thorough testing and validation in 12 hospitals, UWF-Net has proven effective in real-world clinical scenarios. Its contribution to enhancing diagnostic precision and facilitating clinical decision-making positions UWF-Net as an important advancement in the application of AI in ophthalmology. This approach represents as a comprehensive solution, aligning cutting-edge AI technology with the evolving requirements of ophthalmological clinical practice.
Acknowledgments
We extend our deepest gratitude to the following individuals and groups who contributed immensely to our multi-center study:
•The ophthalmic technicians and doctors from the Eye & ENT Hospital of Fudan University for their participation in the survey.
•The entire ophthalmology department of The Affiliated Eye Hospital of Nanjing Medical University.
•Lin Pang, Jiayu Li, Chunhua Li, Wen Zhang, Guofeng Zhan, Feng Gao, Wei Fang, Yujuan Cai, Xiaoguang Wang, Juan Liu, and Xiaolong Qi from Ningxia Eye Hospital, People’s Hospital of Ningxia Hui Autonomous Region.
•All ophthalmologists from The First Affiliated Hospital of Anhui Medical University.
•Ya Qu, Hao Wang, Peng Gu, Tongtao Zhao, Yiming Ren, Shujia Huo, Jiajia Yao, Li Ran, Xinyue Huang, Sha Li, Fang Wang, Lian Tan, Zui Tao, Dongmei Qi, Ping Duan, Qian Yu, Xu Zhou, Yu Wang, and Xiaoxiao Cui from Southwest Hospital/Southwest Eye Hospital, Third Military Medical University (Army Medical University).
•Zhengpei Zhang, Haiyang Liu, Sujuan Ji, Lina Guan, Yalu Liu, Jie Li, Aiqin Sheng, Wei Fan, and Meili Li from The Affiliated Xuzhou Municipal Hospital of Xuzhou Medical University.
•The ophthalmology team at Suqian First Hospital of Jiangsu Province Hospital.
•The ophthalmology department of The First Affiliated Hospital of Soochow University.
•Junbo Rong, Lijuan Lang, Limin Xu, Luxi Zhang, and Kexin Guo from The First Affiliated Hospital of Zhengzhou University.
•Xue Chen, Hui Hang, and Wen Fan from Jiangsu Province Hospital.
•Bo Lu, Mengsu Tang, Xuedong Li, Chunxia Wang, Lingfeng Jiang, Bo Zou, Tianxiao Zhang, Lu Lu, Wenkai Zhou, Xiao Han, Yuan Ning, Nan Chen, Dong Shi, and Xiaoxia Ding from The Fourth Affiliated Hospital of China Medical University, Eye Hospital of China Medical University.
•Haiyan Wang, Weimei Ma, Laiqiang Qu, and Jin Deng from Shaanxi Eye Hospital, Xi’an People’s Hospital (Xi’an Fourth Hospital).
Their invaluable cooperation and expert insights have been pivotal to the success of this research. We also thank all the anonymous reviewers for their contributions to improving this manuscript.
This study was supported by the National Natural Science Foundation of China (82020108006 and 81730025 to Chen Zhao, U2001209 to Bo Yan), the Excellent Academic Leaders of Shanghai (18XD1401000 to Chen Zhao), and the Natural Science Foundation of Shanghai, China (21ZR1406600 to Weimin Tan).
Compliance with ethics guidelines
Qiaoling Wei, Zhuoyao Gu, Weimin Tan, Hongyu Kong, Hao Fu, Qin Jiang, Wenjuan Zhuang, Shaochi Zhang, Lixia Feng, Yong Liu, Suyan Li, Bing Qin, Peirong Lu, Jiangyue Zhao, Zhigang Li, Songtao Yuan, Hong Yan, Shujie Zhang, Xiangjia Zhu, Jiaxu Hong, Chen Zhao, and Bo Yan declare that they have no conflict of interest or financial conflicts to disclose.
D.Lin, J.Xiong, C.Liu, L.Zhao, Z.Li, S.Yu, et al. Application of Comprehensive Artificial intelligence Retinal Expert (CARE) system: a national real-world evidence study. Lancet Digit Health, 3 (8) (2021), pp. e486-e495.
[2]
N.Panwar, P.Huang, J.Lee, P.A.Keane, T.S.Chuan, A.Richhariya, et al. Fundus photography in the 21st century: a review of recent technological advances and their implications for worldwide healthcare. Telemed J E Health, 22 (3) (2016), pp. 198-208.
[3]
C.S.Tan, M.C.Chew, J. vanHemert, M.A.Singer, D.Bell, S.R.Sadda. Measuring the precise area of peripheral retinal non-perfusion using ultra-widefield imaging and its correlation with the ischaemic index. Br J Ophthalmol, 100 (2) (2016), pp. 235-239.
[4]
M.Maruyama-Inoue, Y.Kitajima, S.Mohamed, T.Inoue, S.Sato, A.Ito, et al. Sensitivity and specificity of high-resolution wide field fundus imaging for detecting neovascular age-related macular degeneration. PLoS One, 15 (8) (2020), Article e0238072.
[5]
H.Demirel, G.Anbarjafari. Image resolution enhancement by using discrete and stationary wavelet decomposition. IEEE Trans Image Process, 20 (5) (2011), pp. 1458-1460.
[6]
K.Zuiderveld. Contrastlimited adaptive histogram equalization. P.S.Heckbert ( Ed.), Graphics gems IV, Academic Press (1994), pp. 474-485.
[7]
A.Swaminathan, S.S.Ramapackiam, T.Thiraviam, J.Selvaraj. Contourlet transform-based sharpening enhancement of retinal images and vessel extraction application. Biomed Tech, 58 (1) (2013), pp. 87-96.
[8]
C.C.Lee, C.Y.Shih, S.K.Lee, W.T.Hong. Enhancement of blood vessels in retinal imaging using the nonsubsampled contourlet transform. Multidimens Syst Signal Process, 23 (4) (2012), pp. 423-436.
[9]
ZhuJY, ParkT, IsolaP, EfrosAA. Unpairedimage-to-image translation using cycle-consistent adversarial networks. Proceedingsof the IEEE International Conference on Computer Vision; 2017Oct 22-29; Venice, Italy. Piscataway: IEEE; 2017. p. 2223-32.
[10]
T.K.Yoo, J.Y.Choi, H.K.Kim. CycleGAN-based deep learning technique for artifact reduction in fundus photography. Graefes Arch Clin Exp Ophthalmol, 258 (8) (2020), pp. 1631-1637.
Y.Ma, J.Liu, Y.Liu, H.Fu, Y.Hu, J.Cheng, et al. Structure and illumination constrained GAN for medical image enhancement. IEEE Trans Med Imaging, 40 (12) (2021), pp. 3955-3967.
[13]
J.Kumar, P.Kohli, N.Babu, K.Krishnakumar, D.Arthur, K.Ramasamy. Comparison of two ultra-widefield imaging systems for detecting peripheral retinal breaks requiring treatment. Graefes Arch Clin Exp Ophthalmol, 259 (6) (2021), pp. 1427-1434.
[14]
HeK, ZhangX, RenS, SunJ. Deepresidual learning for image recognition. Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition; 2016Jun 27-30; Las VegasNV, USA. Piscataway: IEEE; 2016. p. 770-8.
[15]
UlyanovD, VedaldiA, LempitskyV. Instance normalization: the missing ingredient for fast stylization. 2016. arXiv:1607.08022.
[16]
IsolaP, ZhuJY, ZhouT, EfrosAA. Image-to-imagetranslation with conditional adversarial networks. Proceedingsof the 2017 IEEE Conference on Computer Vision and Pattern Recognition; 2017Jul 21-26; HonoluluHI, USA. Piscataway: IEEE; 2017. p. 1125-34.
[17]
WooS, ParkJ, LeeJY, KweonI. CBAM: convolutional block attention module. In:Proceedings of the European Conference on Computer Vision (ECCV); 2018 Sep 8-14; Munich, Germany. 2018. p. 3-19.
[18]
Asymmetric loss for multi-label classification.In:Proceedings of the IEEE/CVF International Conference on Computer Vision; 2021 Oct 10-17; Montreal, QC, Canada. 2021. p. 82-91.
[19]
I.Goodfellow, J.Pouget-Abadie, M.Mirza, B.Xu, D.Warde-Farley, S.Ozair, et al. Generative adversarial networks. Commun ACM, 63 (11) (2020), pp. 139-144.
[20]
FuH, WangB, ShenJ, CuiS, XuY, LiuJ, et al. Evaluation of retinal image quality assessment networks in different color-spaces. In:Medical Image Computing and Computer Assisted Intervention—MICCAI 2019: 22nd International Conference; 2019 Oct 13-17; Shenzhen, China. Berlin:Springer; 2019. p. 48-56.
[21]
KöhlerT, BudaiA, KrausMF, OdstrčilíkJ, MichelsonG, HorneggerJ. Automaticno-reference quality assessment for retinal fundus images using vessel segmentation. Proceedingsof the 26th IEEE International Symposium on Computer-Based Medical Systems; 2013Jun 20-22; Porto, Portugal. Piscataway: IEEE; 2013. p. 95-100.
[22]
PiresR, JelinekHF, WainerJ, RochaA. Retinalimage quality analysis for automatic diabetic retinopathy detection. Proceedingsof 2012 25th SIBGRAPI Conference on Graphics, 22-25; OuroPreto, Brazil.Patterns and Images; 2012 Aug Piscataway: IEEE; 2012. p. 229-36.
[23]
K.He, J.Sun, X.Tang. Single image haze removal using dark channel prior. IEEE Trans Pattern Anal Mach Intell, 33 (12) (2010), pp. 2341-2353.
[24]
X.Guo, Y.Li, H.Ling. LIME: low-light image enhancement via illumination map estimation. IEEE Trans Image Process, 26 (2) (2016), pp. 982-993.
[25]
GuoC, LiC, GuoJ, LoyCC, HouJ, KwongS, et al. Zero-reference deep curve estimation for low-light image enhancement. In:Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020 Jun 13-19; Seattle, WA, USA. 2020. p. 1780-9.
[26]
Y.Jiang, X.Gong, D.Liu, Y.Cheng, C.Fang, X.Shen, et al. EnlightenGAN: deep light enhancement without paired supervision. IEEE Trans Image Process, 30 (2021), pp. 2340-2349.
[27]
HuangG, LiuZ, VanDer Maaten L, WeinbergerKQ. Densely connected convolutional networks. In:Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition; 2017 Jul 21-26; Honolulu, HI, USA. 2017. p. 4700-8.
[28]
DosovitskiyA, BeyerL, KolesnikovA, WeissenbornD, ZhaiX, UnterthinerT, et al. An image is worth 16 × 16 words: transformers for image recognition at scale. 2020. arXiv:2010.11929.
[29]
Y.Zhou, M.A.Chia, S.K.Wagner, M.S.Ayhan, D.J.Williamson, R.R.Struyven, et al. A foundation model for generalizable disease detection from retinal images. Nature, 622 (7981) (2023), pp. 156-163.
[30]
RonnebergerO, FischerP, BroxT. U-Net:convolutional networks for biomedical image segmentation. In:Proceedings of Medical Image Computing and Computer-Assisted Intervention—MICCAI 2015: 18th International Conference; 2015 Oct 5-9; Munich, Germany. Berlin:Springer; 2015. p. 234-41.
[31]
Z.Wang, A.C.Bovik, H.R.Sheikh, E.P.Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process, 13 (4) (2004), pp. 600-612.
PDF
(2854KB)
