Ground Truth Reliability and Clinical Generalisability in Deep Learning for Reticular Pseudodrusen—Response

We thank Kuang and Tian [1] for their interest in our study, ‘Deep Learning-Based Detection of Reticular Pseudodrusen in Age-Related Macular Degeneration’ [2]. We appreciate the recognition of its comprehensive validation of the deep learning (DL) model for detecting reticular pseudodrusen (RPD) on optical coherence tomography (OCT) scans in multiple external datasets. We also appreciate the three helpful points that Kuang and Tian have highlighted, which provide a useful opportunity to further clarify the findings of this study and discuss future directions from this work. First, Kuang and Tian described how the reliance on annotations of RPD from a single grader for training the DL model raises concerns regarding ‘potential systematic bias’, as it ‘systematically [introduces] subjective judgements into the model training’. They then queried if the observations of a higher level of model-grader agreement than the inter-grader agreement might indicate that the DL model ‘overfitted [to the] personal interpretation style of that specific expert’. We agree in part with Kuang and Tian that the DL model trained on annotations from the single grader would inevitably reflect the criteria used by this grader, as DL segmentation models fundamentally seek to produce predictions aligned as closely with the ground-truth labels provided. However, we respectfully suggest that Kuang and Tian's interpretation of the higher level of agreement that the DL model had to the four retinal specialists (‘model-grader’) than between the specialists themselves (‘inter-grader’) is not accurate. Specifically, these findings do not indicate that the DL model overfitted to the criteria used by the single grader, but rather showed how such criteria used by the DL model (reflecting the criteria used by the single grader) fall at least within the variation of the criteria used by the four retinal specialists. Second, Kuang and Tian helpfully point out that external validation of the DL model in this study was limited to the task of detecting RPD presence and not quantification of their extent. This is correct, which is why we limited the description of how this DL model achieved expert-level performance when evaluated in external datasets only to the task of detecting RPD. We agree with Kuang and Tian that evaluation of the segmentation performance compared to inter-grader agreement in external datasets would be useful. However, we believe that it would be even more useful for the DL model to be externally evaluated directly against clinically relevant outcomes. These include the development of late age-related macular degeneration (AMD) [3] or visual function loss [4, 5], in eyes with non-late AMD when compared to retinal specialists. We are currently conducting such further evaluations. To facilitate further independent external evaluation of the segmentation performance of our DL model, we had made it publicly available. Finally, Kuang and Tian also helpfully underscore how this DL model was only developed and evaluated exclusively using OCT scans from one type of device. This was indeed the first limitation that we had described in the discussion section. We agree that further work is needed to evaluate the performance of this DL model on scans from other OCT devices, and to finetune the model using additional OCT scans from other devices if required. Further work to leverage foundation models for retinal images like RETFound could also further improve the generalisability of this model. Again, we hope that the public release of our DL model would serve as a catalyst for our field to engage in such further work, to ultimately help to prevent irreversible vision loss for our patients with this important AMD phenotype. We therefore wish to thank Kuang and Tian again for their interest and helpful comments on our recent study [2]. This work was supported by the National Health & Medical Research Council of Australia (APP1027624 [R.H.G.]; GNT1194667 [R.H.G.], GNT2008382 [Z.W.]; GNT1181010 [R.H.G., Z.W.]), the Macular Disease Foundation Australia (research grant [Z.W., R.H.G.]), National Institutes of Health (R01AG060942 [Y.B.], OT2OD32644 [A.Y.L.]), and Research to Prevent Blindness (an unrestricted grant [Y.B., A.Y.L.]). CERA receives operational infrastructure support from the Victorian Government. The sponsor or funding organisation had no role in the design or conduct of this research. R.H.G. reports personal fees from Roche/Genentech, Bayer, Novartis, and Apellis, Belite Bio, Ocular Therapeutix, Complement Therapeutics, Boehringer Ingelheim Pharmaceuticals, Character Bioscience, Janssen, AbbVie and Astellas outside the submitted work. The other authors declare no conflicts of interest. Data sharing not applicable to this article as no datasets were generated or analysed during the current study.