A Representation-Space Early Warning Framework for Safe Medical AI Screening: Detecting Structurally Atypical Normal Predictions in OCT

Deep learning models for OCT classification now achieve high diagnostic accuracy, but accuracy alone does not guarantee safety. Models can confidently label cases as normal even when subtle structural abnormalities are present, leading to unsafe normal predictions. This paper introduces an Early Warning System (EWS) as a post-classification safety layer that operates in representation space rather than decision space. The framework performs within-class structural consistency analysis using Mahalanobis distance to identify geometrically atypical normal predictions, complemented by disease-direction analysis and clinical prioritisation. Evaluated on the Kermany OCT2017 dataset with duplicate leakage corrected, the EWS flags approximately 13% of true normal cases as structurally atypical, reflecting boundary-region variability requiring cautious review rather than classification error. Notably, the majority of these cases are undetectable using conventional confidence or entropy-based methods. The study also provides a systematic comparison of distance metrics, demonstrating that Mahalanobis distance outperforms Euclidean and cosine measures in high-dimensional feature spaces, particularly under covariance-aware modelling and dimensionality reduction. This work reframes safety in medical AI as a geometric consistency problem in representation space, introducing a practical and extensible framework for post-classification reliability assessment.