Monte Carlo conformal prediction for quantifying uncertainty in radio galaxy classification under ambiguous ground truth

Dramatically increasing data volumes are forcing astronomers to adopt automated methods for the identification and classification of astronomical objects. Although deep-learning models are often well-suited to this task, obtaining a measure of uncertainty on their predictions is challenging. Here we consider the suitability of Monte Carlo conformal prediction (MCCP) set size and confidence as measures of model uncertainty for the astronomical classification of radio galaxies. We demonstrate this approach using model predictions from a pre-trained radio galaxy foundation model, fine-tuned on a smaller set of labelled radio galaxies. We calibrate the MCCP by obtaining annotator-derived soft label distributions, i.e. probability distributions over classes instead of single class assignments, for each of these labelled radio galaxies and compare the resulting set sizes and confidence scores to predictive entropy measures for each galaxy obtained using a supervised Bayesian deep-learning model trained using Hamiltonian Monte Carlo (HMC). The comparison reveals only a weak correlation between the measures. We suggest that this indicates that MCCP and predictive entropy capture fundamentally different aspects of the total uncertainty, and conclude that the applicability of MCCP in this context is severely limited by the practical considerations associated with obtaining soft label distributions for specialised classifications.