What question did this study set out to answer?

This research aims to develop advanced representation learning methods that can effectively handle limited, unlabeled, and heterogeneously distributed data.

April 24, 2026Open Access

Beyond labels and centralisation: representation learning through data curation

Key Points

This research aims to develop advanced representation learning methods that can effectively handle limited, unlabeled, and heterogeneously distributed data.
Proposed a self-supervised batch curation strategy using Fréchet ResNet Distance for scoring unlabeled data.
Introduced FedMPR, a federated learning framework that prunes irrelevant weights during local model training.
Developed FedQuad to optimize representation space by minimizing intra-class distance with a quadruplet loss function.
The proposed methods significantly enhanced accuracy and feature quality on benchmarks like CIFAR10 and CelebA-Gender.
Improvements were noted in model robustness and generalization across diverse federated scenarios.
The algorithms demonstrated consistent performance superiority over existing baselines.

Abstract

Deep learning has achieved remarkable progress in vision, language, and multimodal tasks; however, its success remains heavily dependent on centralised, large-scale, and fully labelled datasets such as Imagenet, Open lab V7, etc. In real world, data is frequently limited, unlabelled, privately owned, and distributed across many devices, making traditional supervised learning challenging to scale. These limitations motivate the development of robust, novel representation learning methods capable of addressing under unlabeled, limited, and heterogeneous data constraints. This thesis addresses these challenges through four main contributions. First, we propose a self-supervised batch curation strategy that scores unlabeled data (batches) using the Fréchet ResNet Distance (FRD) as bad or good, enabling the semantically related and informative batches to improve feature quality under limited and unlabeled data regimes. Second, we introduce FedMPR, a federated parameter-selection framework that adaptively prunes irrelevant weights during local training for each client, improving representation robustness and generalization under highly non-i.i.d. data settings. To further analyse distributional data heterogeneity, the thesis also presents CelebA-Gender, a novel gender classification dataset designed to evaluate complex attribute-based data shifts and compare to the real world cases. Third, we present FedQuad, a framework that incorporates a reformulated quadruplet loss to minimise intra-class distance and maximise inter-class distance while mitigating representational collapse on global representation space. Finally, the thesis investigates partial federated model training combined with self-supervised learning, leveraging a frozen DINOv3 as a backbone and a lightweight projection head (Multilayer) to enable robust and computation-efficient representation learning under extreme client heterogeneity and limited participation. As a result, our experiments on many benchmarks such as CIFAR10, CIFAR100, TinyImageNet, and CelebA-Gender demonstrate that the proposed algorithms consistently outperform existing baselines in terms of accuracy, representation robustness, and feature consistency across many federated scenarios. To sum up, these contributions advance representation learning by enabling more generalisable, efficient, and data in one place based systems learning without relying on large, labelled, or centrally collected datasets.

Bookmark

View Full Paper