ABSTRACT Conventional clustering methods rely on unified features for unsupervised grouping and cannot adapt to users' personalized preferences or fine‐grained semantic needs. Recent advances in multi‐modal learning and large language models (LLMs) allow visual clustering to be adapted to user preferences through brief textual prompts. However, these LLM‐based approaches typically follow a two‐stage process that separates proxy learning from clustering, leading to a gap between user preferences and the learned clusters. To this end, we propose an end‐to‐end Personalized Visual Clustering with Self‐representation‐driven Sparse Subspace Learning (PVC‐S3L). We first use a LLMs to generate reference words based on user preferences, then match each image to the most relevant reference word, which is used to initialize the proxy embeddings. We then develop a self‐representation loss that encourages proxy embeddings to be sparsely represented by their neighbors, so that the learned embeddings reflect the underlying subspace structure, and jointly optimize it with cross‐modal alignment and text regularization losses. This unified objective tightly couples proxy learning with clustering, resulting in clusters that better reflect user‐specific semantics. Experiments on multiple public image benchmarks demonstrate that our method consistently outperforms state‐of‐the‐art baselines in clustering accuracy.
Jing et al. (Sun,) studied this question.