A unified benchmarking framework for vector databases in scalable embedding-based image retrieval systems

This is the authors' abstract. We don't add key points for this paper.

Modern image retrieval and similarity search systems rely on high-dimensional embeddings that are produced by deep neural networks. To store and query these embeddings, vector databases are used. However, the lack of standardized, workload-aware benchmarks makes selecting the right system for a given workload challenging. This paper presents an adaptive benchmarking framework that systematically evaluates vector databases using realistic image-embedding workloads. The framework integrates CLIP ViT-B/32 (512-dimensional) and ResNet-50 embedding models with a consistent indexing and querying pipeline, thereby enabling fair and reproducible evaluation across the entire data lifecycle. We benchmarked four leading vector database systems–FAISS, Milvus, Qdrant, and pgvector–using CIFAR-10 and STL-10 image embeddings scaled from 100 to 10,000 vectors, with cross-dataset validation confirming that relative system rankings are consistent across different embedding distributions, with additional scalability tests up to 5 × 10 4 vectors. Our experiments reveal distinct performance trade-offs: FAISS inserts 10,000 vectors in 15 ms and answers queries in under one millisecond, but lacks built-in persistence. In contrast, pgvector, because of the transactional overhead typical of relational systems, needs 14.9 s for the same insertion task. Milvus balances 47–56 ms query latency with perfect Recall@5 across the lifecycle scales (100–10,000 vectors), with a small drop to 0.98–0.99 at the extended scalability scale of 50,000 vectors due to IVF cluster-boundary quantization error, while Qdrant, which offers the quickest update performance among persistent systems (approximately 2.7 ms across all tested scales), compromises on recall variability (0.88–1.00). At all tested scales, each of FAISS, Milvus, and pgvector scored perfect Recall@5. To support deployment decisions, a workload-aware multi-objective scoring model is suggested that considers latency, memory, and accuracy according to application requirements. The framework thus provides practical deployment insights for practitioners and also provides a reproducible foundation for vector database evaluation in AI-powered image retrieval.