Elastic Vector Databases and Billion-Scale Disk-Backed ANN Indexing on Cloud Object Storage

Retrieval-Augmented Generation (RAG), enterprise semantic search, and multi-modal agentic memory systems increasingly require indexing billions of high-dimensional embedding vectors across hundreds of thousands of isolated tenant namespaces. Conventional first-generation vector databases rely on monolithic in-memory Hierarchical Navigable Small World (HNSW) graphs that pin full-precision FP32 vectors and adjacency lists inside expensive cluster DRAM, resulting in prohibitive cloud infrastructure costs and poor resource utilization under bursty, multi-tenant query distributions. This paper presents a disaggregated, serverless Elastic Vector Database architecture that couples billion-scale disk-backed Approximate Nearest Neighbor (ANN) indexing with cloud object storage (Amazon S3 / GCS) and local NVMe SSD caching. Synthesizing production engineering advances from Pinecone Serverless, Milvus 2.4 Knowhere DiskANN, and Turbopuffer object-storage-native indexing, our architecture separates stateless write/query compute workers from durable object storage through four key innovations: (1) geometric slab partitioning via balanced k-means centroid trees in RAM, (2) single-pass Vamana graph construction with compressed Product Quantization (PQ) codes in memory and full-precision vectors on NVMe/S3, (3) `io_uring` asynchronous beam-search prefetching that coalesces 4-kilobyte SSD sector reads, and (4) log-structured merge (LSM) object-storage compaction. We evaluate the architecture on a 1.5-billion-vector, 768-dimensional benchmark dataset deployed on AWS EKS with Amazon S3 and local NVMe cache tiers. Empirical benchmarks demonstrate that the proposed storage-disaggregated DiskANN + S3 architecture reduces cluster DRAM consumption by 89.2% and lowers monthly cloud hosting costs by 78.5% compared to in-memory HNSW clusters, while sustaining 95.8% Recall@10 at a 99th-percentile query latency of 11.4 milliseconds for hot NVMe namespaces and 168 milliseconds for cold S3 cache-miss namespaces. Research Domain & Specialization Track Master of Science in Information Technology (MSIT) — Cloud Storage Engines, Vector Search, and AI Data Infrastructure. Core Technical Highlights Empirical Production Case Studies: Rigorous architectural blueprints and telemetry evaluation across enterprise cloud deployments. Four-Tier Reference Architecture: Formal specification from edge ingress to distributed hardware execution pools. Mathematical Formulation: Formally derived algorithmic and optimization models with closed-loop verification. Multi-Modal Benchmarking: Comparative empirical baselines under fault-injection and synthetic workload scaling. Author Information Author: Sonu Kumar SinghRole: Senior Consultant Affiliation: Independent Researcher Publication Status: Preprint / Working Paper (MSIT AI & Cloud Systems Research Series) Reproducibility & Open Science All included architecture diagrams, telemetry schemas, and benchmark curves are open-access under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.