Scalable Cost-Optimized HPC Cluster on Google Cloud Platform

Abstract High-Performance Computing (HPC) clusters are essential for researchers handling complex computational tasks and big data analysis; yet, access to these powerful resources remains constrained. Most researchers rely on shared super-computing centers with limited availability and strict usage policies, creating bottlenecks that impede research projects requiring substantial computational power. This paper presents an approach that leverages Google Cloud Platform’s (GCP’s) computational resources to create a highly scalable, cost-effective, and self-managed HPC cluster. Our solution enables researchers to dynamically provision and decommission compute resources based on demand, optimizing costs while maintaining high performance. We provide a comprehensive architectural overview of the core components driving this self-hosted HPC setup, including detailed cost analysis and performance metrics from a real-world deployment. Our analysis demonstrates that this approach could achieve up to 83% cost reduction compared to traditional regular solutions, while maintaining comparable performance levels, making it a viable alternative for research organizations seeking flexible HPC capabilities.