A scalable hybrid recommendation framework for e-commerce using distributed big data processing

The online retail platforms of the last decade have gone through an exponential expansion, with Amazon being a prime example. This expansion has resulted in the generation of a vast amount of user–product interaction data, which traditional single-node systems struggle to process effectively. This research presents a scalable, deployment-oriented e-commerce recommendation system that uses Apache Spark, Hadoop, Pig, Hive, and Mahout within a Dockerized big data ecosystem. The pipeline integrates collaborative filtering using Alternating Least Squares (ALS) and content-based filtering using TF–IDF into a hybrid recommendation engine, and is automated end-to-end for ingestion, preprocessing, modeling, and evaluation, with a Flask web interface for monitoring and configuration. The system is evaluated on Amazon review datasets ranging from a 4.6-million-rating modeling subset to a full ingestion corpus exceeding 40 million records. On a real three-node Apache Spark/Hadoop cluster (HDFS replication factor 3), we empirically demonstrate sub-linear growth of training time with data volume, a measurable multi-node speedup, and fault tolerance under a deliberately induced node failure. The hybrid model improves recommendation quality over the standalone models, attaining an RMSE as low as ≈ 0.94 and Precision@10 up to ≈0.74 across the evaluation datasets. This research provides a comprehensive account of the design, implementation, evaluation, and future directions for scalable big data analytics in recommendation systems.