Design of spark movie recommendation system based on latent factor model

The traditional movie recommendation system has many problems in using common algorithm, such as relying on users' historical preference data, "cold start" problems for new movies and new us-ers, and poor recommendations for users with special tastes. Given the above situation, the system framework is built based on the MoviesLens dataset of GroupLens, using Spark, Flume, Kafka, and other big data technologies, and the SVD++ algorithm and Bias-SVD algorithm are used to improve the SVD algorithm and Funk-SVD algorithm in the traditional latent factor model, so as to improve the latent factor model. The recommendation effects and the corresponding RMSE and MAE values of the improved hidden semantic model on the dataset are compared, and the appropriate algorithms are se-lected to optimize the model of collaborative filtering algorithm based on nearest neighbors. By quantifying indicators such as movie genre, release time, region, recent hotness ranking, long-term hotness ranking, number of resources, and user ratings, data analysis is performed using visual charting techniques to find the movie resources that best match the users, and finally, the generated TopN list and rating of recommendation results are presented to users in offline and real-time, completing the construction of the movie recommendation system. The experimental results show that the selection of suitable algorithms applied to the movie recommendation system has greatly improved the recommendation effect compared with the traditional method, the accuracy rate has also been greatly improved, and the system is more in line with the user's personalized requirements.