SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives

In this work we introduce a new optimisation method called SAGA in the spirit SAG, SDCA, MISO and SVRG, a set of recently proposed incremental gradient with fast linear convergence rates. SAGA improves on the theory SAG and SVRG, with better theoretical convergence rates, and has support composite objectives where a proximal operator is used on the regulariser. SDCA, SAGA supports non-strongly convex problems directly, and is to any inherent strong convexity of the problem. We give experimental showing the effectiveness of our method.