An end-to-end direct reinforcement learning approach for multi-factor based mean-variance portfolio optimization

This paper introduces an end-to-end online portfolio decision model within the framework of direct reinforcement learning, seamlessly integrating the multi-factor model and mean-variance (MV) portfolio optimization. Recognizing that the classical decision process, which separates estimation and portfolio optimization into a two-step scheme, may accumulate estimation errors jeopardizing overall performance, our approach unifies these steps into a performance-oriented online decision process. This integration is achieved by tuning the neural network parameters directly with respect to the reward function, designed as a combination of the prediction error and realized MV utility. Specifically, we employ a neural network to estimate future returns and generate the factor loading matrix, enabling the computation of inputs for the MV portfolio optimization model. The network parameters are optimized with respect to the updated reward using the gradient method. We develop an online updating scheme for computing the gradient in backpropagation by providing explicit formulas for MV portfolio derivatives through the portfolio optimization layer. Utilizing real market data, we evaluate the proposed method against several benchmark portfolios in out-of-sample tests. The experiments demonstrate that our approach not only outperforms these benchmarks across various performance metrics but is also transparent to factor analysis, a favorable trait for practitioners.