Kolmogorov-Arnold inspired convolutional networks for enhancing PPO-based online reinforcement learning

Proximal Policy Optimisation (PPO) is a well-known deep reinforcement learning (RL) algorithm that relies on a multilayer perceptron (MLP) to approximate PPO’s policy and value functions of the PPO. However, the representational capacity of MLPs may limit the performance of the PPOs. Recent research has shown that the Kolmogorov-Arnold Network (KAN) is a powerful substitute for MLPs, outperforming them across various domains. Although conventional KAN has demonstrated exceptional improvements over MLP-based PPO, it still has limitations, particularly when capturing complex correlations in input observations. Additionally, KAN heavily relies on the grid hyperparameter, and the performance is susceptible to the grid selection. To overcome these limitations, we propose a convolution-based Kolmogorov-Arnold network (Conv-KAN) as an alternative to MLP and conventional KAN in PPO. The novelty of the proposed approach lies in the integration of convolutional layers with the Kolmogorov-Arnold framework, which significantly enhances the model’s ability to capture feature correlations. Convolutional layers are inherently effective in modelling spatial dependencies and structured local features in the input observations. By leveraging this capability, the Conv-KAN aligns more naturally with the Kolmogorov-Arnold decomposition and improves the representation of complex patterns in the input space. As a result, this integration leads to more effective policy learning and decision making in RL tasks. Furthermore, the proposed approach is insensitive to the selection of grid size. Experimental results show that our Conv-KAN-based PPO outperforms MLP and recent KAN-based PPO across several RL benchmarks by approximately 12.9% and 108.7% in HalfCheetah and Hopper environments, respectively.