ABSTRACT Standard deep learning optimisation is typically conducted on shape‐fixed loss surfaces. However, shape‐fixed loss surfaces may impede optimisers from reaching flat regions closely associated with strong generalisation. In this work, we propose a new paradigm named deformation mapping to deform the loss surface during optimisation. Moreover, we design various vertical deformation mappings (VDMs) and further analyse their contributions to the training process. Theoretically, we prove that deforming the loss surface enhances the optimiser's ability to filter out sharp minima in deterministic settings. Furthermore, by incorporating diffusion theory, we demonstrate that VDM exponentially reduces the escape time from sharp minima under stochastic noise and momentum. Empirically, visualisations of loss landscapes demonstrate that VDMs locate significantly flatter minima compared to standard optimisation. Furthermore, integrating VDMs into the training of various deep neural networks produces consistent accuracy gains on ImageNet, CIFAR‐10, and CIFAR‐100, with negligible additional computation. Notably, PreResNet‐20 on CIFAR‐100 achieves a 1.46% increase in top‐1 accuracy. These results indicate that the deformation mapping is a promising paradigm for improving optimisation and generalisation in deep learning. The code is available at https://anonymous.4open.science/r/Vertical‐Deformation‐Mapping‐2324 .
Chen et al. (Wed,) studied this question.