Photorealistic image style transfer based on explicit affine transformation

Global or local style transfer often relies on matrix transformations 1, 2, 3, 4 5. In any scale of the image feature space, the representation of color can be seen as the projection result of the features at that scale onto different coordinate bases 1, 5. Numerous studies have also demonstrated the validity of this approach. Starting from early non-neural network-based image style transfer algorithms, there have been approaches that utilize translation and scaling transformations to transform the coordinate bases of images, and most recently, the work of Neural Style Transfer neural style preset employs a simple network structure to predict the matrix parameters of affine transformations.We believe that if we want to solve for the global affine transformation matrix, we can completely abandon neural networks and directly solve it through the statistical features of the original image. First, we employ the Euler method to iteratively optimize the squared loss of the covariance matrix, quickly obtaining the optimal transformation matrix. Then, based on this, we adopt a stepwise transformation approach, decomposing the affine transformation into translation, rotation, and scaling transformations. We first determine the translation transformation based on the statistical differences between the content image and the style image. Then, by leveraging the distance-preserving property of rotation transformations, we provide the analytical form of the rotation matrix. Furthermore, we use basis transformation to nonlinearly blend and superimpose image channels, and solve for the optimal rotation matrix under different blending modes. Finally, we linearly combine the rotation matrices under different blending modes to obtain the final approximate result. Extensive experiments have demonstrated that the approximate solution can achieve results comparable to numerical solutions in image and video color transfer tasks. Meanwhile, our algorithm achieves near-optimal results in terms of runtime and quality trade-offs compared to existing algorithms. This advantage is particularly prominent in video realistic style transfer tasks.