GBM-QTMT: Gradient Boosting Machine-based fast QTMT partition decision for VVC inter-coding

Abstract Versatile Video Coding (VVC) is the next-generation video coding standard. VVC proposes a new partitioning block structure called Quadtree with Nested Multi-Type Tree (QTMT) that introduces a more flexible partition shape using a quadtree (QT) and a nested multi-type tree (MTT) splitting compared to the previous splitting algorithms, namely Quadtree plus Binary Tree (QTBT) and QT structures adopted in High Efficiency Video Coding (HEVC). QTMT significantly improves coding efficiency, but it brings considerable computational complexity, which limits VVC's practical applications. To efficiently address this problem of redundant processing in QTMT structures in inter-mode prediction, this paper proposes a fast QTMT inter-partitioning algorithm based on gradient boosting machines (GBM). The proposed algorithm is divided into three steps. In the first step, the average local variance (ALV) is extracted from each coding unit (CU) to determine their homogeneity. Then, a classification-based gradient boosting machine (GBM) is employed to analyze and build a binary classification model from the extracted ALV features. The GBM model is employed to extract and efficiently obtain suitable thresholds for each QT CU size and a threshold between QT and MTT modes. In the last step, a fast QTMT partition decision algorithm is performed based on the extracted thresholds. The experimental results show that the proposed algorithm reduces a significant amount of encoding time, while the loss in coding efficiency is negligible.