Visual quality assessment for decision making in standardization projects

In the context of the development of compression standards for visual media, typically, most decision making relies on the measurement with one or more objective quality metrics. In many cases, a small number of very simple metrics, such as the PSNR or the SSIM, are applied in decision making processes, e.g., in the context of adoption of coding tools in to a draft specification. THis applies to a variety of visual media under consideration, such as classical 2D video or various representations of immersive visual media like dynamic point clouds or meshes. Given the rise of learning-based coding tools and -apparently- competitive end-to-end learned coding schemes, as well as the increasing number of filtering blocks inside or outside of the coding loop of conventional coding schemes, the suitability of such metrics may be questioned. This is due to a potential lack of correlation with mean opinion scores acquired by subjective assessment, especially if specific artifacts, such as temporal consistency, are not well reflected by the metric. This problem can be even more significant for more advanced, potentially learning-based metrics, which may show unexpected behavior if being applied to compression artifacts which have not been known or seen by the time of training the corresponding metric. Advisory Group ISO/IEC SC 29/AG 5 MPEG Visual Quality Assessment is tasked with evaluating and recommending metrics and testing procedures for the use in standardization projects inside the body of MPEG Working Groups developing compression standards for visual media. This webinar presents recent insights in the performance of metrics and subjective assessment methods for a variety of visual media types. The evaluation includes laboratory tests as well as remote and on-site expert viewing sessions which are frequently conducted during MPEG standardization meetings. The results and the performance of such subjective tests are assessed and used to benchmark objective metrics commonly used or considered for application in the development process. Furthermore and outlook is provided to the dataset of compressed video for study of quality metrics (CVQM) which is currently being developed in AG 5 and which includes reconstructed video sequences from a set of conventional and learning-based coding schemes.