Mining Bugs in Linux to Assess the Effectiveness of Automated Variability Testing: Master’s Thesis, Computer Science

The Linux kernel is a highly complex software system that offers more than 14,000 configuration options to support a wide range of hardware platforms and use cases. This leads to a high number of possible Linux kernel configurations. While configurability makes Linux versatile, it also complicates testing the kernel. In part, this is due to variability bugs, which are defects that occur only in specific configurations. As exhaustive testing of the Linux kernel is infeasible, automated integration testing systems, such as Intel's Linux Kernel Performance (LKP) system, utilize samples (i.e., a set of configurations) to test the kernel. Although understanding, identifying, and resolving variability bugs in the Linux kernel is the subject of extensive research, and despite LKP's impact on Linux kernel development, the LKP system has rarely been the subject of research. In this thesis, we analyze historical LKP data to assess the reproducibility, computational effort, and effectiveness of LKP tests. We contribute a set of tools for the acquisition and processing of LKP data. Through this tooling we can collect more than 180,000 data points from 2018 to 2025, including test results and configurations. We enhance the LKP data by providing a tool to emulate complete LKP samples. Additionally, we develop a prototype for automatic reproduction of LKP build tests, which we use to reproduce more than 15,000 LKP tests, and compile more than 4,000 kernel configurations. We use the aforementioned tools for a comprehensive evaluation of LKP's testing approach, and Linux kernel configurations. Our results show that reproducing LKP tests is challenging because the build process is inconsistent, and many tests rely on missing or inaccessible data. Some of our most important findings indicate that different configuration categories used by LKP (e.g., defconfig, randconfig, allyesconfig) vary in the effort required to test them and their effectiveness in finding defects. In general, configurations which require a high testing effort are also the most effective. However, we see that even effective configurations often redundantly find the same defects as other configurations. We conclude that random configurations used by LKP are effective in finding unique defects, with a moderate testing effort. Overall, our experiments provide the first detailed assessment of LKP as a large-scale automated variability testing system. We characterize the samples used for testing, and provide insights for variability bug researchers looking at LKP as a data source. Furthermore, we offer recommendations that could enhance the efficiency of LKP and similar testing systems.