Uniform and scalable sampling of highly configurable systems

Abstract Many analyses on configurable software systems are intractable when confronted with colossal and highly-constrained configuration spaces. These analyses could instead use statistical inference, where a tractable sample accurately predicts results for the entire space. To do so, the laws of statistical inference requires each member of the population to be equally likely to be included in the sample, i.e., the sampling process needs to be “uniform”. SAT-samplers have been developed to generate uniform random samples at a reasonable computational cost. However, there is a lack of experimental validation over colossal spaces to show whether the samplers indeed produce uniform samples or not. This paper (i) proposes a new sampler named , (ii) presents a new statistical test to verify sampler uniformity, and (iii) reports the evaluation of and five other state-of-the-art samplers: , , , , and . Our experimental results show only satisfies both scalability and uniformity.